How to Test Foundation Models Evaluations? 2026 iOS AI Tutorial

A unit test can prove that your app applies a fixed rule to a fixed input. It can check a boundary, a state transition, or whether a required field is present. That’s valuable, but it cannot tell you on its own whether an open-ended generated response is useful, on-topic, or suitable for your product.

Apple describes Evaluations as a framework for measuring the quality of intelligent features. Its documentation and WWDC26 framework session explain the roles of evaluation samples, metrics, evaluators, and results. Use those pieces to answer different questions. The dataset defines what you test. The metric states what you care about. The evaluator applies a scoring method. The report helps you inspect what happened.

Keep these results separate in your test strategy. A deterministic assertion that checks a response field is not a quality score. An aggregate evaluation score is not proof that every sample passed. A human reviewer can catch a mismatch that the evaluator missed, but that review should be recorded as a distinct judgment.

A small, tidy dataset can make a feature look better than it is if it contains only easy inputs. Start from real use cases and user requests, then include variations that can change the expected response: incomplete context, ambiguous intent, unusual phrasing, and requests outside the feature’s scope.

Apple’s dataset design guidance is the reference for designing evaluation datasets. Use it to shape a sample set around what your app must do, not around whichever examples are easiest to score. If you generate synthetic samples, review them before treating them as representative. A generated example can be fluent and still fail to express a realistic user need.

For each sample, record enough context to reproduce the evaluation: the input, relevant app state, expected behavior, and any evaluator instructions. Keep the sample set versioned. If you change it while comparing prompt strategies, you will no longer know whether a score shift came from the prompt or from a different test set.

Avoid treating a single successful example as evidence of a general improvement. One response can reveal a bug, but it cannot represent the range of inputs your users may send. Use the evaluation report to find specific failures, then add a sample only when it captures a meaningful product requirement rather than a one-off wording preference.

An evaluator is only as useful as the question it answers. If your requirement is “the response must include a valid identifier,” define a check that directly tests that condition. If the requirement is “the answer should be helpful,” first decide what helpful means for your feature. A score with a vague rubric can look precise while measuring something your users do not care about.

Apple documents EvaluationTrait in its EvaluationTrait reference. Follow the documented API and examples for your installed toolchain. Don’t assume that an evaluator’s score is an objective measure of quality simply because it is returned as a result. Test the rubric against examples that you or your team have already reviewed.

Keep the scope of the metric narrow enough to interpret. If a response gets a weak score, you should be able to tell what to investigate: prompt wording, missing context, tool selection, or the rubric itself. When you cannot explain what a score means, don’t use it as a release gate.

When you compare prompt strategies or model configurations, hold the evaluation conditions steady. Run both candidates against the same sample-set version. Apply the same evaluator instructions. Keep the app context consistent. Then compare aggregate results and inspect individual samples where the candidates differ.

Apple’s Evaluations documentation describes using evaluations to compare strategies and track quality changes. Use that capability to investigate a specific change, not to claim a general performance ranking from one run. A difference in a single generated result is a reason to examine the case. It is not, by itself, evidence that one configuration will be better for every user or every run.

Use a change record that answers four questions:

• What changed: prompt text, configuration, app context, or evaluator?
• Which version of the sample set did you use?
• Did you keep the scoring instructions and review method constant?
• Which individual results explain the aggregate movement?

If you change the evaluator at the same time as the prompt, you’ve changed both the system and the measuring stick. Separate those changes where possible. If you cannot separate them, label the comparison as inconclusive and rerun with a controlled setup before deciding to ship.

For an agentic feature, the final text may look acceptable even when the app chose the wrong tool, skipped a required action, or called a tool when it should not have. Define expected behavior for the interaction, not just the last sentence shown to the user.

Apple’s tool-calling evaluation guidance covers evaluating tool-calling behavior. Use it to structure samples around the request and expected action. Then inspect the call trace and the final result as related but distinct evidence.

For each scenario, write down:

• What the user asked and what context the app had.
• Whether a tool should be called, and which action is appropriate.
• What information the tool result should contribute.
• What the final response must communicate.
• What should happen when the tool is unnecessary, unavailable, or given insufficient input.

Include negative cases. A feature that calls the correct tool for a clear request may still behave badly when the request is ambiguous. Decide where human review is required, especially when an incorrect action could have consequences that a text-only score would not capture.

Apple documents that Evaluations can integrate with Swift Testing. Use the Evaluations documentation and the framework’s WWDC26 session to follow the supported setup for your project and toolchain. Put the evaluation in the appropriate app test target, and keep ordinary deterministic tests alongside it.

A practical run looks like this:

• Confirm that the project opens with the intended Xcode installation and target platform.
• Identify the test target that can exercise the feature and its required context.
• Add or update representative evaluation samples for the behavior under test.
• Define the metric and evaluator instructions before comparing results.
• Run the test target using the documented Swift Testing integration.
• Review aggregate results to spot a broad change, then inspect individual samples to find the cause.
• Record the tested configuration, sample-set version, and evaluation outcome so another run can be compared.

Use the evaluation report to understand the group-level result. Use individual sample results to diagnose concrete failures. Use ordinary assertions to prove fixed requirements. These are different forms of evidence; none should be used as a substitute for the others.

Do not copy a universal pass threshold from another app. Your acceptance criteria depend on the product requirement, the risks of an incorrect response, the evaluator’s agreement with human review, and the cases represented in your dataset. If you set a release gate, explain what it protects and what it does not.

Evaluations that you run through Xcode require a compatible macOS development environment. Apple’s Xcode system requirements are the source to check for supported operating-system requirements. The Xcode release record also matters: Apple published Xcode 27.1 on October 15, 2026, so validate the exact Xcode version your project requires rather than assuming that every Mac environment has the same toolchain.

Before making a remote Mac part of your test workflow, verify that you can open the representative project, reach the required test target, run the evaluation, and retain results in a form your team can review. Also record the inputs and configuration needed to reproduce the run. This is a validation plan, not a claim that a particular hosted configuration or rental has been tested with your project.

Use a representative project, not an empty sample, for acceptance. Confirm that the test target can run and that you can inspect both the aggregate report and individual results. If any part fails, resolve the environment issue before interpreting the evaluation as a product-quality failure.

Frequently asked questions

How do I run Foundation Models Evaluations with Swift Testing?

Add an evaluation to the appropriate app test target and follow Apple’s documented integration, including the EvaluationTrait guidance. Keep deterministic assertions for inputs, state changes, and other fixed behavior. Then inspect the evaluation report: review individual examples as well as aggregate results, because a passing test run does not prove that every generated response meets your product requirements.

How can I tell whether an AI evaluation metric fits my app?

Start with the user-visible requirement, then check whether the evaluator can assess it directly. A format requirement may support a clear rule; tone, usefulness, or factual adequacy may need a rubric and human review. Test the evaluator against examples you have already judged. If its scores disagree with your acceptance criteria, revise the rubric or add a review step instead of treating the score as ground truth.

How should I check for quality regressions after changing a prompt or model configuration?

Run the old and new configurations on the same versioned sample set, with the same evaluator and scoring rules. Compare aggregate results, but also inspect examples whose scores changed and cases that matter most to users. Record the change under test and the evaluation output. A difference in one run is evidence to investigate, not proof that one configuration is generally better.

What should evaluation samples include for an AI feature that calls tools?

Represent the user request, relevant context, expected tool choice or action, and acceptable final outcome. Include cases where a tool is unnecessary, where information is missing, and where the expected action should not occur. Judge the call trace separately from the final response. Apple’s tool-calling guidance can help shape the dataset, but review whether each example reflects your app’s actual behavior.

The decision is not “local Mac or remote Mac” in isolation. It is whether your environment can run the required Xcode test target, preserve the evaluation inputs, and let you review the output. If your local Mac already does that reliably, adding another environment may only add maintenance. If local access is inconsistent or you need a separately available macOS test machine, a remote Mac can be worth evaluating. A local machine can tie runs to one workstation, leave tests unavailable when that machine is in use, and make it harder to separate development from repeatable build work. A remote setup adds access and environment-management steps, so verify those before depending on it.

If you need a temporary macOS environment to validate Xcode and Swift Testing, review CALMVPS Mac rental options and the available order page only after you’ve defined a representative project and acceptance checklist. For recurring, heavy workloads or tests that require local physical interfaces, assess whether purchasing and maintaining a Mac is a better fit. Whichever route you choose, keep the sample set, evaluator, and review criteria fixed when you compare results.