THE KEY ANSWER
Evals are repeatable tests of an AI system on tasks with a defined outcome or criteria. Combine checkable rules, expert evaluation, and auxiliary model evaluation. Test both task execution and correct abstention from action.
A test case starts with a user decision
Describe the input, available data, and expected result. For extraction, this may be a set of fields; for an assistant, it may be following the correct procedure; and for an agent, it may be an actual state change in the system. A "done" message is not sufficient if the order has not been saved. The criterion should match what the user considers the completion of the work.
Also consider situations where action is undesirable: lack of consent, incorrect permissions, or incomplete data. A set composed solely of tasks to be executed may reward a system that acts too hastily. Your own list of cases should derive from the process and real reports, not just from a catalog of example prompts.
Context and references: Anthropic: Demystifying evals for AI agents
Match the evaluation method to the properties of the outcome
Numbers, identifiers, and format can often be checked by code. Compliance of a response with a procedure may require an expert. An evaluating model is useful for a larger number of texts, but its verdict must also be verified. Do not replace a lack of criteria with another question like "Is the answer good?"
Propose a scale with a description of the differences. Separate critical errors from minor linguistic awkwardness. A single average may hide a rare violation that the business does not accept. Show results by task type and error category. This allows you to decide whether a solution can be deployed for part of the scope, rather than accepting or rejecting the whole.
Comparison must be repeatable
Preserve the version of the data, configuration, and evaluation method. When changing the model, do not simultaneously modify all examples. Separate the development set from the acceptance set to avoid tuning the system only to known answers. For tasks with variable outcomes, run several trials and show the variance.
The test environment should start from a known state. An agent must not use an order created in a previous trial or inherited permissions. Also control for tool failures to avoid confusing an environment error with a model error. A result without the ability to reproduce the cause has limited value in a release decision.
How to transfer the result to the release process?
Establish minimum deployment conditions and classes of errors that block publication. After a change, compare the result with the current version and check fresh examples. If improving one area worsens another, the decision should consider the weight of both tasks. Do not choose a version solely based on a favorable aggregate number.
After deployment, add representative errors to the set, removing unnecessary user data. Regularly review cases whose expectations have become outdated. Evals are part of product maintenance. They should help make decisions faster, not create an elaborate ceremony unrelated to actual quality.
WHERE TO START
Bring this into your project.
- Record the input, initial state, and expected effect.
- Add cases requiring refusal or clarification.
- Separate critical errors from cosmetic ones.
- Version the data and control the independence of trials.
Choose one thing your process is missing today. It's a useful topic for your first conversation with the team.
QUESTIONS AND ANSWERS
Frequently asked questions.
How many examples are needed to start?
Start with a set covering the most important types of tasks and errors. The number depends on the diversity of the process. A small, well-described set is more useful than a large set without reliable expectations.
Can a second model evaluate the first?
It can support evaluation, but it should not be an unverified arbiter. Check its alignment with human evaluation, disputed cases, and susceptibility to the content it evaluates.
Sources and context
- Anthropic: Demystifying evals for AI agents ↗
The source describes tasks, trials, and various methods for evaluating agents. A detailed acceptance plan should derive from the requirements of a specific product.
Prepared by the ALGOV team. Current as of September 8, 2026. Examples describe possible scenarios, not results from client projects. How we create our guides.
