How to evaluate AI agent accuracy and reliability

Grade the state the agent changed

A booking agent can report success without creating a reservation. Your evaluation should verify the destination state, not only the wording of its final message. Anthropic’s evaluation guide distinguishes tasks, trials, graders and outcomes, and describes code-based, model-based and human grading. Evaluation reference.

For an email-drafting workflow, define a passing trial as: correct ticket selected, draft created, required facts supported and no message sent. That definition makes a safe draft successful even when the agent appropriately requests clarification.

Use a small dataset with explicit coverage

Start from real task categories with private information removed. Keep examples that require a normal answer, clarification, refusal or escalation. Label each case with expected actions and forbidden actions. A list of easy demonstrations gives a misleading score even if the calculations are correct.

A 90% score can describe very different evidence

With 9 passes from 10 independent trials, the observed rate is 90%. A 95% Wilson interval is approximately 59.6%–98.2%. With 90 passes from 100 comparable independent trials, it is approximately 82.6%–94.5%. These are calculated examples, not benchmark results. They show why the sample size belongs beside the score. Statistical method.

Use the evaluation calculator to inspect your counts. An interval does not compensate for unrepresentative examples, inconsistent grading or a test suite contaminated by prompt tuning.

Separate quality from permission failures

Do not average unauthorized actions away inside an overall quality score. Establish a distinct safety gate and review every violation. Choose release criteria before comparing prompt versions.

Make the comparison reproducible

Keep task IDs stable. Freeze the model identifier, workflow version, tool fixtures and grading rules. Run the baseline and candidate against the same cases in isolated test environments. Investigate regressions case by case before accepting an aggregate improvement. A model grader needs a rubric and calibration against human review; it is another component with errors.

Next action

Convert a small task spreadsheet using the CSV to JSON converter. Include expected outcome, forbidden action and evidence location. Then pair offline tests with production monitoring. An observed pass rate is evidence about that evaluation setup, not proof of universal reliability.

Add a release decision record

For each candidate release, keep the dataset version, model identifier, workflow revision, grading rules and result counts. Explain which cases changed from pass to fail and whether those failures affect permissions, factual support or ordinary completion. This prevents an aggregate improvement from hiding a regression in the most consequential workflow.

Use the same task definitions when comparing versions, but avoid repeatedly tuning against every available test. Retain a held-out set for a final check and document when a case becomes part of the development suite. Otherwise, the score may reflect familiarity with the test rather than a transferable improvement.

Write the release decision in operational terms: which workflows may run, which still require review and what triggers a rollback. A finite evaluation cannot prove universal safety. It can provide concrete evidence for a bounded decision when the dataset, assumptions and remaining failure modes are visible to the person who owns the system.

Sources and review notes

Anthropic evaluation terminology and NIST proportion-interval methods checked October 14, 2026. The task examples, allocation and arithmetic are original illustrative material. No external benchmark leaderboard or client success rate is claimed.