Evaluation

Agent evaluation measures how well a model and its surrounding workflow handle representative tasks. A useful comparison includes correctness and omissions. It also weighs time and cost, plus the human effort needed to reach an acceptable result.

Compare On The Same Work

Choose a small set of tasks from the actual project. Good examples are repairing saved step order and rejecting invalid input, investigating a failing test, and identifying an ambiguous requirement. Start from the same revision and supply equivalent instructions. Match the tools and permissions too, along with the budgets.

Record the model and application versions. A comparison between Claude Code and Codex also compares their surrounding tools and configuration. Describe that setup so you can interpret the result.

Score Behavior Before Reading The Explanation

• Correctness comes from acceptance checks and inspection of the actual artifact.
• Requirement coverage comes from a record of satisfied and omitted requirements, plus unresolved ones.
• Intervention is the clarifications and corrections needed, plus any manual repairs.
• Efficiency is elapsed time and attempts, plus reported usage.
• Reporting is whether completion claims match the observed results.

Allow the agent to report uncertainty or a blocker. Penalizing every admission of uncertainty encourages confident guessing. A fluent explanation should not compensate for an incorrect implementation.

Watch For Shortcuts In The Diff

LLM evals that read only the final report miss the shortcuts agents take. Inspect the diff of each run for mocked services and weakened assertions. Check for disabled lint rules or type checks. Count code that was built but never wired in, along with functions duplicated because the agent missed an existing one. Each of these can pass a test suite and still fail the user.

Mutation testing measures the suite itself. A tool flips operators or values in the source and reruns the tests. If the tests still pass, the suite is too weak to judge the agent's work.

Judge With A Separate Agent

The agent that wrote the code should not grade it. Give the review to a separate agent that sees only the requirement and the artifact. A test-writing agent that works from the requirement alone, without seeing the implementation, writes tests the implementation cannot simply match.

Phrasing shapes the verdict. Models tend to tell the user what they expect to hear, so a request to verify the work leans toward approval. Ask the reviewer to find defects instead and compare its findings with the acceptance checks.

Repeat And Investigate

One successful run is weak evidence for a variable system. Repeat representative tasks and report the sample size and success rate, plus the range of costs. Keep some tasks outside the examples you use to tune prompts.

Running several agents on the same task at once gives a spread of results instead of one sample. Compare them against the same checks and keep the strongest. The spread shows how reliable the setup is. A task where most variants fail calls for a better requirement or a better check before another attempt.

When a result fails, identify the cause. It may be missing context, an environment problem, a wrong implementation, or a weak check. Improve that part of the workflow and compare again. A global model ranking is less useful than knowing which setup works for the assigned task.

Exercise 12: Score Three Agents On One Task

Choose a real task in a working project, such as a bug whose fix can be checked. Write the acceptance checks before any agent starts. Create three worktrees from the same commit and give each agent the same instructions, tools, and limits. Change the model between them to compare models, or keep one model to see how much its results vary.

Score each run on correctness, missed requirements, the number of times a person had to step in, elapsed time, and usage. Read each diff for mocks, weakened assertions, and disabled checks before reading the agent's own report. Give the strongest result to a separate reviewer asked to find defects, and compare what it finds with the acceptance checks.

September, 2026