Evaluating an AI agent in production: why traces matter
An average success rate hides an agent's path failures. This method links traces, outcome evidence, cost and reliability so teams can compare versions without trusting a single score.
An AI agent does more than produce text. It plans, calls tools, reads data and may change a business system. Evaluation therefore cannot stop at the final answer. Two agents can show the same sentence even though one used the right source while the other bypassed the task or exploited a weak grader.
This article proposes an applied research method: keep the complete episode trace, require evidence for the outcome, classify failures and publish a quality curve against cost and latency. The goal is a reproducible measurement before and after changing a model, prompt or tool.
The short answer
Evaluate an agent on representative scenarios with a known initial state and an independent oracle. Record steps, tool calls, observations, checks and stopping decisions. Return three outcomes instead of one score: evidence-supported success, evidence-supported failure and unknown. Report quality by scenario, cost, latency, policy violations and stability across repetitions.
Nexxom diagram. A trace becomes a measurement only when it is linked to outcome evidence and a published decision rule.
Why an average score is not enough
An interactive benchmark often reduces a complex trajectory to pass or fail. That is convenient, but it can hide a weak outcome check. The paper Can Agent Benchmarks Support Their Scores? shows that a check verifying only a click or button can report success without proving that the intended business state was reached. The authors separate Evidence Pass, Evidence Fail and Unknown.
The same issue appears in security evaluations. AgentDojo uses 97 realistic tasks and 629 test cases to observe agents calling tools over untrusted data. A global score does not show whether the agent failed the task, resisted injection or simply stopped before a dangerous action.
An agent can also optimize the metric instead of the goal. The NIST Center for AI Standards and Innovation documents solution contamination through Internet access and grader gaming. A trace can expose these behaviours, while a final number confuses them with genuine improvement.
What to capture in a trace
A useful trace is a structured log, not a raw conversation copy. Each event should keep an episode identifier, timestamp, type and link to the previous step. Sensitive data must be redacted before storage.
| Element | Measurement question | Example field |
|---|---|---|
| Context | Which version and permissions were active? | model, prompt_version, policy_version, principal |
| Action | Which tool did the agent call and with which parameters? | tool_name, arguments_hash, authorization_decision |
| Observation | What data was returned? | source_id, retrieval_score, content_hash |
| Check | Which rule accepted or rejected the step? | check_id, verdict, evidence_refs |
| End state | Which real state was reached? | state_before, state_after, stop_reason |
Do not automatically log a model's private chain of thought. For reproducibility, tool events, useful inputs and outputs, evidence and control decisions are usually enough. The NIST AI 800-2 draft stresses describing protocol, attempts, feedback and evaluation conditions.
A six-step protocol
1. Define the episode and expected state
Start with an observable business action, such as creating a ticket with the right priority or answering a request while citing an internal policy. Describe the initial state, permissions and success definition before running the agent. A scenario needs a verifiable end, not only a convincing sentence.
2. Build a case matrix
Sample easy, ambiguous, long and adversarial cases. Keep a frozen test set separate from development cases. WebArena and BrowserGym show the value of environments where an agent navigates application state rather than answering an isolated question. For an enterprise, gradually replace public tasks with anonymised internal scenarios.
3. Freeze comparison variables
For two versions, keep task model, context, tools, time limit and policy constant. If a change affects several variables, create a separate experimental condition. Repeat each case with controlled seeds or conditions, then publish trials and exclusions.
4. Verify outcome evidence
Attach an oracle to every task: database state, executable test, reference document or double human annotation. A language judge can help triage, but it should not be the only evidence for a critical action. NIST's Building Evaluation Probes into Agentic AI describes probes that link a claim to trusted documents and preserve an audit trail.
5. Classify failures by step
Use a simple taxonomy for planning, tool selection, authorisation, observation, execution, verification and communication. An agent that selected the right tool but received stale data has a different defect from an agent that ignored a prohibition. This granularity makes remediation actionable and avoids changing a model when the connector is at fault.
6. Publish bounds instead of false precision
Calculate evidence-supported success on Evidence Pass cases. Report Evidence Fail and Unknown separately, and provide an interval or bounds when the sample is small. Add cost per episode, p50 and p95 latency, tool calls, policy violations and abstention rate.
| Measure | Operational formula | Decision it supports |
|---|---|---|
| Evidence success | Evidence Pass / evaluated episodes | Can this task be entrusted? |
| Uncertainty | Unknown / evaluated episodes | Does the oracle need work? |
| Efficiency | total cost / Evidence Pass | Is the gain worth its spend? |
| Robustness | success without violation / episodes | Is the system safe enough? |
| Stability | dispersion across repetitions | Is the result reproducible? |
Reading a trace without overclaiming
Read from output to cause. Check the final state first, then evidence, tool call and authorisation. Look for gaps between what the agent says it did and what logs confirm. This review can be automated for triage, but rules and a human sample must remain documented.
NIST describes an iterative process in Analyzing Transcripts from AI Agent Evaluations: prepare logs, write a rubric, validate a scanner on known examples, then scale analysis. A practical rule is to retain traces from a representative sample even when cost prevents publishing every episode.
Test cheating risk explicitly. NIST's practices for detecting cheating recommend transcript review, explicit environment rules and reproducible affordance descriptions. For example, forbid access to solutions and record network requests when the task requires it.
Example decision table
| Trace observation | Cautious interpretation | Next action |
|---|---|---|
| High score, unknown evidence | The grader does not prove the final state yet | Strengthen the oracle and recalculate |
| Good result, higher cost and p95 | The model succeeds but is harder to operate | Add a budget and stopping rule |
| Failures concentrated on one tool | The connector or permissions are likely at fault | Test the tool separately, then the agent |
| Variable success across repeats | The policy is unstable or the scenario underspecified | Reduce randomness, enrich context and repeat |
| Rare but severe violation | The mean hides a tail risk | Block production and add an adversarial test |
Limits and safeguards
Traces do not prove everything. An event can be missing, forged or expensive to retain. Logs may expose personal data, secrets or security details. Define retention, hash identifiers, separate sensitive evidence and control who can replay episodes.
An oracle is rarely perfect. An automated test may verify state but miss a business consequence. Human annotation understands context but introduces disagreement. Document oracle versions, measure agreement and mark ambiguous cases Unknown instead of forcing them into success.
Evaluation must not become a permanent leaderboard. An agent changes with tools, data and policies. Keep a versioned regression suite and rerun it for each material change. The survey of LLM agent evaluation reminds us that capabilities, behaviour, reliability and safety are different objectives requiring different protocols.
Nexxom's production decision method
For a first pilot, choose ten to thirty representative episodes, define an oracle for each and capture a minimal trace. Reject any condition without evidence. Compare two versions on the same frozen set and publish Evidence Pass, Evidence Fail, Unknown, cost and p95. Then add an adversarial scenario and a blind human review of traces.
This sequence gives a more useful decision than an average score: it says whether a system is ready for a specific task, why it fails and which investment will reduce risk. This connection between research, instrumentation and engineering is how Nexxom turns an agent demonstration into an operable system.
Primary sources and further reading
- Can Agent Benchmarks Support Their Scores?, 2026.
- AgentDojo, 2024.
- WebArena, 2023.
- BrowserGym, 2024.
- NIST AI 800-2, 2026 draft.
- NIST Building Evaluation Probes, 2026.
- NIST transcript analysis, 2026.
- NIST cheating evaluations, 2025.
- NIST transcript review practices, 2025.
- HELM evaluation framework, Stanford CRFM.
- Inspect AI, UK AI Security Institute.
- LLM agent evaluation survey, 2025.
