Every Application Has Two Surfaces—and Agent Evals Should Diagnose the Second
Agent experience is not an alternative to agent evals. It is the product surface those evals should help us diagnose and improve.
- Agent Experience
- AI Agents
- Evals
I was in a Codex session recently, trying to design an MCP tool.
Codex kept grilling me.
Should this be one flexible query tool or several narrow tools? Which fields should be required? Should the tool return raw records or a summary? How should pagination work? What should happen when a valid query is too broad?
I knew the product intent, the sensitive data, and the dangerous actions. But I did not know how to answer every question about what an agent would find easiest to use.
My first reaction was: we should test the agent experience with agents.
Then I went looking to see whether this was already a known practice.
It is.
Anthropic has published almost exactly this workflow: prototype tools, create realistic evaluation tasks, run agents against them, inspect transcripts and tool-call metrics, and revise the tool descriptions, schemas, and boundaries. Modern agent-eval guidance already treats the full trace or trajectory—not only the final answer—as evidence.
So this post is not proposing a new alternative to evals. Calling it a separate testing discipline would just rename work that the agent-evals community is already doing.
The useful idea is narrower.
Any application that agents are expected to operate now has two product surfaces: one for humans and one for agents. Evals are how we test the agent surface. But to improve that surface, we have to treat it as a product variable instead of hiding it inside a single score for the entire agent system.
One Application, Two Surfaces
The human surface is familiar: pages, buttons, forms, navigation, notifications, and the visual hierarchy through which a person understands the product.
The agent surface is made of different materials:
- Tool names and descriptions
- MCP tools and resources
- Input and output schemas
- Stable identifiers and structured results
- Context files and machine-readable documentation
- Permissions, confirmations, and side-effect boundaries
- Errors and recovery instructions
- Audit information carried through the workflow
I am using surface as a product-design term, not proposing a new protocol.
The current Model Context Protocol tool specification already formalizes much of this interface: names, descriptions, input schemas, output schemas, behavior annotations, structured results, and errors that models can use to correct a call. Those fields are not incidental metadata. They shape what the model perceives it can do.
The same backend capability can therefore produce two very different experiences.
A human can scan a screen, infer hierarchy, explore, and back out. An agent discovers capabilities through context and contracts. It has to infer intent from tool names, descriptions, fields, results, and errors.
An application can have excellent UX and poor agent ergonomics. It can also expose a clean tool surface behind an awkward human interface.
That is why the distinction matters—even though the way we test the agent side is still evals.
AX Is the Object; Evals Are the Method
This is the cleanest vocabulary I have found:
- Agent experience is the quality of the environment and interface we give the agent.
- An agent eval is a method for measuring how the agent system performs on defined tasks.
- A trace or trajectory is the evidence of what happened between the task and the outcome.
- A controlled comparison helps determine whether changing the surface actually improved performance.
UX is not an alternative to usability testing. It is what usability testing helps us understand and improve.
AX should be treated the same way. It should not become a fashionable synonym for evals.
Surface-Focused Agent Eval
Use Evals to Diagnose the Interface Layer
Grade the outcome, inspect the trajectory, change one surface variable, and replay.This Pattern Already Exists
The strongest evidence is not a vague analogy. It is existing practice.
Anthropic’s 2025 article Writing effective tools for agents—with agents recommends realistic, outcome-based tasks; optional checks for expected tool use; raw transcript review; and metrics such as tool calls, errors, runtime, and token consumption. It explicitly uses those eval results to improve tool descriptions, parameters, pagination, returned context, and tool boundaries.
Anthropic’s later guide, Demystifying evals for AI agents, defines the transcript or trajectory as the complete record of a trial and distinguishes it from the final state of the environment. It also recommends repeated trials because agent behavior is non-deterministic.
The original τ-bench paper evaluates agents interacting with users, domain policies, and API tools, then checks the resulting database state. Its pass^k metric asks whether an agent succeeds consistently across repeated trials, not whether it got lucky once.
OpenAI’s current agent-evaluation guide explicitly includes questions such as whether the agent selected the right tool or violated a safety policy, then recommends using trace results to refine tool surfaces alongside prompts, routing, and guardrails. Its trace-grading guide defines the trace as the end-to-end record of decisions, tool calls, and reasoning used to diagnose why an agent succeeded or failed.
OpenAI has also reported the same kind of product lesson from its own systems. In Inside OpenAI’s in-house data agent, the team describes how overlapping tools confused the agent and how consolidating the tool set improved reliability.
The language itself is not mine either. Netlify says it introduced Agent Experience in 2025, and in 2026 it published AXIS, an open-source framework that runs scenarios against real agents and captures tool calls, responses, recovery attempts, time, and token use. That does not make AX a settled cross-industry standard, but it shows that the product lens and the measurement pattern are already emerging in practice.
So the method is established: realistic tasks, graders, repeated trials, state verification, trace inspection, and regression suites.
That is agent evaluation.
What the Two-Surface Framing Adds
An agent eval usually measures an entire system:
model + prompt + harness + tools + context + data + environment
That is necessary because the user experiences the combined result.
But a combined score does not automatically tell us which layer to improve.
If the agent fails to find a record, the model might be weak. The prompt might be ambiguous. The data might be missing. The tool description might be misleading. The input schema might invite an invalid query. The output might omit the identifier needed for the next step.
The two-surface framing turns the agent-facing interface into an explicit diagnostic target.
It asks whether the failure came from:
- Discovery: Could the agent find the relevant capability?
- Selection: Could it distinguish overlapping tools?
- Contract: Were the required arguments and defaults legible?
- State: Did results carry stable identifiers and enough context for the next step?
- Recovery: Did an error provide an actionable next move?
- Safety: Were permissions, side effects, and confirmation boundaries enforced?
- Efficiency: Did the interface cause redundant calls, over-fetching, or avoidable context use?
These are not new grader types. They are hypotheses about why an eval passed or failed.
That matters because the wrong diagnosis creates the wrong fix. A team can keep adding prompt instructions when the tool contract is the problem. It can upgrade the model when a clearer result shape would have solved the failure more cheaply. It can also over-correct the tool when the real issue is missing data or a bad task definition.
AX is useful when it helps us locate the change—not when it gives old work a new name.
How to Run a Surface-Focused Eval
1. Start With the User’s Job
Write tasks in outcome language.
Do not say, “Call query_events with status=failed.” Say, “Find Acme’s checkout failures from the last 24 hours and explain the dominant cause.”
Ground the tasks in real product usage and include missing identifiers, empty results, ambiguous requests, partial failures, and actions near a permission boundary.
2. Grade Outcomes and Policy First
Define what must be true at the end:
- Was the correct answer or state produced?
- Was it supported by evidence?
- Were permissions and business rules respected?
- Did the agent avoid an unauthorized side effect?
Tool-path checks can be useful, but they should be applied carefully. There may be several valid trajectories. A grader should not reject a correct strategy merely because it differed from the designer’s preferred sequence.
3. Hold the Other Variables Constant
If the question is whether a schema is better, keep the model, prompt, harness, task set, data, and budget stable.
Then compare surface variants.
Without that control, a higher score may come from a model change, a larger token budget, or an easier environment rather than the interface improvement you intended to measure.
4. Inspect the Trace
Outcome graders tell you whether the job succeeded. The trace helps explain the path.
Look for repeated wrong-tool choices, invalid arguments, unnecessary clarifications, redundant calls, oversized responses, lost identifiers, failed recovery, unsafe attempts, latency, and cost.
The final answer can look plausible even after a terrible interaction. Conversely, an unusual trajectory may still be correct. Read traces as diagnostic evidence, not as a rigid script the agent must copy.
5. Replay Across Multiple Trials
One success proves very little in a non-deterministic system.
Run the same tasks more than once, use held-out cases, and compare both average success and consistency. Keep the failures that taught you something as regressions.
A Concrete MCP Decision
Return to the original design question.
Variant A exposes one generic tool:
query_records(entity_type, filters, fields, sort, limit, cursor, include, format)
Variant B exposes two narrower tools:
list_failed_payments(account_id, since, limit?)get_payment_attempt(payment_id)
Now give both variants the same tasks, data, model, prompt, harness, and budget. Run multiple trials.
Grade whether each agent found the correct failures, explained the dominant cause with evidence, stayed read-only, and handled ambiguous accounts safely. Then inspect the traces for invalid parameters, over-fetching, retries, lost identifiers, tool-call count, latency, and token use.
If Variant B is more reliable, that is evidence that narrower tools are better for this workflow.
It is not proof that narrow tools are universally superior.
If Variant A performs just as well with fewer calls after two confusing fields are removed, keep the generic tool.
This is the important shift: stop arguing from taste and run the eval.
Agents Can Help Without Becoming the Judge
Agents can generate candidate tasks, execute trials, cluster failures, inspect transcripts, and propose tool changes. Anthropic has shown that agents can even optimize tools against a held-out evaluation set.
But the same agent that designed a tool should not be the only agent used to validate it. It knows too much about the intended path.
Use fresh contexts. Preserve held-out tasks. Verify important outcomes deterministically when possible. Calibrate model graders against human judgment. Keep product semantics, permissions, policy, and acceptable risk with people who are accountable for them.
Agents are valuable participants in the eval loop. They are not an excuse to remove evaluation discipline.
Put the Lens in the Repository
A short repository rule is enough to establish the practice:
Treat the agent-facing surface as an explicit variable in agent evals. For material tool or schema changes, define outcome and policy graders, hold the model and harness stable, compare surface variants across repeated trials, inspect traces for interface friction, and preserve useful failures as regression cases.
I revised the accompanying Agent Experience Review skill around that principle. It is not a new eval framework. It is a compact playbook for running a surface-focused agent eval and deciding whether the evidence points to the tool contract or somewhere else in the system.
The Claim I Still Believe
Any application that agents are expected to use is becoming one system with two product surfaces: one experienced by people and one experienced by agents.
The testing method for the second surface is not mysterious. It is agent evals, supported by traces, production telemetry, controlled comparisons, and human judgment.
The product mistake is allowing the agent surface to remain invisible inside those evals.
When an agent fails, we should not immediately blame the model. When it succeeds, we should not assume the interface is good. We should know which part of the system made the behavior easier, harder, safer, or more expensive.
That is the role of agent experience as a product lens.
Not a replacement for evals.
A way to aim them at a product surface we are only beginning to take seriously.