---
name: agent-experience-review
description: Run a surface-focused agent eval to diagnose and improve agent-facing product interfaces. Use when comparing MCP tools, function schemas, APIs, resources, context files, structured outputs, error contracts, permission boundaries, or multi-tool workflows, especially when agents select the wrong tool, supply invalid arguments, over-fetch, loop, or fail to recover.
---

# Agent Experience Review

Treat agent experience as an eval target, not a new kind of eval. Measure the whole task outcome, inspect the trace to diagnose interface friction, change one surface variable where possible, and replay.

Keep product meaning, policy, permissions, acceptable risk, and grader calibration with accountable humans.

## Collect the Inputs

Gather:

- The surface under review: tool definitions, schemas, resources, results, errors, examples, and permission behavior
- Representative jobs and verifiable outcomes
- The production-like model, prompt, harness, data, environment, and resource budget
- Safety and policy rules
- Existing traces or production failures, when available

State the hypothesis. Example: "Agents omit the time boundary because the generic query schema makes scope unclear."

## Define the Eval Before Changing the Surface

Create 5-20 tasks in user language without naming the intended tool or parameters. Include:

- Normal workflows grounded in real product use
- Missing or ambiguous identifiers
- Empty, stale, invalid, or partial results
- Requests near permission or write boundaries
- Tasks requiring evidence in the final answer

Define graders in this order:

1. **Outcome:** Was the correct answer or environment state produced?
2. **Policy:** Were permissions, business rules, and stop conditions respected?
3. **Evidence:** Is the answer supported by the tool results?
4. **Trajectory:** Did an important tool choice, argument, or handoff occur?

Do not overspecify one tool sequence when several valid strategies can produce the correct, safe outcome.

## Isolate the Surface Variable

When comparing surface variants, hold these stable where possible:

- Model and inference settings
- System prompt and agent harness
- Task set and graders
- Data and environment state
- Tool budget, token budget, and time limit

Report any variable that could not be controlled. Do not attribute an improvement to the surface when the model, prompt, data, or budget changed at the same time.

## Run Repeated, Isolated Trials

1. Start each trial from clean state and production-like context.
2. Do not reveal the intended tool or ideal call sequence.
3. Run multiple trials for important cases; one pass is not validation.
4. Preserve the full trace: calls, arguments, results, retries, recovery, and final output.
5. Keep held-out tasks for final comparison.
6. Never cross a real destructive or external-write boundary. Use read-only access, mocks, dry runs, or a safe test environment.

## Grade Outcomes, Then Diagnose Traces

Grade the final answer and environment state first. Then inspect the trace for:

- **Discovery:** The relevant capability was not found or understood.
- **Selection:** Overlapping tools were confused.
- **Contract:** Arguments, defaults, or scope were unclear.
- **State:** Results lacked stable identifiers or context for the next step.
- **Recovery:** Errors or partial results did not support a safe retry.
- **Safety:** Side effects, permissions, confirmation, or stop conditions were unclear.
- **Efficiency:** The surface caused redundant calls, over-fetching, latency, or context waste.

Treat trajectory signals as diagnostic evidence. An unusual path is not wrong if it reaches the correct, safe outcome.

## Change the Lowest Responsible Layer

Consider changes in this order:

1. Product capability or workflow boundary
2. Tool name, description, scope, or separation
3. Input or output schema
4. Error and recovery contract
5. Permissions or confirmation behavior
6. Context and examples
7. Prompt, model, or harness

This order is a diagnostic checklist, not a rule that the surface must be at fault. Change the smallest layer supported by repeated evidence.

## Replay and Report

Run the same tasks against the revised surface and compare:

| Task | Outcome before | Outcome after | Trace evidence | Surface change | Confounds or risk |
|---|---|---|---|---|---|
| User-language job | Score and state | Score and state | Calls, errors, retries, cost | Isolated interface change | Unknowns |

Report:

- **Setup:** Model, harness, tasks, graders, environment, trials, and budgets
- **Result:** Outcome, policy, evidence, consistency, latency, token use, calls, and errors
- **Diagnosis:** Surface friction supported by traces
- **Keep:** Changes supported by replay and held-out evidence
- **Reconsider:** Changes that moved or added friction
- **Regressions:** Cases to preserve for future surface changes
- **Human decisions:** Unresolved semantics, policy, permissions, risk, or grader calibration

Agents may generate candidate tasks, analyze traces, and propose changes. Do not use the designing agent's self-reported preference as ground truth, and do not claim validation from a single successful run.