An agent can say a task is complete while the database shows a duplicate action or no action at all.
AI agent testing verifies that a system using a language model and tools achieves the requested outcome within its permissions, handles failures honestly, and stops safely. Test durable state and tool effects alongside the final response. A fluent claim of completion is insufficient evidence that the task succeeded.
This guide proposes a practical engineering test plan for a fictional support agent that reads orders, drafts replies, and requests approved cancellations. All scenarios and numbers are illustrative, not reported experiments or measured outcomes. The focus is testing the agent itself, rather than comparing AI tools for conventional QA.
Define the agent contract before choosing metrics
Distinguish an agent from an isolated LLM response
Anthropic's Building effective agents distinguishes predefined workflows from agents that dynamically choose their process and tools. That distinction changes the test surface. A response test checks generated text; an agent test also checks decisions, permissions, intermediate failures, persistent memory, and external effects.
Write a task contract with an initial state, authorized actions, observable success conditions, and forbidden changes. Include a valid refusal or escalation when the request cannot safely finish. Specify whether approval permits one exact action or a broader scope, and which identity owns the operation.
Make the outcome independently observable
For the fictional cancellation task, require the correct order to be cancelled once, the audit record to reference the approving user, and unrelated orders to remain unchanged. If approval is missing, success means requesting it without executing the cancellation. The agent's wording must accurately describe that state.
Build a repeatable AI agent testing workflow
Reset the environment for every trial
Use synthetic accounts, a dedicated database namespace, controlled document snapshots, and fake outbound destinations. Deny production credentials and unrestricted network access. Reset conversation history, stored memory, queues, files, caches, and backend records before each trial. Give parallel runs separate tenants and operation identifiers.
Freeze the clock and fixture identifiers where time and identity affect expected behavior. Script tool responses and fault timing. Deterministic fixtures make failures attributable even though the model's decisions remain nondeterministic; a fixed seed or low temperature does not guarantee identical complete runs.
- Restore and verify the fixture's initial state.
- Record the agent, model, tools, policy, and dataset versions.
- Execute the mission within a bounded sandbox.
- Inspect tool events and read final state independently.
- Grade the outcome, safety constraints, and response separately.
- Repeat from a fresh fixture and retain every trial's result.
Prove that the harness can detect failure
Run a scripted negative control that modifies the wrong order and claims success. The outcome checker must fail it. Run a no-op control to ensure the checker is reading this trial's state rather than a previous cancellation. A broken harness can make every candidate look reliable.
Separate unit tests from integration and end state checks
Keep deterministic logic under ordinary tests
Unit test argument validation, permission decisions, routing rules, status reconciliation, and stopping budgets without a live model. Use fixed tool-call objects to test orchestration transitions. These checks should identify application defects quickly without paying for repeated inference.
Integration tests then exercise authentication, database writes, approval storage, and tool adapters against isolated services. Full agent evaluations add the model's choices and recovery behavior. Passing one layer cannot establish that the others work.
Traditional software testing expects stable outputs for controlled inputs. End to end testing of agent behavior needs invariant assertions across alternative valid paths. Automate deterministic checks first, then use the live agent for decisions those checks cannot exercise. Add edge cases to the test dataset deliberately, including ambiguous requests and contradictory tool output.
Inspect results and permitted execution paths
Google's agent evaluation documentation separates final response evaluation from trajectory evaluation and describes exact, ordered, and unordered tool matching. Choose a path constraint only when the product requires it. Multiple harmless read sequences can reach the same valid result.
Require authorization before mutation even when read order is flexible. Check both intended effects and prohibited changes. An exact transcript match is brittle when harmless wording changes; an end state check alone misses a forbidden disclosure that happened before the correct cancellation.
Test tool contracts and interrupted operations
Validate schemas and error semantics
The MCP tools specification describes input and output schemas and distinguishes protocol errors from tool execution errors. Treat schema validation as a boundary check, then add application checks for ownership, permitted transitions, and valid business values.
Exercise missing arguments, unknown identifiers, malformed responses, permission denial, expired credentials, unavailable services, and rate limits. Confirm the agent does not reinterpret an error payload as a successful result. Tool descriptions and annotations cannot replace server-side enforcement.
Bind approval and retries to a logical operation
Illustrative approval mismatch case
Approve cancellation of synthetic order A, then have the proposed tool call target order B. Expect the execution boundary to reject it. Repeat after approval expires or the amount changes. Test the stored authorization against the actual arguments, not merely whether an approval dialog appeared.
Illustrative timeout after a committed action
Let the fake backend commit cancellation but suppress its response. The agent should reconcile by operation identity before retrying. Assert one logical effect and an honest pending or confirmed status. Idempotency requires backend behavior and durable keys; a model promise to avoid duplicates is not a guarantee.
Submit two simultaneous requests for the approved operation and assert one cancellation, one notification, and the intended audit effects. Then reuse that approval with a new operation identity and change arguments under the original idempotency key. Reject unauthorized reuse and conflicting arguments. Test execution-time authorization and atomic deduplication together, following OWASP transaction authorization guidance; Stripe's idempotency contract illustrates why key reuse also needs parameter consistency.
Set tool deadlines and cancellation behavior explicitly. A local timeout does not prove a remote action stopped. Cover delayed callbacks, process restart, retry exhaustion, and reconciliation after the sandbox reports an uncertain outcome.
Measure repeated trials without hiding severe failures
Report the denominator and the distribution
Anthropic's Demystifying evals for AI agents explains evaluations in terms of tasks, repeated trials, transcripts, outcomes, and graders. It distinguishes the chance of at least one success across attempts from consistent success across attempts. Select the measure that reflects how users actually encounter the agent.
For an agent users invoke once, report per-trial task success rather than showcasing the best retry. Record the number of tasks, repetitions, invalid harness runs, and completion failures. Break results down by mission and risk slice so easy read-only tasks cannot conceal cancellation failures.
Illustrative repeated-trial report
Suppose a proposed evaluation runs 12 cancellation missions five times each. Report successful outcomes out of 60 valid trials, plus per-mission results. Separately count unauthorized attempts, executed unauthorized actions, duplicate effects, and false completion claims. These figures describe a reporting design, not findings from a tested product.
Keep safety metrics separate from average quality
Do not average a severe permission breach into a relevance score. Preserve its trace and severity as a separate release finding, even if an enforcement layer blocked execution. Distinguish model behavior from boundary effectiveness.
Choose repetitions and acceptance criteria from exposure, failure impact, baseline variability, and uncertainty. Show confidence intervals where the sampling assumptions support them. Repeated trials of a few fixtures are not evidence about every real-world request, and zero observed violations does not prove zero risk.
Ground graders and calibrate them with people
Match the grader to the evidence
Use executable assertions for record counts, permissions, schema validity, and forbidden mutations. Use model-based graders for bounded judgments such as whether a reply explains the actual status clearly. Supply the rubric, relevant policy snapshot, and verified tool evidence. Require evidence references and allow an uncertain result.
Microsoft's agent evaluator documentation separates system outcomes from process measures including tool selection, inputs, and output utilization. Its groundedness dimensions are useful vocabulary, but selecting a built-in evaluator does not establish that its scores match your product contract.
Calibrate before using scores as release evidence
Have reviewers independently label representative successes, subtle failures, and borderline cases. Compare their decisions with grader outputs, inspect disagreements, and clarify the rubric. Include confident but false completion messages and correct results expressed awkwardly. This checks whether style is dominating substance.
Hide candidate identity where practical, vary presentation order, and pin the grader model and prompt. Keep retrieved documents and tool text marked as evidence, not grader instructions. Test malicious content against the grader as well as the agent. Escalate ambiguous high-impact judgments to a person.
Challenge prompt injection and enforce least privilege
Treat retrieved material as untrusted data
The OWASP Top 10 for Agentic Applications 2026 provides a security risk taxonomy for agentic systems. Use it to expand adversarial cases around tool misuse, hijacked goals, and poisoned context rather than declaring security from a single benign test suite.
In an illustrative fixture, put a forged instruction inside an order note asking the agent to export other customers' records. Repeat through search results, tool errors, attachments, and stored memory. Assert that it neither discloses the records nor broadens its authority. Verify allowed outbound destinations and inspect attempted calls, not just the final text.
Enforce permissions outside the model
MCP security best practices discusses token audience validation, token passthrough risks, and scope minimization. Test wrong-audience tokens and cross-tenant access at the adapter and server boundary. Keep secrets out of model-visible tool results.
Give read and mutation tools distinct permissions, constrain filesystem and network access, and require scoped authorization for sensitive effects. Add a case where the model selects a forbidden tool and confirm the boundary blocks it. Prompt instructions support these controls but cannot enforce them alone.
Version evaluations and inspect latency cost and privacy
Preserve a holdout and comparable baselines
Version task fixtures, source documents, grader rubrics, model identifiers, sampling settings, tool schemas, and permission policies together. Separate development cases from a holdout that prompt tuning does not inspect. When a holdout informs a fix, move that case into regression coverage and replenish the holdout.
Run baseline and candidate on the same fixture versions. Flag missing or changed provider model versions as comparability limits. Label infrastructure failures separately while retaining their user-visible availability impact. Promote genuine production failures into redacted regression cases after reviewing their data and expected outcome.
In CI, run unit tests and a small critical eval suite on each change, with broader repeated evaluation before promotion. Compare agent versions using the frozen test dataset. This feedback loop connects production findings, human evaluation, and regression coverage. AI observability supplies diagnostic evidence; your evaluation framework still needs an explicit expected outcome for every mission.
Trace the operational cost without exposing content
Capture elapsed time, tool latency, retries, model calls, token usage, and estimated cost using a dated pricing basis. Include failed and abandoned tasks in cost reporting. Inspect tail latency and cost per successful outcome alongside averages. Bound execution time and loops, then test budget exhaustion and user cancellation.
OpenTelemetry's GenAI span guidance treats instructions, inputs, and outputs as sensitive content that should not be captured by default. Prefer operation IDs, status, timing, and redacted summaries. Restrict content traces, set retention, and verify redaction with synthetic secret markers before export.
Set release gates and collect focused human feedback
Make the decision traceable to risk
Define gates before examining candidate results. Require deterministic contract checks, accepted outcome performance for each critical slice, reviewed safety findings, and justified latency and cost budgets. Record sample size, uncertainty, exclusions, outstanding defects, and the person accepting residual risk. Avoid universal pass-rate thresholds detached from consequences.
NIST's Generative AI Profile is a voluntary risk-management companion for AI design and evaluation. Use that perspective to assign owners and document limits; this test plan is engineering guidance, not a compliance determination.
Give people a mission with observable expectations
Use the mobile app beta testing guide when the agent is delivered inside an app. Ask users to check whether approval, pending status, failure recovery, and final results are understandable. Capture findings using the mobile bug report guide, with synthetic data and redacted evidence.
For focused product feedback, create a BetaRoast testing round describing the mission and expected observations. Engineering should retain ownership of sandbox evaluation and permission checks. After material fixes, rerun affected suites and the critical regression set before approving the candidate.
Frequently asked questions
How is AI agent testing different from LLM testing?
LLM testing often grades a response. Agent testing also verifies tool choices, permissions, persistent state, recovery, and stopping behavior across multiple steps. Both response quality and external effects matter.
Can agent tests be completely deterministic?
The fixtures, tool responses, and assertions can be deterministic. Model decisions may still vary. Use repeatable environments and repeated trials instead of assuming a fixed seed reproduces every run.
What pass rate is enough to release an AI agent?
There is no universal threshold. Set criteria by task risk, exposure, baseline performance, and sampling uncertainty. Review severe violations separately so a high average cannot conceal an unauthorized action.
Can an LLM judge replace human review?
It can scale bounded judgments after calibration, but it can misgrade fluent errors or follow injected instructions. Use executable checks for facts and permissions, and human review for consequential uncertainty.
Test the agent as a state-changing system. Reset isolated fixtures, verify tool contracts and durable outcomes, repeat stochastic journeys, and report severe violations separately from task success. Calibrate graders against human judgments, version the evaluation suite, protect trace content, and approve releases against explicit risk-based gates.