AI Agent Evaluation

Write self-contained pass criteria that state both the required behaviour and the precise failure condition.

Select Exact, Regex, or LLM-as-judge matching for a case based on whether wording, patterns, or meaning determines correctness.

Assign deterministic, semantic, and behavioural graders to the failures each evaluation case is intended to detect.

Use human and model verdicts, TPR, and TNR to distinguish judge-criteria problems from genuine agent failures.

Classify each agent requirement as searchable knowledge, a live system value, a calculation or table lookup, or a standing prompt rule.

Design a retrieval source with clear ownership, scope, exclusions, indexing readiness, and a precise search-triggering description.

Review a candidate knowledge document for unsafe content, stale material, fragmented answers, and data that should instead come from an integration.

Map live values and actions to narrowly scoped REST or MCP tools that can be reached in every deployment environment.

Place each requirement in exactly one appropriate agent component: instructions, tool, knowledge, skill, memory, guardrail, or model.

Write a system prompt using the prescribed section order, including scope, hard limits, fallback behaviour, workflow, tool policy, and output format.

Write non-overlapping tool descriptions that state when to use each tool, when not to use it, and which inputs are required.

Design tool responses and failure messages that return only the fields and next-step guidance the agent needs.

Create an ordered skill with explicit preconditions, confirmation points, tool-call limits, and failure handling.

Use preview traces to identify routing, requirement-placement, and invented-parameter errors before publishing the agent to a sandbox.

Interpret completion, usage, tool-call, and latency metrics within a clearly stated reporting period and baseline.

Calculate and explain p50, p90, and p99 latency, including when low run volume makes percentile conclusions unreliable.

Convert observed latency and AI-action baselines into evaluation graders that detect performance and cost regressions before release.

Classify each business risk by whether it belongs in the prompt or requires PII protection, human review, tool confirmation, or a usage budget.

Choose Disabled, Observe, or Enforce mode for a governance policy using false-match evidence, operational readiness, and incident impact.

Configure failure behaviour so the agent recovers safely from blocked, refused, or timed-out approvals and tool calls.

Present and justify the finished agent as a five-layer release loop linking eval evidence, grounded data, agent structure, observability baselines, and governance controls.