How to test an LLM agent

Why an LLM is not tested like regular code

A classic test: fixed input → expect(actual).toBe(expected). With an LLM this breaks for three reasons:

• Non-determinism. The same input gives different but equally correct answers: "Order cancelled" and "Your order was successfully cancelled". String equality does not work;
• The oracle problem. For free text there is no single "correct" answer, so it is unclear what to compare with;
• Hidden dependencies. Behaviour is changed by the model version, the prompt, the temperature, RAG data, tool results — and none of those looks like "broken code".

So you check properties, not exact text, and measure a success rate over a set of examples, not pass/fail on one:

• Deterministic checks wherever possible: valid JSON against a schema, required fields, no forbidden words, length, the right tool called with the right arguments, data changed in the DB;
• Semantic checks for free text: embedding similarity, fact checks against a reference, LLM-as-judge;
• Statistics: run a case N times and require, for example, ≥ 90 % success, or compare the average score between versions.

The pyramid stays: the most fast deterministic tests with a mocked model (the logic around the LLM: parsing, routing, error handling), fewer eval-set runs with the real model, fewest e2e.

Eval set, metrics and LLM-as-judge

An eval (evaluation) is an automated run of a set of examples through the system with scoring of the results. These are "regression tests" for LLM functionality.

A golden dataset is a curated set of cases: input, expected behaviour (not necessarily exact text), tags (category, difficulty). Where to get them: real logs and tickets (anonymised), edge cases, known bugs (every bug found becomes a case), synthetic examples generated by a model and checked by a person. The set must cover negative scenarios: refusals, ambiguous requests, attempts to bypass the rules.

LLM-as-judge: another model grades the answer against a rubric ("is the answer based only on the provided documents? 1–5", "yes/no: does the text contain personal data"). Practices that make a judge reliable:

• a clear rubric and binary or narrow scales instead of "rate the quality";
• ask for a short justification before the score;
• judge with a different model than the one under test (self-preference bias), temperature 0;
• calibrate the judge on human-labelled examples and measure agreement with humans;
• account for position and length bias: a longer answer is not necessarily better.

Metrics: pass rate per category rather than one average; refusal rate; average and p95 tokens, cost and latency per case. Every change of prompt, model or RAG is compared with a baseline on the same set.

Testing an agent: tools, trajectory, outcome

An agent is harder than a chat: it has steps and side effects. Three levels are checked:

1. A single tool: ordinary tests: contract, argument validation, errors, idempotency. No LLM needed here;
2. The model's decision: whether the right tool with the right arguments was chosen for this request (tool selection accuracy), and no extra ones were called;
3. Trajectory and outcome: the whole sequence of steps and the result in the world (DB state, a created ticket), not just the final text.

The main technique is to record the trajectory (a trace of all messages and calls) and assert on it and on the system state:

test('refund request creates exactly one refund', async () => {
  const run = await agent.run('Refund order 1042', { seed: fixtureDb });

  // path: the right tools, no extra ones
  expect(run.toolCalls.map((c) => c.name)).toEqual(['get_order', 'create_refund']);
  expect(run.toolCalls[1].args).toMatchObject({ orderId: 1042 });

  // outcome in the world, not just text
  expect(await db.refunds.count({ orderId: 1042 })).toBe(1);

  // constraints: budget and steps
  expect(run.steps).toBeLessThanOrEqual(6);
  expect(run.usage.totalTokens).toBeLessThan(8000);
});

Practices:

• Mock the outside world (tools, APIs, time) for fast and stable runs; keep a separate set with real integrations in a sandbox;
• Do not require a single path if several are correct: check the mandatory steps, forbidden actions and the final state;
• Record / replay: recorded model responses for deterministic regressions of the agent logic;
• Side effects: idempotency, human confirmation of dangerous actions (human-in-the-loop), tool permissions by least privilege;
• Multi-turn dialogue: lost context, contradicting clarifications, a change of intent;
• Failures: a tool returned an error or timeout, an empty response, an unexpected format — the agent must not loop or invent a result;
• Limits: max steps, tokens, time, cost; behaviour when a limit is reached.

Evals in CI and production monitoring

Testing an LLM system does not end at release: behaviour drifts with model updates, data changes and new kinds of user requests.

In CI:

• a fast network-free layer: unit tests of parsing, routing, schemas and error handling with a mocked model on every commit;
• the eval set with a real model on PRs that touch the prompt, model, tools or RAG (expensive and slow, so a "smoke" subset on the PR plus the full set nightly);
• a quality gate: pass rate no lower than a threshold and no worse than the baseline by more than the allowed margin; cost and latency limits;
• pinned model and prompt versions so a difference is explainable; for non-determinism allow retries and a tolerance.

In production:

• Observability: log (carefully with PII) the prompt, model version, tool calls, tokens, latency, cost; a trace per request;
• Online evaluation: a sample of real conversations is run through the judge; user feedback (👍/👎) and escalations;
• Canary release and A/B of a new model or prompt;
• Alerts: a spike in cost, refusal rate, tool errors, answer length;
• Every real incident → a new case in the golden dataset.