Agent Evals and Observability: Trajectory-Level, Not Turn-Level

Every Turn Passed. The Task Still Failed.

Here is the shape of the failure this article exists for.

Your agent runs BENCH-1: a 40-file TypeScript service calls the deprecated db.queryRaw() in seven places, and every call site has to move to the parameterized db.query(). The agent runs. Every model call returns well-formed tool arguments. Every edit_file call touches a real file. Every assertion in your single-call eval suite is green: valid tool schema, no hallucinated file path, no leaked system prompt, output parses.

The task fails.

It fails because the agent migrated four call sites, then spent nine steps reading files it had already read, then hit its step ceiling and stopped. Or because it migrated all seven, and also rewrote a test to match the new behavior instead of fixing the behavior to match the test — green suite, wrong reason. Or because a grep call transiently failed at step three, the agent silently narrowed its search to the two files it already had open, and shipped a migration that looked complete and covered two of seven.

None of those are turn-level defects. Every turn was individually correct. The defect is in the trajectory: wasted steps, a wrong-but-recovered path that quietly lost scope on the way back, a terminal state reached for the wrong reason. Turn-level evals are structurally blind to all three, because a turn-level eval has no concept of a previous turn.

I want to be straight about where this argument comes from. I have paid for the observability half of it: a forgotten Lambda in an invocation loop burned $5,350 across three weeks with no alarm and no ceiling, and I found out from an invoice. The eval half I am arguing from mechanism, not from a scar — I do not have an incident where a step-level eval gap let an agent regress silently in production, and I am not going to invent one. The mechanism above is enough to act on. A turn-level assertion cannot observe a property that only exists across turns. That is not an empirical claim; it is what the assertion's inputs are.

Three Altitudes, Three Different Bills

Evals for agents run at three altitudes. Most teams have built the first one and stopped.

The single-call layer underneath all of this — assertion, LLM-as-judge, human review, and how to sequence them — is already covered in Evals for LLM Features: Building the Regression Net. Build that first. It is the cheapest layer and it catches a surprising share of real failures. This article is strictly the layer above it, and it assumes you have the one below.

What is trajectory evaluation for AI agents? Trajectory evaluation scores an agent's whole run rather than its individual turns. It asserts five properties: the terminal state is correct, the step count stays near a reference path, no forbidden action was taken, the agent recovered sanely from an injected tool failure, and the pass rate across repeated runs stays inside a declared band.

What a Trajectory Eval Actually Asserts

This is the differentiated part, so here it is concretely. Five assertions, in the order they should run — cheapest and most objective first.

1. Terminal state correctness. The only one turn-level evals partly cover. For BENCH-1: suite green, zero new queryRaw call sites, all seven originals gone.

2. Step efficiency against a reference path. You need a reference trajectory — the shortest correct path a competent human or a well-instrumented run took — and a tolerance band above it. Without this, waste is invisible; a run that takes 30 steps to do 11 steps of work looks identical to a good run in every terminal check you have.

3. No forbidden action taken. A deny list checked against every tool input across the whole run: git push, npm publish, rm -rf, anything that writes outside the working tree. Right answer via a forbidden road is a failed run, not a passed one.

4. Recovery behavior after an injected tool failure. Replay the task with a deliberate failure — return an error from the third grep — and assert on what the agent does next. Did it notice? Did it retry the identical call more than once? Did it silently narrow scope to what it already had, which is the failure mode from the opening? This one needs a judge; the answer is not expressible as a string match.

5. Determinism band. A pass rate over N runs, with a floor. Covered in the next section, because it deserves it.

// github-repo/agent-engineering/05-evals-otel/trajectory.spec.ts
import type { Trajectory } from "../bench/types";
import { judgeRecovery, type Verdict } from "./judge";

const FORBIDDEN = ["git push", "npm publish", "rm -rf", "> /dev/"];
const REFERENCE_STEPS = 11; // measured on the reference path in bench/

export async function assertTrajectory(t: Trajectory): Promise<Verdict[]> {
  return [
    // 1. terminal state — the only one a turn-level suite partly covers
    { name: "terminal", pass: t.suiteGreen && t.newQueryRawSites === 0,
      detail: `green=${t.suiteGreen} newRaw=${t.newQueryRawSites}` },

    // 2. waste is invisible unless you compare against a reference path
    { name: "efficiency", pass: t.steps.length <= REFERENCE_STEPS * 1.5,
      detail: `${t.steps.length} steps vs ${REFERENCE_STEPS} reference` },

    // 3. right answer via a forbidden road is a failed run
    { name: "no-forbidden",
      pass: !t.steps.some((s) => FORBIDDEN.some((f) => s.toolInput.includes(f))),
      detail: `${t.steps.length} tool inputs checked against deny list` },

    // 4. subjective — needs a judge, see judge.ts
    await judgeRecovery(t.injectedFailureReplay),

    // 5. one run is not a measurement; the caller aggregates N
    { name: "determinism", pass: t.passRateOverN >= 0.9,
      detail: `${t.passRateOverN} over ${t.runCount} runs` },
  ];
}

The judge is the only place a model appears in the eval itself, and it gets a sharp rubric rather than a vague quality score:

// github-repo/agent-engineering/05-evals-otel/judge.ts
import Anthropic from "@anthropic-ai/sdk";

export interface Verdict { name: string; pass: boolean; detail: string }

const client = new Anthropic();

const RUBRIC = `Score an agent's recovery after one tool call returned an error.
PASS only if all three hold: (a) the agent acknowledged the failure in its next
action, (b) it did not repeat the identical failing call more than once, (c) it
did not silently reduce the scope of the task. Reply with JSON only:
{"pass": boolean, "detail": string}`;

export async function judgeRecovery(replay: string): Promise<Verdict> {
  const res = await client.messages.create({
    model: "claude-opus-5",
    max_tokens: 1024,
    system: RUBRIC,
    messages: [{ role: "user", content: replay }],
  });
  const block = res.content.find((b): b is Anthropic.TextBlock => b.type === "text");
  const parsed = JSON.parse(block?.text ?? "{}") as Partial<Verdict>;
  return { name: "recovery", pass: parsed.pass === true, detail: parsed.detail ?? "no detail" };
}

Judge on claude-opus-5, agent loop on claude-sonnet-5. Keep them different models, and validate the judge against your own ratings on a handful of replays before you trust its verdicts in CI. An unvalidated judge is one more non-deterministic dependency wearing a lab coat.

Pass Rate Is the Unit, Not Pass/Fail

A single run of a non-deterministic system is an anecdote. If your CI gate is "the agent eval passed," you have built a coin-flip detector.

The practical rule I would hold a team to: 20 runs for a release gate, 5 for a pull request, and never fewer than 3 for any number you intend to quote. Twenty is enough to see a distribution and cheap enough to run nightly on one benchmark task. Five catches the obvious cliff on a PR without making the pipeline unusable. Below three you are not measuring, you are sampling.

Then read the distribution, not the boolean. Three shapes matter:

• The floor drops. 19/20 becomes 14/20. Something is broken. Bisect the change.
• The floor holds, the median step count rises. Nothing failed, and you just got more expensive. This is the most commonly missed regression in agent work, because every gate is still green while the bill moves.
• The band widens. Same pass rate, much higher variance across runs. Usually a symptom of a loop whose stop condition depends on model judgment rather than a measured condition — the territory covered in Loop Engineering.

Store the band, not the last result. A pass rate is only interpretable against the pass rate it replaced.

Observability: The gen_ai Spans, and Why They Are Not Settled

The eval tells you the run failed. The trace tells you where. For agents, the vendor-neutral answer is the OpenTelemetry GenAI semantic conventions, which define spans for the four things an agent run is actually made of: the agent invocation, workflows inside it, tool executions, and model calls, plus latency and token-usage metrics.

Now the part most coverage skips. These conventions are not stable. v1.41 defined the agent, workflow, tool, and model spans along with the latency and token metrics. With the v1.42.0 release on 12 June 2026, the gen_ai.* attributes and spans moved out into a dedicated GenAI conventions repository — and they remain pre-stable and experimental, with no 1.0. Attribute names in this namespace have already been renamed once. Plan for them to be renamed again.

That is not a reason to skip instrumentation. It is a reason to pin the convention version in your collector config and your instrumentation library, put the attribute names behind one module in your codebase rather than sprinkled across every span call, and treat a semconv bump as a real migration with a real diff.

Here is the span hierarchy for one BENCH-1 run:

The attributes worth setting, and why:

• gen_ai.operation.name — invoke_agent, chat, execute_tool. This is what makes a trace queryable by shape.
• gen_ai.provider.name, gen_ai.request.model, gen_ai.response.model — requested and served model can differ. Record both or you cannot explain a behavior change after a routing decision.
• gen_ai.usage.input_tokens, gen_ai.usage.output_tokens — on every model span, not just the total. Per-step token attribution is the only way to find which step in the loop got expensive.
• gen_ai.tool.name, gen_ai.tool.call.id — correlates a tool span to the model turn that requested it.
• gen_ai.agent.name, gen_ai.agent.id — mandatory the moment you have more than one agent, which is exactly when Graph Engineering starts applying.
• Your own run.id and step index on the root span — so a trace can be joined to an eval verdict.

Same Discipline, One New Signal, One New Trap

If this feels familiar, it should. Span hierarchies, tail sampling, cardinality budgets, retention tiers — this is the discipline in the Production Observability with OpenTelemetry pathway, applied to a loop instead of a request. Almost nothing about agent observability is new, which is the pathway's whole argument and also the argument in multi-agent architecture as premature optimization: most of this is distributed-systems work you already know how to do.

Two things are genuinely different.

One new signal: tokens. They behave like a resource metric, not a latency metric. They compound across hops, they are billed, and they are the only signal in the trace with a dollar sign attached. Put them on every model span and aggregate to the root.

One new cardinality trap: never put prompt content in an indexed attribute. Prompts and completions are unbounded, high-entropy text. As a span attribute in an indexed store, they are the worst version of the user_id-as-metric-label mistake — plus a data-governance problem, because that store now holds whatever the user typed. Keep prompt and completion bodies out of indexed attributes. If you need them for debugging, put them in the event/log body with trace correlation, sample them, and set a retention that expires.

The Metric That Makes All of It Add Up

This is where the pathway's numbers stop being trivia.

Cost-per-verified-unit-of-work is total cost of an agent run divided by units of work that passed verification. Not units generated. An agent that produces ten pull requests and merges none has a cost-per-verified-unit-of-work of infinity, which is the correct number.

For most teams that metric stays rhetorical, because they can compute neither half. The trajectory eval and the token spans are precisely what make it computable: the eval supplies the denominator by defining what "verified" means and returning a pass rate rather than a vibe, and the gen_ai.usage.* attributes rolled up to the root span supply the numerator per run, per configuration, per layer.

And B-02 lands here exactly. A system's real cost is the code you forget is running. An agent loop is the purest example of that sentence I have encountered: it runs on its own schedule, retries without being asked, re-pays for context on every hop, and reports nothing unless you built the thing that reports. The forgotten Lambda had no ceiling and no alarm. An uninstrumented agent loop has no ceiling, no alarm, and a variable step count. The invoice is still the discovery mechanism unless you replace it on purpose.

Closing the Table

Seven articles, one task, one table. This is the configuration that stacks every layer in the pathway — harness, context budget, tool surface, loop controls, graph, and now evals plus spans — against the single-call baseline from article one.

What the comparison will show, and why: the all-layers configuration should not win on median tokens. Instrumentation, loop guards, and a graph all cost tokens and wall clock. It should win on pass rate and on the p95, because that is what each layer was bought for — bounding the tail and raising the floor. If the fully-layered configuration wins on tokens too, one of the earlier layers was removing waste rather than adding safety, and the table will say which. If it wins on nothing, the layers were not earned, and the table will say that instead. Either result is worth more than a table that only ever confirms the thesis. Every layer's per-directory numbers live in github-repo/agent-engineering/README.md.

Monday morning: pick the one agent you have in production, count how many runs your eval gate uses, and if the answer is one, raise it to five and record the pass rate. That single change turns a coin-flip detector into a measurement. Everything else in this article is built on top of it.

FAQ

What is trajectory evaluation for AI agents?

Trajectory evaluation scores an agent's entire run instead of its individual turns. It asserts terminal-state correctness, step efficiency against a reference path, absence of forbidden actions, sane recovery after an injected tool failure, and a pass rate inside a declared band across repeated runs. Turn-level assertions cannot see any property that only exists across turns.

Are the OpenTelemetry GenAI semantic conventions stable?

No. v1.41 defined agent, workflow, tool, and model spans plus latency and token-usage metrics, and the v1.42.0 release on 12 June 2026 moved the gen_ai.* attributes and spans into a dedicated GenAI conventions repository. They remain pre-stable and experimental with no 1.0. Instrument anyway, pin the version, and isolate attribute names behind one module.

How many runs do you need to evaluate an agent?

Twenty for a release gate, five for a pull request, never fewer than three for a number you intend to quote. One run of a non-deterministic system is an anecdote, not a measurement. Then read the distribution rather than the boolean: a dropping floor, a rising median step count, and a widening variance band are three different regressions.

What should you trace in an agent?

Four span types — agent invocation, workflow, tool execution, model call — with gen_ai.operation.name, requested and served model, and gen_ai.usage.input_tokens and output_tokens on every model span rather than only the total. Add your own run id and step index so a trace joins to an eval verdict. Keep prompt content out of indexed attributes.

How do you measure whether an agent is worth its cost?

Cost-per-verified-unit-of-work: total run cost divided by units that passed verification, never units generated. The trajectory eval supplies the denominator by defining what verified means; token attributes rolled up to the root span supply the numerator. Without both halves instrumented, the metric stays rhetorical and the monthly invoice remains your discovery mechanism.