What Agent Engineering Actually Is

The Two Failure Modes That Showed Up First

Mid-2025. I made a call that looked premature to most of the engineers around me: put AI-assisted development into real products, not a sandbox. Ask mode first. Then agent mode. Then agentic features shipping behind decisions I had to sign. The room averaged around eight years of experience and had watched a few hype cycles die, so credibility came from production, not slides.

Two failure modes showed up before anything else.

The first was context loss. A session that was sharp for twenty minutes would quietly stop honoring a constraint stated at the start. Not refuse it. Forget it. It would reintroduce the exact pattern it had removed an hour earlier, because the instruction forbidding it had scrolled out of what the model was actually attending to. Nothing errored. The work drifted back toward the mean.

The second was confident wrong answers. Not refusals, not exceptions, not malformed output. A plausible answer in the same register as every correct answer, with no signal separating the two. That one is worse, because it moves the cost instead of removing it: generation got cheap, and review got expensive by roughly the same amount.

Neither failure mode is a prompt problem, which took me longer to accept than it should have. Context loss is a budgeting problem in the code that assembles the request. Confident wrongness is a verification problem in the code that consumes the response. Both live in the software around the model.

What Agent Engineering Actually Is

Agent engineering is the practice of designing the software layer around a model — the harness — rather than the prompts inside it. It covers four surfaces: the harness (state, tools, permissions), the loop (stop conditions, retries, progress), the graph (who talks to whom), and the evals that prove any of it works.

The model is a component. The engineering is everything holding it. That definition is deliberately unglamorous, and it should tell you where the leverage is: swap claude-sonnet-5 for a different frontier model tomorrow and your harness decisions survive. Swap the harness and everything changes.

80% Renamed, 20% Genuinely New

Read ten agent-engineering posts and you will find orchestration, retries, idempotency, structured logging, schema design, and timeouts. All correct. None of it new. It is the same discipline you would apply to any unreliable remote dependency, which is what a model is.

So be precise about the 20% that is new, because that is where the unpriced risk lives:

Non-determinism has a price, and almost nobody is pricing it.

Not "models are non-deterministic" — everyone says that, as a quirk to apologize for. The engineering fact is that non-determinism converts into three measurable line items: tokens you re-pay on every hop, wall-clock time you cannot bound in advance, and a pass rate that is a distribution rather than a boolean. A system of deterministic components has failures you reproduce. A system built around a planner that samples has failures you characterize statistically, and characterizing costs money every time.

Illustrative arithmetic, inputs shown: a six-step loop carrying a 12,000-token conversation forward re-sends that conversation every step. Roughly 72,000 input tokens to produce a forty-line diff. Nothing failed. That is the happy path price of a loop, and most designs never write it down. Add a retry and you have paid for the context twice, for a plan that may not resemble the one you retried.

Same argument I make about org-level spend in AI cost accountability, pushed down to a single request.

The Four Surfaces

Read that bottom-up. Each surface is a place where you either spend the non-determinism price deliberately or absorb it by accident.

The harness is the state contract, the tool contract, and the permission contract: what the model can see, what it can call, and what it may do without asking. Highest behavioral leverage, least written about it, because it is plumbing rather than an architecture diagram. Two teams running the same model with different harnesses do not get slightly different results; they get different products. Agent harness design builds one in about 200 lines with no framework, to show how little of this is framework work.

Context is the harness's hardest sub-problem, and it deserves its own treatment because the failure is counterintuitive: models degrade well before the window fills. Treat context as a size limit and you will keep stuffing it until quality quietly falls off. Treat it as a depleting budget — spent by every tool result, every retry, every hop — and compaction and handoff artifacts stop being tricks and become capacity planning. That is context engineering for long-running agents, and it is where my first failure mode, context loss, gets engineered away rather than prompted around.

The loop is stop conditions, retry budgets, and progress detection. An agent without a stop condition is a while (true) that can spend money. The hard part is not the max-iteration cap; it is detecting no progress — an agent that is still producing well-formed tool calls while going nowhere, which has no error code and trips no alarm. Loop engineering covers budget guards, circuit breakers, and goal-drift detection.

The graph is topology: how many agents, what state they share, who hands off to whom. It is also the surface every tutorial jumps to first, and the one most systems should reach for last. Graph engineering argues how to build one when it is earned — and disambiguates the term from knowledge graphs, which is a live collision in search results.

The evals are the only surface that can tell you whether the other three helped. Turn-level evals will pass while the trajectory fails, which is the specific reason "we have logging" does not mean you can debug an agent. Agent evals and observability works at trajectory level and bridges into the OpenTelemetry GenAI conventions — which, worth saying plainly, are still pre-stable at v1.42.0 (12 June 2026) despite how settled everyone writes about them.

One more surface you drive but do not own: the tool interface. Tool and MCP design covers why twelve tools can perform worse than three with identical capability, and why I was wrong about MCP being a context pipe.

The Four Costs

Before any of that, there is a bill. I derived it for multi-agent systems in multi-agent architecture is premature optimization, and I am not re-deriving it here — read that post, because the same four costs apply to every surface above, not just to the graph:

1. State handoff — what gets lost between steps.
2. Failure-mode multiplication — n components, more than n failure modes.
3. Non-deterministic debugging — the repro is a distribution, not a case.
4. Context and cost compounding — every hop re-pays for context.

Every article in this pathway asks the same question of its layer: which of these four does this cost me, and what does it buy that pays for them?

What Is Genuinely New vs. What Got Renamed

This is the table to keep. Left column: your existing toolkit, which transfers almost intact. Right column: what changes about it when the caller is a model, plus the three items with no prior art at all.

If your agent design document addresses the left column and not the right, you have written a distributed-systems design document. That is not worthless. It is just not finished.

BENCH-1: The Task That Runs Through All Seven Articles

Claims about cost are cheap unless the same task is priced at every layer. So one benchmark runs through this entire pathway.

BENCH-1. A 40-file TypeScript service calls a deprecated internal helper db.queryRaw() in seven places. Migrate every call site to the parameterized db.query() equivalent, introduce no new queryRaw call sites, and leave the test suite green.

Scored per run: input tokens, output tokens, wall-clock seconds, pass/fail. Reported over 20 runs as pass rate, median tokens, and p95 wall clock. Twenty runs, not one, because a single green run tells you nothing about a sampled planner — that is cost 3 from the list above, made operational.

The baseline is the configuration with no agent engineering in it at all: one model call, no tools, no loop.

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

// BENCH-1 baseline: one shot, no tools, no loop, no verification.
export async function baseline(repoDigest: string): Promise<string> {
  const res = await client.messages.create({
    model: "claude-sonnet-5",
    max_tokens: 8000,
    messages: [
      {
        role: "user",
        content:
          `Migrate every db.queryRaw() call site to the parameterized ` +
          `db.query() equivalent. Return a unified diff only.\n\n${repoDigest}`,
      },
    ],
  });

  const block = res.content[0];
  return block.type === "text" ? block.text : "";
}

Note what the baseline cannot do, structurally. It cannot read a file it was not handed, so repoDigest has to carry the whole relevant surface up front, and a 40-file service does not fit cleanly. It cannot run the test suite, so it cannot know whether it succeeded. Its output is a diff a human has to verify entirely.

That is what the pathway table will show: the baseline is not primarily a quality failure, it is a verification failure. Every later configuration spends tokens to move verification from the human to the harness. So the comparison worth watching is not which config is cheapest — it is which config's pass rate justifies its token bill. Cost per verified unit of work, not cost per attempt.

Where Judgment Enters

The frameworks are commodity, and more so next quarter. Anyone can wire a graph; the libraries are good and improving, and the tutorial teaching you the topology is free.

What is not commodity is the set of decisions no library makes for you. How much context this step deserves. Which tool the model may call without a human in the path. What "no progress" means for your task, in your domain, at what iteration. Whether this workflow needs an agent at all, or whether you are about to pay four costs for a switch statement with better marketing. Every one of those is a judgment call with a price attached, and getting it wrong does not throw.

This is the thing I got right by accident in 2025 and only understood later. AI did not make judgment obsolete. It made judgment the only scarce skill. Generation went to near-zero cost, so the bottleneck moved to deciding what is worth building and reviewing what the machine produced. The engineers who won with AI were not the fastest adopters — they were the ones who worked out the review ratio before they scaled the generation.

Agent engineering is what that review ratio looks like encoded in software instead of in a person's attention. Everything after this article is a surface where you either encode it or pay for not having.

Monday morning: open your agent design doc and find the four costs. If they are not written down with a number next to each, you do not have a design — you have a topology.

FAQ

What is an agent harness?

An agent harness is the code around a model that assembles its context, exposes its tools, enforces its permissions, and decides when it stops. It is not the prompt and not the framework. Two teams using the same model and the same library with different harnesses ship measurably different products, because the harness determines what the model can see and do.

Is agent engineering just prompt engineering with extra steps?

No — they operate on different surfaces. Prompt engineering optimizes the text inside a single request. Agent engineering designs the system that constructs requests, executes tool calls, bounds loops, and verifies results across many turns. Context loss and confident wrong answers are the two most common agent failures, and neither is fixable by rewriting a prompt.

What's the difference between a workflow and an agent?

Who decides the next step. In a workflow, you wrote the control flow and the model fills in steps; execution is deterministic and debuggable. In an agent, the model chooses the next action from available tools, so the path varies per run. Workflows are cheaper and more testable. Choose an agent only when the path genuinely cannot be enumerated in advance.

Do I need an agent framework?

Not to start, and starting without one teaches you what the framework is hiding. A working loop — call model, execute tool, append result, check stop condition — is roughly 200 lines. Adopt a framework once you need durable execution, checkpointing, or replay across process restarts. Adopting one to get a loop buys abstraction you cannot yet debug.

Is agent engineering a real engineering discipline?

Yes, but most of it is inherited rather than invented: retries, idempotency, backpressure, tracing, and contract design transfer almost unchanged. What is genuinely new is narrow and specific — non-determinism as a cost line, context as a depleting budget, and a planner you cannot unit test. That narrow part is real engineering, and it is where the unpriced risk sits.