Multi-Agent Architecture Is Premature Optimization With a New Name

The SaaS I built for millions of users it never had

The schema was multi-tenant from day one. Tenant isolation, row-level scoping, a provisioning layer that could stand up a new customer environment without a deploy. Above that, abstraction layers I was proud of — swappable storage backends, a plugin system for features no one had asked for, an event bus for a load that existed only in my head. I designed it for the millions of users I was certain were coming.

It never launched to market. It missed its window while I was building infrastructure for scale it never saw. Today it quietly serves two or three internal users. Months of engineering, spent building for the problem I fantasized about instead of the one I had.

Now look at a multi-agent diagram. A planner that decomposes the request. A retriever that fetches context. A critic that reviews intermediate output. An executor that runs the tools. An orchestrator sitting on top, routing state between all of them, retrying failures, reconciling disagreements. Clean boxes, clean arrows. It looks like the kind of system a serious team ships.

It's the same move. I built tenant isolation for tenants I didn't have. Plenty of teams are building critic-executor handoffs for a failure they've never watched happen. I've told the SaaS version of this story before — this is the same mistake wearing an agent diagram.

The tell: do you have a multi-agent problem, or a multi-agent aesthetic?

Over-engineering is a confession that you didn't understand the problem. The agent-layer version of that confession has a specific tell, and it shows up in how you answer one question: what did the single agent fail at?

"It felt fragile" is aesthetic. "The output wasn't reliable enough" is aesthetic. Both are feelings about the system, not observations of it. They describe your discomfort, not the model's behavior. You can't fix a feeling with an orchestrator.

"It kept losing the user's original constraint after the third tool call and silently dropped it from the final answer" is a problem. That's a witnessed failure with a location and a mechanism. You watched it happen, you can point to where, and now you have something a second agent might actually be built to fix — a dedicated step that re-injects the constraint, or a check that verifies it survived. The difference between the two answers is the difference between engineering and decoration.

What does "we need multi-agent" actually mean when no one can name the failure?

It means the team has decided how the system should look before observing how it fails. That's the architecture-review equivalent of picking the answer before reading the question. When "we need multi-agent" arrives with no named single-agent failure behind it, it isn't a technical requirement. It's a preference for a diagram that signals sophistication, and it will cost you like one.

I'm not skeptical from the sidelines. I built the orchestrator.

I'm not the engineer who's never built one. I'm the engineer who built one and can tell you what it cost before it paid for itself.

My core job on the R&D group is POCs and MVPs — clarifying fuzzy requirements, proving a technical concept works, giving stakeholders something real enough to approve budget against. The bottleneck was never the idea. It was cycle time: how long from "here's a concept" to "here's something you can click." Ideas sat in a queue waiting on full dev loops just to get validated, and slow validation stalls decisions and budget.

So I built a custom agent-orchestrator that runs the build through AI agents, each following a defined step of our process, and turned it into a repeatable framework the team uses without me in the room. MVPs now ship 3x faster than they used to. I'll be precise about that number: 3x is the system's number, not mine. I didn't get three times faster at building. The system I encoded lets the team move at that speed. Leverage is the system you build, not the work you do faster.

Here's the part the diagram never shows. The orchestrator didn't pay for itself on day one. It cost first. Every boundary between agents turned free-floating context into state that now had to be packaged and reconstructed on the other side — context that lived free inside one agent's head suddenly needed a contract to cross a wall. When a run produced a wrong MVP, the debugging wasn't "read the stack trace." It was replaying a non-deterministic conversation across four steps to find which agent quietly went off the rails. There was a real stretch where coordination cost more than it returned. It became worth it only after I'd paid those costs down deliberately. That's the concrete referent for the framework below.

The four costs multi-agent has to pay off before it's worth it

Coordination isn't free, and it isn't cheap. Before you split one agent into several, four costs come due. Each one has a test you can write directly into the design doc, and if you can't answer it, you're not ready to add the boundary.

State handoff cost

Inside a single agent, context is ambient. The model carries the user's intent, the constraints, the intermediate results, all in one place. No serialization. The moment you introduce a second agent, every piece of shared understanding has to be explicitly packaged, passed, and reconstructed on the other side. That handoff is where constraints get dropped and nuance gets flattened into whatever fit in the message.

Design-doc test: For each agent boundary, write down exactly what state crosses it and what happens if a field is lost or malformed. If you can't specify the contract, the boundary is a leak you haven't found yet.

Failure-mode multiplication

One agent has one failure surface. Add a second and you don't get two independent surfaces — you get their product. Each agent can fail on its own, and each handoff between them can fail, and a correct output from one agent can be misused by the next. The combinations grow faster than the agent count.

Design-doc test: List every agent and every handoff, then ask what a confidently wrong output at each point does downstream. If the honest answer for any node is "I'm not sure," that node is an unowned failure mode.

Non-deterministic cross-agent debugging

A single agent that misbehaves is hard enough. The output isn't deterministic, so you're reasoning about tendencies, not fixed logic. Across multiple agents, you're debugging a non-deterministic conversation. The same input can route differently on two runs. Reproducing a failure means reproducing a whole chain of stochastic steps, and the bug might live in any of them or in the handoff between them.

Design-doc test: Pick a plausible failure and describe, step by step, how you'd reproduce and localize it to a specific agent. If the procedure is "add more logging and stare," you don't have a debugging story, you have hope.

Cost/latency compounding

Every agent is at least one model call, usually several. Split a task across four agents and you've roughly multiplied token cost and stacked latency end to end — often serially, because one agent waits on the last. A judge or critic loop, which you'll likely need, adds another full pass. The user waits longer and you pay more, per request, forever.

Design-doc test: Estimate token cost and wall-clock latency for the multi-agent path versus a single agent, at your real request volume. If multi-agent costs materially more and you can't name what you're buying with it, the answer is already in the spreadsheet.

I've written before about the ladder for designing LLM-powered features: start with the simplest thing that could work and climb only when it fails. The same ladder logic applies one level up, at the agent layer. The rungs are just more expensive here.

Of these four, the orchestrator paid off state handoff cost fastest, because I could enforce the contracts explicitly and the process steps mapped cleanly to boundaries. The one that took longest to become worth it was non-deterministic cross-agent debugging. That cost didn't go away when I built the system — it went away when I built the evals that told me which step regressed. Which is the next problem.

What it looks like when it goes wrong

Here's the failure that surprises teams: a naive multi-agent system can hallucinate more than the simpler design it replaced.

The mechanism is straightforward. More agents and more tool calls mean more intermediate outputs, and every intermediate output is a place where the model can be confidently wrong. In a single-agent design, a shaky intermediate result often stays visible in the same context, where the model can still reconsider it. In a naive multi-agent chain, agent one produces a plausible-but-wrong fact, hands it to agent two as settled input, and agent two builds on it as ground truth. The error doesn't get caught. It gets laundered. Each downstream step treats the previous step's output as fact and adds confidence to a mistake.

The only thing that stops this is a verification or judge loop you deliberately build in — a step whose explicit job is to check intermediate outputs before they propagate. That loop isn't a bonus feature. In a multi-agent system it's load-bearing, and it's another full model pass you're paying for, which loops back to the cost you priced above. If your multi-agent design doesn't have a verification step, you haven't reduced hallucination risk. You've added surface area for it.

This is a generation problem, not a retrieval problem, and the distinction matters for where you put the fix. I've covered retrieval failure versus generation failure in detail — if you can't tell which one you're looking at, start there, because a judge loop aimed at the wrong layer catches nothing.

The eval gap gets more dangerous, not less, at the agent layer

Most teams have observability. Almost none have evals. They can see what the system did — the logs, the traces, the token counts — but they can't score whether what it did was correct, or catch a regression before a user does.

In a single-agent system, that gap is a nuisance. You notice quality slipped, you squint at the logs, you eventually find it. Annoying, survivable. In a multi-agent system the same gap changes category. Without evals, a failure surfaces as "the system is flaky" — unattributable and undebuggable. You know something regressed. You can't say what. With per-agent evals, the same failure surfaces as "step 3's classifier regressed after the prompt change" — a fixable, ownable bug with an address.

Why does "we have logging" not mean you can debug a multi-agent failure?

Logging tells you what happened. It doesn't tell you whether what happened was right. In a non-deterministic chain of agents, that's the whole game — you can have complete traces of every call and still not know which step produced the wrong answer, because "wrong" is a judgment logging doesn't make. Evals make that judgment per step, which is what turns a flaky system into a located bug. I've laid out the mechanics in building the LLM regression net; the point here is narrower. Multi-agent multiplies the number of places a regression can hide, and observability without evals gives you a bigger haystack, not a way to find the needle.

The bar multi-agent has to clear

Here's the test, and it's blunt: coordination overhead is justified only when it measurably beats a single agent with tools running Plan-and-Execute or ReAct — on a named metric.

Not a vibe. A number. Success rate, cost per request, latency, correction count — pick the one that matters for your use case and measure both designs against it. If the multi-agent version doesn't beat the single-agent baseline on that metric by a margin that pays for the four costs above, you don't have a justification. You have an aesthetic with a benchmark you never ran.

My orchestrator cleared this bar for one reason: it beat the prior process on a measured cycle-time number. 3x, against a baseline we actually tracked. It wasn't approved because it looked more architecturally serious on a slide. It earned its complexity against a metric.

The design-review test

The question I ask, every time, when a multi-agent design lands on my desk: what did the single agent fail at, specifically, that you watched happen?

If the answer is a witnessed failure with a location and a mechanism, we're having an engineering conversation about whether a second agent is the right fix. If the honest answer is "we haven't tried the single agent yet" — that's the verdict. Stop. Run the single agent first. You can't price coordination against a baseline you never established.

Frequently asked questions

Is multi-agent architecture ever the right call?

Yes — when a single well-scoped agent with tools has been pushed to a specific, witnessed failure, and a multi-agent design measurably beats it on a named metric. The problem isn't multi-agent. The problem is reaching for it before the single-agent baseline exists. Coordination is a cost you pay to buy something specific; if you can name what you're buying and prove the trade, it's the right call.

How do I know if my team has a multi-agent problem or just a multi-agent aesthetic?

Ask what the single agent failed at. If the answer describes a feeling — "it felt fragile," "it wasn't reliable enough" — that's an aesthetic. If it describes a witnessed failure with a location and mechanism — "it dropped the user's constraint after the third tool call" — that's a problem. Only the second one tells you what a second agent should be built to fix.

What's the biggest hidden cost of multi-agent systems?

State handoff. Inside one agent, context is ambient — intent, constraints, and intermediate results all live in one place. Every agent boundary forces that state to be explicitly serialized, passed, and reconstructed, and that handoff is where constraints get silently dropped. It's the cost teams see last because it's invisible on the diagram, where boundaries look like clean arrows instead of leaks.

Why do multi-agent systems hallucinate more than single-agent ones?

More agents and tool calls mean more intermediate outputs, and each one is a chance for a confidently wrong answer. In a naive chain, one agent's wrong output gets handed to the next as settled fact and built upon, so the error gets laundered instead of caught. Without a deliberate verification or judge loop, a multi-agent design adds hallucination surface area rather than reducing it.

Do I need evals before I build a multi-agent system?

Yes. Without evals, a multi-agent failure surfaces as "the system is flaky" — unattributable and undebuggable. With per-agent evals, the same failure becomes "step 3 regressed" — a fixable, ownable bug. Most teams have observability but not evals, and that gap is survivable in single-agent systems and dangerous in multi-agent ones, because coordination multiplies the places a regression can hide.

What should I compare a multi-agent design against before building it?

A single agent with tools running Plan-and-Execute or ReAct. Measure both on a named metric — success rate, cost per request, latency, or correction count. Multi-agent is justified only when it beats that baseline by a margin large enough to pay for state handoff, failure-mode multiplication, non-deterministic debugging, and compounding cost and latency. If you never ran the baseline, you have nothing to justify against.

Your Monday morning

Pull up your current multi-agent diagram. Next to each agent boundary, write the specific single-agent failure it was added to fix — the one you watched happen, with a location and a mechanism, not a feeling.

Any boundary that gets a real, witnessed failure written next to it stays. Any boundary that gets "it felt cleaner" or "it seemed more reliable" or a blank — cut it back to a single agent with tools, and leave it that way until a real failure shows up and earns the boundary back.

Everything above reduces to one honest sentence: what did your single agent fail at, that you watched happen? If you can't write it, you're not ready for a second agent. Run the first one.

This is the middle piece of a three-part argument about architecture discipline in the AI era. The opening piece covers the same instinct applied to code itself: why AI-generated code fails the 3am-fixability test. The closing piece takes it to leadership: why cost, not quality, is now the AI accountability question. If this landed, the over-engineered SaaS story is the same mistake one architectural layer down.