From AI Adoption to AI Accountability: Why Cost, Not Quality, Is the Leadership Problem

The bill that took three weeks to notice

In 2022 I was mid-transition into a tech lead role. New scope, new meetings, new set of things I was supposedly now responsible for watching. Somewhere in that handover, a Lambda function I had deployed a few weeks earlier went into an infinite invocation loop.

I don't remember the exact trigger anymore. Some downstream call failed in a way that made the function retry itself back into existence. What I remember is what wasn't there. No billing alarm. No concurrency cap. Nobody explicitly watching the AWS bill during the handover, because watching the bill hadn't been assigned to anyone: it had just quietly belonged to whoever built the thing, and I'd built it, and I was busy becoming a tech lead.

It ran for weeks.

Not minutes. Not an afternoon before someone noticed a spike. Weeks, invoking itself over and over, each invocation billed, each one invisible, because nothing in the system was designed to say anything about it. No alert fired because I hadn't set one up. No dashboard flagged it because I wasn't looking at a dashboard. The function did exactly what a function does under load: it ran when called. It just never stopped being called.

I found out the way most people find out: the AWS invoice landed at the end of the month, and the number was wrong. $5,350, for one function, in one month. I remember sitting with the line item longer than the number justified, because the number wasn't really the problem. The problem was the three weeks of silence before it. A system had been misbehaving in production for weeks and the only thing that eventually noticed was a finance document, not an engineer, not a monitor, not a teammate glancing at a graph.

The money stung. What stung more was explaining it. I was newly in a seat where I was supposed to be the person who caught exactly this kind of thing before it became a line item. Instead I was the one who'd shipped it, and the one who had to walk someone through why it happened, freshly into a role where that explanation was supposed to be beneath me by now. Discovered on the bill, not the design review. That sentence has sat in my head for four years, because it names the actual failure precisely: not that a mistake happened, but that the mistake had no way to announce itself except through money, weeks after the fact.

I added a billing alarm immediately, on everything. But the real change was underneath that: a system's real cost is the code you forget is running, not the code you write. The code you're actively building gets your attention by default: you're staring at it, testing it, reviewing it. The code that's already shipped and quietly executing in a loop gets none of that unless you built something to force it to.

Why "are we using AI?" just became "what is it costing us?"

That used to be a side conversation. For most of the last two years, the dominant question engineering organizations asked about AI was some version of "are we using it enough." Adoption was the scoreboard. Leaders compared how many teams had Copilot seats, how many workflows had an agent bolted onto them, how fast the org was moving up the adoption curve relative to competitors.

That question has quietly stopped being the interesting one. A 2026 survey found that cost is now the top concern for engineering leaders adopting AI, cited by roughly 42% of respondents, ahead of quality and ahead of security. Ahead of quality is the detail worth sitting with. For two years, the anxious question about AI-generated code was "can we trust what it wrote." That anxiety hasn't disappeared. It's been overtaken by a more basic one: what did generating, retrying, and reviewing that code actually cost us, and can we account for it.

That's the natural next question after adoption, not a surprising one. Once AI usage stops being a pilot and becomes infrastructure — running in CI, in code review, in customer support, in internal tooling — it starts generating a bill with the same shape as every other piece of infrastructure. The industry conversation has moved accordingly, from "drive adoption" to "prove value, prove ROI." That's a healthier question. It's also a harder one, because proving value requires a denominator, and most organizations that raced to adopt never built one.

The mechanism: no ceiling, quiet compounding, late discovery

Here's the belief this all traces back to: a system's real cost is the code you forget is running, not the code you write. Apply it past a single Lambda function, at the scale of an organization's AI workflows, and it explains the 42% almost exactly.

What a forgotten Lambda and a retrying agent have in common

The Lambda mechanism had three parts. No ceiling: nothing capped how many times it could invoke itself. Quiet compounding, because each invocation was small and unremarkable on its own, invisible in aggregate until someone finally added it up. And late discovery — the only thing that eventually noticed was a monthly invoice, not a system built to notice.

Map that onto an AI agent workflow and the parts line up exactly. An agent that retries a failed tool call has no ceiling unless someone puts one there. It will keep trying, and each retry is a fresh round of tokens billed at whatever rate the model charges. A multi-step agent chain that keeps reloading context on every hop compounds quietly. Each individual context load looks cheap; the cumulative token spend across a long-running task looks nothing like the cost of any single step. And most teams discover the actual number the same way I did: on a monthly invoice from the model provider, not in a design review where someone asked "what stops this from running away."

The failure isn't new. It's the same structural gap, wearing new infrastructure.

Why "pay-per-use" never meant "safe by default"

Pay-per-use is sold as the responsible option. It sounds conservative: you only pay for what you use, no idle capacity, no waste. That framing is backwards. A provisioned box gives you a ceiling for free: it can only do so much work before it falls over, and that failure is loud and immediate. Pay-per-use removes that ceiling. It doesn't fail loudly when it runs away. It just keeps running, and keeps billing, until a human decides to look. Serverless didn't make runaway cost harder to hit; it made it easier, because the thing that used to stop you by accident now has to be built on purpose.

A lot of multi-agent architecture is premature optimization wearing a new name: more moving parts marketed as sophistication, adopted before the single-agent version has even been pushed to its limit. The cost mechanism is the sharper version of that same instinct: more agents, more hops, more retries, sold as capability, each one an additional place where the ceiling needs to exist and usually doesn't.

I've made this exact call at the single-feature level before, deciding where an LLM-powered feature needs guardrails baked into the architecture rather than bolted on after launch, in designing LLM-powered features for production. This is the same decision, just made at organizational scale instead of one feature's scope: the ceiling is either a design-time decision or it's a discovery you make on an invoice.

Two comfortable answers that don't work

Faced with a real cost problem, most organizations reach for one of two answers that feel like action but aren't.

The vendor cost calculator measures spend, not value

Every major AI vendor now ships a cost dashboard: tokens consumed, dollars spent, usage trends by team, by model, by day. These are useful for what they measure and useless for the question that actually matters. They tell you how much you spent. They tell you nothing about whether that spend produced anything. A team that burns $40,000 a month on an agent workflow and merges nothing usable looks identical on a vendor dashboard to a team that burns $40,000 and ships production code daily. Both are just a spend line, going up. A calculator that only measures the numerator will always tell you spend is rising. It was never built to tell you whether the rise is earning anything.

"Measure what matters" names no metric

The other comfortable answer is worse, because it sounds like guidance while giving none. "Measure what matters" is a sentence every leadership deck can agree with and no engineer can act on, because it names no metric, no denominator, no owner. It's the kind of line that gets nodded at in a review and then nobody does anything differently on Monday, because there was never a specific number attached to it. A piece of guidance that survives being printed on a poster isn't guidance anymore. It's a mood.

The gap in both answers is the same: neither one names a metric that connects spend to outcome. That gap is exactly where a lot of "engineering metrics that matter" conversations stall: teams instrument dashboards for engineering metrics that measure motion without ever forcing the question of what the motion produced.

The one metric: cost-per-verified-unit-of-work

This is the section the rest of the piece exists to earn.

The definition

Cost-per-verified-unit-of-work is a formula, not a philosophy:

Cost-per-verified-unit-of-work = total cost of an agent run ÷ units of work that passed verification.

A unit of work is whatever your domain's actual definition of "done" is: a merged pull request, a resolved ticket, a correct answer. The denominator is explicitly not units generated. An agent that produces ten pull requests and none of them merge has generated ten units and verified zero. Its cost-per-verified-unit-of-work is infinite, which is the correct number, because infinite is what "we paid for this and got nothing" should read as.

This matters because generating code was never the constraint. Verifying it is. I've made that case in more detail arguing for letting agents merge autonomously only once verification is solid enough to trust. Any cost metric that ignores verification is measuring the easy half of the problem and calling it done.

A worked example

The numbers below are illustrative math, not a reported statistic. They exist to show how the formula works, not to claim any real team's results.

Say two agent workflows each spend $100 on a task. Workflow A produces three pull requests, and all three merge after review. Workflow B also produces three pull requests, at the same $100 spend, but all three bounce in review and get rewritten by hand before anything ships. Cost-per-token, cost-per-run, and every vendor dashboard treat these identically: $100 spent, three PRs generated, done. Cost-per-verified-unit-of-work does not. Workflow A costs roughly $33 per verified unit. Workflow B costs infinite, because the verified count is zero: the $100 bought review time and rework, not merged code. Same spend, same output volume, entirely different value, and only one of the two metrics can tell them apart.

Why the 3x number is meaningless without this

I built a custom agent-orchestrator that ships MVPs three times faster than we used to. It's a real system, still running, and it earns its credit. But 3x is the system's number, not mine, and on its own it's an incomplete claim. Three times faster at what cost, verified by what standard? Leverage without a cost denominator is just faster spending with a better story attached. The number only means something once you can say 3x faster per verified unit of work, not 3x faster at generating things that still need a human to check them. That's the same generation-review imbalance that shows up when teams look at how AI is reshaping team topologies through the lens of a generation-review ratio.

This is the cost-side version of the DORA-to-DevEx correction

The engineering measurement world has already run this correction once, on a different axis. DORA metrics measured throughput: deployment frequency, lead time, change failure rate. They were useful and incomplete, because a team can hit every DORA number while burning out, drowning in cognitive load, or shipping fast in a way that quietly degrades everything downstream. The correction was DevEx: measuring flow, cognitive load, and developer satisfaction alongside the throughput numbers, because throughput alone was rewarding the wrong behavior. I've written about what that correction actually requires in practice, and the short version is that measuring only motion always eventually rewards busywork.

Cost-per-verified-unit-of-work is that same correction, applied to spend instead of speed. Cost-per-token is DORA's mistake with a dollar sign on it: a throughput number that rewards verbose generation and punishes nothing. Cost-per-verified-unit-of-work asks what DevEx asked: not how much happened, but how much of what happened was worth having happened.

Monday morning: ship the ceiling before the workflow

None of this requires a FinOps initiative or a quarter of planning. It requires five things, this week, in order.

1. Add a spend alarm at a number low enough to actually fire. A workable starting point: 1.5x last week's actual spend for that workflow, not a round number pulled from nowhere. Not a number set so high it never triggers and exists only to say you have one. Set it low enough that hitting it forces a real question, before the next agent workflow ships, not after.
2. Put the concurrency or token ceiling in the same pull request as the workflow. Not a follow-up ticket. Not a backlog item labeled "tech debt: add guardrails." If the ceiling isn't in that PR, the workflow doesn't ship with it. It ships with hope, on someone's promise to circle back.
3. Assign ownership of that ceiling by end of day the workflow ships. A name, not a distribution list, not "the platform team" in the abstract. Someone specific gets paged if it fires, and everyone building the workflow knows who that is before it goes live.
4. Replace whatever cost dashboard exists today with one line: cost-per-verified-unit-of-work, defined for your context. Decide, in writing, what "verified" means for this workflow — merged, resolved, correct — before the first invoice arrives, not after someone asks why the number looks fine and the outcomes don't.
5. Ask this week, of every AI workflow already in production: what stops this from running unbounded, and who gets paged if it does? If the honest answer is "nothing" and "nobody," that's this week's fix, not next quarter's.

If your actual cost problem is runway rather than architecture — you're watching a burn rate, not a design decision — that's a different conversation, and I've written about cutting cloud cost under startup constraints separately. Don't conflate the two; one is a design-time decision, the other is a survival-mode one.

The same test applies here that applies to any 3am incident: can whoever's on call explain the bill, not just the outage.

This closes out a three-part argument about architecture discipline in the AI era: the architecture reality of AI-generated code that only the model understands, the discipline of not reaching for multi-agent before it's earned, and now, the accountability question of what any of it costs.

Frequently asked questions

Is cost-per-verified-unit-of-work the same as cost-per-token?

No, and the difference is the entire point. Cost-per-token rewards verbose generation: an agent that writes more, retries more, and re-reads more context looks more expensive per token, but the metric never asks whether any of that output was usable. Cost-per-verified-unit-of-work divides spend by outcomes that passed verification, so a workflow that generates a lot of unusable output gets punished for it instead of just recorded as "busy."

Do we need a FinOps team to do this?

No. This is an architecture decision, not a finance function. A budget alarm and a concurrency ceiling are things an engineer configures in the same PR as the workflow, the same way you'd add a rate limiter or a timeout. FinOps can help you interpret spend across the org later. It shouldn't be the reason the first ceiling doesn't exist.

What counts as "verified" in practice?

Whatever your team's actual definition of done already is, made explicit for this workflow. A merged pull request for a coding agent. A resolved ticket, closed by the requester, for a support agent. A correct answer against a held-out test set for a QA or research agent. The specific bar matters less than picking one in writing before the workflow ships: an undefined "verified" is just cost-per-token wearing a better name.

Isn't this just cloud cost optimization for AI?

Not quite. Cloud cost optimization is mostly about reducing spend for the same outcome: right-sizing instances, cutting idle capacity. Cost-per-verified-unit-of-work isn't asking you to spend less. It's asking whether the spend you already have is producing anything, and it will happily tell you to spend more on a workflow with a low cost-per-verified-unit and less on one with a high one, regardless of which one's raw dollar total is bigger.

Won't a concurrency or token ceiling break legitimate long-running agent work?

Yes, if you set it with zero margin above expected load. That's a badly set ceiling, not an argument against having one. The fix isn't skipping the ceiling; it's sizing it against the actual expected range of the workflow, with headroom, and treating a ceiling that fires as a signal to investigate, not an outage to silence. A loop that legitimately needs more room asks for more room, on purpose, from a named owner. That's a different failure mode than a loop nobody's watching burning $5,350 in silence.

If you're staring at a design doc or a whiteboard sketch of an agent workflow right now and the ceiling is the part you haven't figured out yet, that's exactly the kind of thing worth a second opinion before it ships, not after the invoice. No fee, no pitch — bring the doc, the diagram, or the half-formed idea to hello@ruchitsuthar.com and I'll give you an honest read on where it's missing one.