Hiring for Judgment in the AI Era: An Interview Playbook

---

I changed how I interview engineers after a single uncomfortable realization. I was running a standard technical screen — implement this function, handle these cases — and the strongest coding performance came from a candidate who, in the follow-up discussion, couldn't explain why any of his choices were correct. He'd internalized patterns well enough to reproduce them fast. But when I handed him a piece of plausible-looking code with a subtle concurrency bug and asked him to assess it, he accepted it. It looked right. He had no instinct that it wasn't.

A few months into the AI-tooling wave, that candidate's weakness became the whole industry's blind spot. When an AI can write the function in seconds, "can you write this function" stops being a useful signal — and "can you tell whether this function is actually correct" becomes the entire job. I'd been screening hard for the skill that just got automated and barely testing the skill that suddenly carries the weight.

This is the playbook I rebuilt from that realization. It's not anti-AI — pretending AI doesn't exist in interviews is as silly as pretending it doesn't exist at work. It's about testing for judgment: the scarce, durable thing that determines whether someone is valuable on a team where the machine writes the first draft. (This is the hiring-side companion to how AI is reshaping team topologies and builds on the judgment framework for AI-augmented hiring.)

What the old interview tested — and why it broke

The traditional technical loop optimized for code production under time pressure: reverse this tree, implement this cache, no looking anything up. It made sense when writing correct code quickly was the differentiating skill.

AI broke the signal in both directions:

The thing you were measuring is now cheap, so measuring it harder just gets you more false positives (people who look fast) and false negatives (people who think well but don't perform the now-automated trick). You have to move the target.

The core shift: have them critique AI output

The single most useful interview format I've adopted: give the candidate AI-generated code that looks correct but is subtly wrong, and ask them to assess it.

This format is gold because it tests exactly what the job now requires:

• Appropriate suspicion. Does the candidate treat plausible-looking code as guilty until proven correct, or do they trust it because it compiles and reads well? The weak signal is someone who accepts; the strong signal is someone who's suspicious in the right places.
• Depth of reasoning. Why is it wrong? Can they articulate the concurrency hazard, the unhandled input, the security implication? Surface-level "this variable name is bad" is different from "this will deadlock under concurrent writes."
• Correction quality. How do they fix it? Do they reach for a clean solution or patch the symptom?

Vary the flaw type across the loop — a race condition, a missing edge case, a security hole (e.g., an injectable query), a wrong abstraction that'll cause pain later. You learn far more from "here's some code, what's wrong with it and how would you fix it" than from "write this from scratch."

Probe judgment, not recall

Beyond code review, structure the loop to surface judgment directly:

• "What would you build, and why?" Give an ambiguous, realistic problem and watch them navigate the decisions — what to clarify, what trade-offs they weigh, what they'd explicitly not build. Deciding what to build is now a bigger part of the job than building it. (Tie this to real architecture decisions.)
• Reason about a system they didn't write. Show an existing design or codebase snippet and ask them to evaluate it, find the risks, suggest improvements. Engineers now spend more time understanding and verifying others' (and AI's) code than writing greenfield — interview for it.
• "Tell me about a time the obvious solution was wrong." Real war stories reveal judgment that's almost impossible to fake. Follow the threads: how did they realize, what did they do, what did they learn.
• Push on trade-offs, not answers. The strong signal isn't "knows the right answer" — it's "reasons well about competing options and is honest about what each one costs."

Crutch vs tool: let them use AI, then watch how

You'll have to decide whether candidates can use AI in the interview. My take: letting them use AI — and watching how — is more informative than banning it. The distinction that matters is the one that predicts on-the-job value:

• Crutch user: accepts AI output uncritically, can't explain why it's correct, and is helpless when the AI is wrong (which they can't detect). This is the candidate who looks productive and ships subtle bugs.
• Tool user: prompts well, verifies what comes back, catches the AI's mistakes, and can explain and defend the result as if they'd reasoned it themselves. This is who you want.

So the rule isn't "no AI" — it's "use whatever you want, and be ready to explain and defend every line." A candidate who can't defend the code their AI produced has told you they'd ship code they don't understand. That's the single most important thing to screen out, because it's exactly the failure mode AI introduces at scale.

Stop rewarding what's now free

Finally, prune the parts of your loop that test the commoditized skill:

• Drop pure from-scratch algorithm grinding as a primary signal. It now mostly measures interview prep, not job performance.
• Stop over-weighting speed of typing correct syntax. Weight quality of reasoning, suspicion, and verification.
• Keep fundamentals — but test them through judgment. You still want people who understand complexity, concurrency, and data structures. Test that understanding by having them evaluate and reason, not regurgitate.

The aim is a loop that would pass a thoughtful engineer who leans on AI well and fail a fast one who can't tell good output from bad — the opposite of what many loops do today.

What to do Monday morning

1. Add one "critique this AI output" exercise to your loop. Take real AI-generated code, plant a subtle flaw (race condition, edge case, security hole), and ask candidates to find and fix it. It'll immediately become your most informative round.

2. Decide your AI policy and make it "use it, then defend it." Let candidates use AI and explicitly assess how they use it — crutch vs tool — and whether they can explain every line.

3. Audit your loop for now-free skills. Find the rounds that mostly test from-scratch code production and replace them with judgment, system-reasoning, and trade-off rounds.

4. Add a "the obvious answer was wrong" question to surface real, hard-to-fake judgment from experience.

Key takeaways

• "Can you write this function" is no longer a useful signal. AI commoditized code production, so the classic loop over-produces false positives (look fast) and false negatives (think well, don't grind). Move the target to judgment.

• The best new format is critiquing AI output. Give candidates plausible code with a subtle flaw and watch for appropriate suspicion, depth of reasoning, and correction quality. Accepting it is the weak signal; catching and explaining it is the strong one.

• Interview for judgment, not recall: what they'd build and why, reasoning about systems they didn't write, real war stories where the obvious answer was wrong, and trade-off thinking over right answers.

• Crutch vs tool is the key distinction. Let candidates use AI and watch how: a tool user directs, verifies, and defends; a crutch user accepts uncritically and can't explain it. Screen out anyone who'd ship code they don't understand.

• Stop rewarding the commoditized skill. Drop pure algorithm grinding and typing speed as primary signals; keep fundamentals but test them through evaluation and reasoning, not regurgitation.

Your next step

Take one round of your current interview loop and ask: does this test something an AI now does for free? If it does, replace it this week with a "here's some plausible code — what's wrong with it, and how would you fix it" exercise using real AI-generated output with a planted flaw. You'll learn more about a candidate's actual value in fifteen minutes of watching them reason about correctness than in an hour of watching them reproduce an algorithm. In the AI era, you're not hiring people to write code. You're hiring the judgment that decides whether the code should ship.

Frequently asked questions

How should technical interviews change because of AI?

Shift the target from code production to judgment. Since AI can write most functions on demand, asking candidates to implement one from scratch mostly measures interview prep rather than job performance and produces both false positives (people who look fast) and false negatives (strong reasoners who don't grind algorithms). Instead, test whether candidates can tell when code is subtly wrong, decide what to build and why, and reason about systems they didn't write — the skills that actually determine value on a team where AI writes the first draft.

What's the best interview format for the AI era?

Give the candidate AI-generated code that looks correct but contains a subtle flaw — a race condition, a missing edge case, a security hole like an injectable query, or a poor abstraction — and ask them to assess and fix it. This tests appropriate suspicion (do they treat plausible code as guilty until proven correct), depth of reasoning (can they articulate exactly why it's wrong), and correction quality (do they fix the cause or patch the symptom). You learn far more from this than from a from-scratch coding exercise.

Should candidates be allowed to use AI during interviews?

Generally yes — letting them use AI and observing how is more informative than banning it. The distinction that predicts on-the-job value is crutch versus tool: a crutch user accepts AI output uncritically, can't explain why it's correct, and is stuck when it's wrong; a tool user prompts well, verifies the output, catches the AI's mistakes, and can explain and defend the result. Make the rule "use whatever you want, but be ready to explain and defend every line," and screen out anyone who would ship code they don't understand.

How do I test for engineering judgment in an interview?

Use formats that surface decisions rather than recall: give an ambiguous, realistic problem and watch how they decide what to build, what to clarify, and what to deliberately not build; show an existing design or codebase and ask them to find risks and suggest improvements; ask for a real story where the obvious solution turned out to be wrong and follow the threads; and push on trade-offs rather than single right answers. Strong candidates reason well about competing options and are honest about what each costs.

Should we stop asking algorithm questions entirely?

Stop using pure, from-scratch algorithm grinding as a primary signal, since it now largely measures interview preparation rather than job performance. But don't abandon fundamentals — you still want engineers who understand complexity, concurrency, and data structures. The change is how you test that understanding: have candidates evaluate, debug, and reason about code and systems rather than reproduce algorithms from memory, so you measure the comprehension and judgment that remain scarce rather than the production that AI has made cheap.