The genie and the contract

The problemAn unpredictable genie

Kent Beck has spent five decades arguing that tests should come first. So it’s worth noticing how he describes the AI coding agents he now works with: an “unpredictable genie” that grants your wishes, often in literal and unexpected ways. You ask for something. It gives you something. Whether that something is what you actually needed is a separate question.

In the same conversation, Beck calls TDD a “superpower” with these agents, because they introduce regressions constantly and a test suite is the cheapest way to catch them. But he also lets slip the detail that should give you pause: he has trouble stopping agents from deleting the tests to make them pass. Tests as both the defense and the thing being gamed — that’s the problem this is about.

A quick word on terms, because one of these acronyms gets used loosely. TDD is the older discipline: write a failing test first, write just enough code to make it pass, tidy up, and repeat. SDD is the newer one: capture what you’re building as a structured technical specification — the API shapes, data models, and acceptance criteria — and treat that document as the contract the code has to satisfy. The important thing is that SDD is a practice, not a product. GitHub’s Spec Kit is one tool that makes it easy to adopt, and it’s the running example used throughout — but everything here applies to any SDD setup. (It also isn’t the same as story-driven development, which works at the looser level of user stories and relates to TDD differently.)

The thesisWhy neither survives the genie alone

Run SDD on its own and the meaning drifts: the agent forms its own reading of an ambiguous spec and builds something internally consistent that still misses what you meant, because prose can’t be executed and nothing pulls the code back to the words. Run TDD on its own and you get the opposite — code that passes the tests in front of it and nothing more, with no spec to say what those tests should assert, so they end up encoding whatever the agent decided to build. Either way: green checkmarks, wrong target.

The reason to combine them isn’t that both are nice. It’s that the spec and the test are two independent encodings of the same intent — one human-readable, one machine-executable — and each covers the other’s blind spot. The spec gives the test a reason to exist, so the suite isn’t just a mirror of the code. The test gives the spec teeth, so the agent can’t quietly reinterpret the prose.

A spec panel on the left and a test panel on the right both point inward at a central agent, framing it from two sides.

WHAT SHOULD BE
Spec
Human-readable intent

THE GENIE
Agent
Literal · fast · unreliable

WHAT WAS BUILT
Test
Machine-executable proof

The spec and the test are two encodings of the same intent, constraining the agent from both directions.

Two encodings of one intent, constraining the agent from both sides. The contract you hand the genie. The spec is how you phrase the wish precisely. The test is the part it can’t weasel out of.

The pipelineOne spine, two gates, a feedback edge

The diagram below shows the shape. A person defines the intent; Spec Kit’s specify and plan commands turn it into a spec and a plan; a person reviews them — as a checkpoint, not a permanent lock. Three details decide whether this actually works in practice.

First, analyze runs before any code. It’s a read-only consistency check across spec, plan, and tasks. The instinct is to save it for the end, but Spec Kit’s own quickstart is explicit that the first pass belongs before implement, while gaps are still cheap to fix. Run it again afterward as a drift review if you like — but the load-bearing pass is the one before the agent writes a line.

Second, a person owns the acceptance test — the next section explains what that means. Third, the agent builds in vertical slices: one small piece of behaviour taken end to end — a single failing test, just enough code to pass it, a clean-up, then the next slice — rather than generating the whole suite up front or emitting hundreds of lines at once, which is what produces plausible-looking code that breaks in ways no one can localize. And code merges only when the tests are green, the suite has been checked to confirm it actually catches bugs, and the committed tests weren’t edited to force the green.

A vertical pipeline from human intent to a merge gate, with analyze as a pre-code gate, a human-owned read-only acceptance test, a vertical-slice code loop, and a feedback edge from the loop back to the spec.

Human Intent

The why and the what

Spec + Plan

Spec Kit · human-reviewed

Analyze — Pre-Code Gate

Consistency check before code

Acceptance Test

Human-owned · read-only

Code Loop

Slice: red · green · refactor

Merge Gate

Tests pass · mutation caught

The corrected pipeline. No mid-stream tool hand-off, analyze as a pre-code gate, a test the implementer can’t edit, and discoveries that flow back to the spec.

The hard partWho guards the guardrails?

It’s easy to picture this as a tidy “spec → test → code” flow and leave it there. The real engineering is in the failure modes — and they all reduce to one question: if the agent writes the spec, writes the tests, and writes the code, what is actually checking anything?

Who decides what “correct” means?

If one model writes all three — the spec, the tests, and the code — they all share the same blind spot. Misread the intent once and you get a spec that encodes the misreading, tests that enforce it, and code that passes them. Everything is green; everything is wrong. The two “independent” encodings only stay independent if a person owns at least one of them — something the agent can’t quietly rewrite.

The best artifact to put a person in charge of is the acceptance test — the test that says “this feature is done and behaves correctly.” A spec written in prose can be honoured to the letter and still violated in spirit; a test is a concrete, pass-or-fail claim. Owning it doesn’t mean a person hand-writes every assertion — the agent can draft the tests. It means a person reviews them, approves what they assert, and stays accountable for them before they become the bar the code is measured against, rather than letting the same agent silently generate and merge its own. In testing terms, the acceptance test is your oracle: the thing that gets to decide whether the code is right. As others working on this have found, human-controlled tests are the last line of defense — because under pressure the agent’s instinct is to delete a failing test rather than fix the code. If the agent can edit the oracle, there is no oracle.

Green is a proxy, and proxies get gamed

“The agent can’t move on until the test passes” sounds like a guarantee. It isn’t. There are many ways to make a test pass without actually satisfying the intent: weaken what it checks, skip it, replace the real component with a stub, hard-code the answer it expects, swallow the error — or simply rewrite the test. The problem is the classic one: the moment “make the tests pass” becomes the goal, passing tests stop being a reliable sign that anything works.

This isn’t hypothetical. Aider — a popular command-line coding agent, and a reasonable choice for the build loop — will by design edit tests it decides are wrong instead of forcing the code to fit them. That’s genuinely useful when a person is driving; it’s a hole when the agent is running on its own. So how do you stop the agent from gaming the gate? Two defenses do most of the work — and a raw pass/fail count is neither of them.

What tests can’t say

Unit tests answer a narrow question: does this function do what I said, in isolation? They say nothing about performance, security, accessibility, or — most importantly — whether the system actually works for a real user. That last gap is exactly where AI-generated code fails: the agent nails an isolated unit and quietly breaks the end-to-end journey that runs through it. So treat end-to-end and integration testing as a first-class tier, not an afterthought — the layer that drives the real application the way a user would and catches what unit tests can’t see. Tools like Playwright MCP even let the agent run a real browser and check its own work against that journey instead of guessing. For requirements that can’t be a test at all, make them an explicit gate in your build pipeline — a hope is not a control.

Refactor is not optional

The TDD loop has three steps, not two: write a failing test (red), make it pass (green), then clean up the code you just wrote (refactor). Left alone, the agent tends to drop that third step and ship the moment the test goes green. But refactoring is where code quality is kept — and skipping it with a fast, tireless agent is how you pile up exactly the mess the whole process is meant to prevent. Keep refactoring in the loop, and watch a complexity or code-churn signal to catch the slice the agent over-built.

The fixThe spec is living, not frozen

There’s a real tension between the two methods. SDD wants the specification pinned down before the code is written — that’s the entire point of having a contract. TDD, by contrast, is partly a way of discovering the design: you often learn that a requirement was wrong, or an interface awkward, precisely while writing the test for it. So what happens the first time a test reveals that the spec itself was incomplete?

The answer is to treat the spec as a living document, not a one-time sign-off. When a requirement changes — or a test exposes a gap — you don’t patch the test to match the code; you update the spec first (a small, reviewed change), update the tests to match, watch them fail, and let the agent make them pass. It’s the same red-green loop, applied to the spec as well as the code — and it carries one rule you don’t break: changes flow spec → test → code, never test → code on their own. Hold that line and changing requirements stop being the thing this approach handles worst and become the thing it handles best.

Reference buildThe gate lives in the environment

The key design choice is where the discipline lives. It’s tempting to make it a property of a specific tool — this one plans, that one codes. Don’t. Put the gate in the environment — your version control and CI — rather than in any single agent. Once it lives there, the coding agent becomes a swappable part rather than the thing you’re trusting.

Your SDD tool owns the planning spine. With Spec Kit that’s constitution → specify → clarify → plan → tasks → analyze → implement. Put the TDD rules in its constitution file: write a failing test before implementation; never mark a task done while tests fail; never modify a committed test just to make it pass. Every command inherits those rules, so you aren’t relying on the agent to remember them over a long session.

The gate is a pre-commit hook and a CI check, not a prompt. Tests pass, mutations caught, committed tests untouched — enforced by the environment, so it holds whether the code came from Spec Kit’s implement, from Aider, or anything else.

The executor is pluggable. Aider’s native --auto-test loop gives a tighter per-change cycle than a batch command and routes models by cost — a strong model for spec and plan, a cheaper one for the high-frequency loop, caching on. Use it, or don’t. The methodology is the gate.

ProportionMatch the ceremony to the risk

A process that can’t scale down gets abandoned. A full ceremony — a project constitution, a spec, a plan, a task breakdown, plus mutation testing — is overkill for a one-line bug fix. So run two lanes, a split that Spec Kit’s own documentation recommends.

A short fast lane and a longer full lane both end at the same shared gate.

FAST LANE
Low blast-radius intent with 1–2 owned tests

Implement
Move directly into the build path when the intent is narrow, contained, and already well-understood.

FULL LANE
New subsystem · regulated work · higher-risk change

Constitution
Define the non-negotiables.

Spec
Clarify what should be true.

Plan
Decide how the work will be approached.

Analyze
Check consistency before code.

Test Loop
Prove the intent through owned acceptance tests.

Shared Gate
Always applies.

Ceremony scales with risk; the gate doesn’t. The fast lane skips the paperwork — but tests, mutation, and read-only enforcement apply to both.

Is the overhead worth it?

None of this is free. You’re paying for more model calls, more test runs, and more back-and-forth than simply letting an agent write the code in one pass, so it’s fair to ask whether the extra cost earns its keep. The way to keep it sane is to spend where it matters: a strong model on the spec and plan, a cheaper and faster one on the high-frequency build loop; run the expensive checks like mutation testing only at merge, not on every iteration; and keep slices small so each loop stays cheap. But the real question is what you’re comparing against. The baseline isn’t a senior engineer typing flawless code — it’s an unsupervised agent shipping plausible, broken code that you then debug for hours. Against that, the overhead is what buys you a result you can actually trust.

When not to do this

And sometimes the answer is that it isn’t worth it at all. Skip the full process for throwaway prototypes and spikes, where the whole point is to learn fast and discard. Skip it for genuinely trivial changes, where setting up the gate costs more than the bug. And be honest about domains where you can’t write a meaningful test cheaply — there the oracle is hollow and the guarantees are theatre. As Den Delimarsky observes from hands-on Spec Kit use, specs are not a panacea. Naming these limits isn’t a hedge — it’s what separates a methodology from a sales pitch.

CloseThe bar just moved up

What survives the automation isn’t typing code. It’s knowing what’s right to want, stating it precisely enough that a literal-minded genie can’t misread it, and being able to tell whether you got it. SDD is how you say it. TDD is how you check it.

The agent will cheerfully delete your tests to make the red go away. So the spec, the suite, and the judgment behind both aren’t ceremony around the real work. With an agent in the loop, they are the real work.