One spec in. Six systems out.

In the first smoke run, Claude Code finished its work and then explained, politely, that it had not committed anything, “per your git workflow rules.” We had written no git rules. There are none in the spec. The agent had read the operator's personal configuration files, sitting elsewhere on the machine, and obeyed them.

We amended the method before freezing it: for every run, the operator's configuration moves out of the way, and the run log records that it did. But the incident is worth keeping. The output of a coding agent depends on things nobody wrote in the specification. The machine it runs on is part of the system.

All six systems implemented every hard rule in the spec correctly: the self-approval prohibition, the dual-approval threshold, the append-only audit log, the logged reads and erasure of personal data. All six even converged on the same reading of the spec's trickiest sentence: “over 10,000 euros” means strictly greater than, so an invoice of exactly 10,000 needs one approval. We verified that boundary by hand on every system. All six stored money as integer cents. All six, independently, enforced append-only on the audit log with database triggers.

And all six started from their README alone and passed their own test suites. Against our ten-point acceptance checklist, executed identically on every system, the worst score was nine out of ten.

The agents are good.

Claude Code, three runs, same model, same day:

• Run 1 built authentication on opaque bearer tokens stored in a sessions table.
• Run 2 hand-rolled a signed JWT instead, stateless, with a fallback signing secret of dev-insecure-secret for whenever the real one is missing. One run's draw: the other two runs did nothing similar. It also shipped a small web UI nobody asked for.
• Run 3 used httpOnly session cookies via a library the other two runs never imported.

Three runs, three authentication architectures. A security review of run 1 tells you very little about run 2.

Copilot CLI, three runs, same model, same day:

• Run 1 chose FastAPI and six pinned dependencies, storing uploaded PDFs on disk.
• Runs 2 and 3 chose zero third-party dependencies, building on two different standard-library HTTP servers, and stored the PDFs as binary blobs inside the database.

Same agent, opposite dependency philosophies on consecutive runs. The right-to-erasure endpoint alone appeared three different ways in three runs: POST .../erase-counterparty-contact-email, POST .../erase-contact, DELETE .../contact-email. Regenerate the system next quarter and your integration is against a coin flip.

The whole matrix, one row per run. We did not compare the two agents against each other, and we will not: the finding is about what an agent is, not about which one to buy. Each agent's rows are for reading against its own.

For the same ten requirements, the test suites alone ranged from 7 cases to 18. All of it is defensible engineering, and it is different engineering every time. Nobody chose between the options.

The spec says an invoice carries a PDF attachment. Three of the six systems store that PDF carefully, validate it, record the upload in the audit log, and offer no way to ever get the document back out. The other three added a download endpoint or return it inline. The spec never said the attachment must be retrievable, and a one-page spec cannot close every gap like this: pin the auth, the storage, the dependencies, and the endpoint names, and it is no longer one page; pin everything and it is the implementation. Six systems answered this gap on their own, in both directions, and every one of them believed it was done.

Five of the six within-agent run pairs were materially different systems, by a bar we set in writing before the first run. The sixth pair, two runs that both took the zero-dependency route, came in just under it. The bar and its six criteria are in the published method.

We ran the same two scanners over all six systems: Semgrep on its default rules, and gitleaks for secrets.

Gitleaks found nothing anywhere. Semgrep raised six errors, and on inspection all six were the same false positive: a query built from fixed clause fragments with properly parameterized values, flagged as raw SQL. Three separate runs of one agent had produced the identical query-building idiom, and the scanner tripped on it identically each time. Meanwhile the one genuine security issue in the entire experiment, that hardcoded fallback JWT secret, was caught by neither tool. We found it by reading the code.

On this sample, the standard tooling scored six false alarms, zero real findings. The real finding took a person reading the code. We are not generalizing from one afternoon; we are saying the afternoon is easy to repeat, and you should repeat it on your own output before assuming your scanners have the variance covered.

Our own first static read of one system claimed approvers could submit invoices. Running the system proved the read wrong; the API refuses with a 403. Reading it had not been enough.

None of the above is a defect. It cannot be fixed by a better model, because it is not a flaw in the model. A coding agent samples one plausible system from the distribution of systems that satisfy your prompt. Run it again and you draw again. All six draws were competent. They were not the same system.

Anyone who works with these models will shrug at that: sampling varies, everyone knows. Everyone knows, and nothing prices it in. Human teams vary too, but nobody re-hires a fresh team every quarter to rebuild the same service; with agents, regeneration is cheap enough that it is becoming routine, and every regeneration is a new draw with no review attached.

On the day the code ships, the variance is invisible. Every one of our six runs would have demoed identically: log in, submit an invoice, approve it, done.

The variance surfaces later, where it is expensive. Your threat model depends on which of three authentication architectures you happened to draw. Your data-protection assessment depends on whether personal data sits in a file on disk or a blob in the database. Your supply-chain review depends on whether you got six dependencies or none. And the review you did on the last system is not evidence about the next one.

That is the question the demo never answers: not “does it work,” but “which system did you get, and can you prove it?”

A factory is built to make that question boring. Not because its output is bit-identical: ours is not either, and anyone promising identical code from a stochastic process is selling something. Taiga's claim is narrower and published: every run produces the same set of artifacts, checked against the same 22-check bar[7], with every step on the record. Agents improvise. Factories repeat: the deliverable set and the bar it must pass, never the bits.

Three runs per agent, one specification, one day, two products that update monthly. This is an observation with a published method, not a study, and we are not dressing it up as one.

It is also not neutral. We sell the thing this experiment argues for, which is exactly why the method, the spec, and the logs are public. Distrust us; check the bundle.

It is fully checkable. The frozen spec, the acceptance checklist, the runner script, the run logs, and the per-system results are published at github.com/mikromikko/one-spec-six-systems. Rerunning the whole thing costs an afternoon. If your diff comes back boring, publish it. We would like to see that result, and we mean that.

And if you are evaluating anyone's AI-built software, ours included, the experiment suggests three questions that cost the vendor nothing to answer:

1. Show me the diff between two runs of the same input. If they cannot produce two runs, that is the answer.
2. Which draw is the one running in production, and what evidence attaches to that draw specifically, not to the tool that made it?
3. When you regenerate, what is guaranteed to stay the same, in writing?

Our own answer to the third question is the published 22-check gate and the fixed artifact set, and you should hold us to exactly this standard.

The next time someone shows you what an agent built, ask what it would have built the second time. One spec in. Which system out?