One spec in. Six systems out.

In the first smoke run, Claude Code finished its work and then explained, politely, that it had not committed anything, “per your git workflow rules.” We had written no git rules. There are none in the spec. The agent had read the operator's personal configuration files, sitting elsewhere on the machine, and obeyed them.

We amended the method before freezing it: for every run, the operator's configuration moves out of the way, and the run log records that it did. But the incident is worth keeping. The output of a coding agent depends on things nobody wrote in the specification. The machine it runs on is part of the system.

The initial functional review found substantial implementation of the specified business rules, and each system passed its own generated tests. That was an incomplete basis for acceptance. The subsequent security review of these same systems found that none was safe to deploy.

The later review matters to the conclusion: a working demonstration and passing generated tests did not establish secure authorization or production readiness. Read both parts and inspect the per-system records in the public bundle.

Claude Code, three runs, same model, same day:

• Run 1 built authentication on opaque bearer tokens stored in a sessions table.
• Run 2 hand-rolled a signed JWT instead, stateless, with a fallback signing secret of dev-insecure-secret for whenever the real one is missing. One run's draw: the other two runs did nothing similar. It also shipped a small web UI nobody asked for.
• Run 3 used httpOnly session cookies via a library the other two runs never imported.

Three runs, three authentication architectures. A security review of run 1 tells you very little about run 2.

Copilot CLI, three runs, same model, same day:

• Run 1 chose FastAPI and six pinned dependencies, storing uploaded PDFs on disk.
• Runs 2 and 3 chose zero third-party dependencies, building on two different standard-library HTTP servers, and stored the PDFs as binary blobs inside the database.

Same agent, opposite dependency philosophies on consecutive runs. The right-to-erasure endpoint alone appeared three different ways in three runs: POST .../erase-counterparty-contact-email, POST .../erase-contact, DELETE .../contact-email. Regenerate the system next quarter and your integration is against a coin flip.

The whole matrix, one row per run. We did not compare the two agents against each other, and we will not: the finding is about what an agent is, not about which one to buy. Each agent's rows are for reading against its own.

For the same ten requirements, the test suites alone ranged from 7 cases to 18. All of it is defensible engineering, and it is different engineering every time. Nobody chose between the options.

The spec says an invoice carries a PDF attachment. Three of the six systems store that PDF carefully, validate it, record the upload in the audit log, and offer no way to ever get the document back out. The other three added a download endpoint or return it inline. The spec never said the attachment must be retrievable, and a one-page spec cannot close every gap like this: pin the auth, the storage, the dependencies, and the endpoint names, and it is no longer one page; pin everything and it is the implementation. Six systems answered this gap on their own, in both directions, and every one of them believed it was done.

Five of the six within-agent run pairs were materially different systems, by a bar we set in writing before the first run. The sixth pair, two runs that both took the zero-dependency route, came in just under it. The bar and its six criteria are in the published method.

We ran the same two scanners over all six systems: Semgrep on its default rules, and gitleaks for secrets.

In the initial scanner pass, Gitleaks reported no secrets and the six Semgrep errors were assessed as false positives in parameterized query construction. Neither tool caught the hardcoded fallback JWT secret identified by manual reading. That was an initial finding, not the only security issue in the experiment: the later review found serious authorization and data-access failures.

These results describe the chosen scanners, versions and configuration on this sample. They do not establish that scanners are useless or that one manual reading was complete. The subsequent review is part of the evidence and changes the acceptance conclusion.

Our own first static read of one system claimed approvers could submit invoices. Running the system proved the read wrong; the API refuses with a 403. Reading it had not been enough.

Implementation variance and security defects are separate findings. Different architecture choices are not necessarily defects, but the later review found defects in these systems that prevented safe deployment. Repeating the same specification produced different code; each result still needed its own security review.

Anyone who works with these models will shrug at that: sampling varies, everyone knows. Everyone knows, and nothing prices it in. Human teams vary too, but nobody re-hires a fresh team every quarter to rebuild the same service; with agents, regeneration is cheap enough that it is becoming routine, and every regeneration is a new draw with no review attached.

On the day the code ships, the variance is invisible. Every one of our six runs would have demoed identically: log in, submit an invoice, approve it, done.

The variance surfaces later, where it is expensive. Your threat model depends on which of three authentication architectures you happened to draw. Your data-protection assessment depends on whether personal data sits in a file on disk or a blob in the database. Your supply-chain review depends on whether you got six dependencies or none. And the review you did on the last system is not evidence about the next one.

That is the question the demo never answers: not “does it work,” but “which system did you get, and can you prove it?”

Taiga does not promise identical generated code. Its documented workflows connect project context, initiatives, implementation and review. The current capability reference states the prerequisites and limits; the agreed scope and the evidence for the exact run matter more than a general claim of repeatability.[7]

Three runs per agent, one specification, one day, two products that update monthly. This is an observation with a published method, not a study, and we are not dressing it up as one.

It is also not neutral. We sell the thing this experiment argues for, which is exactly why the method, the spec, and the logs are public. Distrust us; check the bundle.

It is fully checkable. The frozen spec, the acceptance checklist, the runner script, the run logs, and the per-system results are published at github.com/mikromikko/one-spec-six-systems. Rerunning the whole thing costs an afternoon. If your diff comes back boring, publish it. We would like to see that result, and we mean that.

And if you are evaluating anyone's AI-built software, ours included, the experiment suggests three questions that cost the vendor nothing to answer:

1. Show me the diff between two runs of the same input. If they cannot produce two runs, that is the answer.
2. Which draw is the one running in production, and what evidence attaches to that draw specifically, not to the tool that made it?
3. When you regenerate, what is guaranteed to stay the same, in writing?

For Taiga too, require the agreed deliverables and the actual checks and review records for your system. The current capability reference describes supported workflows and limits; a published checklist is not proof that a particular output passed it.

The next time someone shows you what an agent built, ask what it would have built the second time. One spec in. Which system out?