Resources

Ten questions to ask an AI development vendor

Standard vendor due diligence audits the vendor's organization. AI moved the risk into the deliverable: which system the agents actually built, and what evidence attaches to it. These ten questions audit the delivery itself. Copy them into your next RFP.

Somewhere in your organization is a vendor security questionnaire a few hundred rows long. It asks whether the vendor encrypts its laptops. Who can enter the office. Whether admin access is reviewed quarterly, whether staff sign NDAs, how passwords are managed. It is a serious instrument, and if a vendor fails it, you are right to walk away.

It does not ask which system the vendor's coding agents actually built for you. When those questionnaires were designed, the question did not exist.

It exists now. In July 2026 we froze a one-page specification and gave it to two coding agents, three times each. Six working systems came back: three different authentication architectures, two opposite dependency philosophies, personal data in a file on disk or in a database blob depending on the run (the experiment). Every one of them would have passed the same demo. And in DryRun Security's March 2026 study, 26 of 30 AI-agent pull requests introduced at least one vulnerability; the study ranks the individual agents too, and the ranking matters less than the rate. The software you are buying was sampled from a distribution. The questionnaire audits the org chart.

The proxy broke

To be exact about what the standard instruments do: the SIG asks a vendor organization about its controls. The CAIQ does the same for cloud services. An ISO 27001 certificate says accredited auditors verified that a management system exists and is followed. These are good tools, and nothing below replaces them. They answer the question they were built for: is this a serious company. The newest editions have begun asking about AI, too. CSA published an AI version of the CAIQ in 2025; it asks how an organization governs its AI, which is still a question about the organization, not about the system the agents handed you.

For two decades the organization was a workable proxy for the work itself. A disciplined organization produced roughly disciplined code, at human speed, through one process, and auditing the process was as close to auditing the output as anyone needed to get. Generation by agents broke the proxy. In our experiment, nothing organizational changed between runs of the same agent, and the systems still diverged exactly where a compliance review cares: authentication, where the data sat, how many dependencies came along. A security review of one run says very little about the next one.

So due diligence needs a second section, aimed at the delivery. Here are the ten questions we would put in it, with what to listen for. None of them asks the vendor to reveal anything proprietary. Each asks for something that either already exists or does not. Our own answers to all ten are at the end.

Before the code

1. What existed before the build started? Show me the threat model and the DPIA, with their dates.

The point of this one is the dates. A threat model written after the code protected nothing while the code was being written, and a DPIA assembled for your audit is a costume. An auditor reads the dates first; so should you. One precision your DPO will raise, and they are right: the DPIA is formally yours to own as the controller. What the vendor owes you is the dated assessment that feeds it.

A good answer
the same dated artifact set on every engagement, produced before the first commit, not heroics for the deal.
Watch for
"we can prepare those." Documentation that can be prepared afterward is being prepared afterward.
Bonus
the artifact set is published before you ask for it.

2. Pick one change from last quarter. At random. Who made it, who approved it, when, and why?

You do the picking; any vendor can curate one good example. What the random pull tests is whether a record exists at all, and if you are a financial entity the stakes are set before you ask: DORA already makes change evidence your problem, and a vendor who cannot produce it hands you governance debt to carry on their behalf.

A good answer
a row from the audit trail: author, approval, timestamp, recorded at the time of the change. When an agent wrote the change, the record says so. The approver does not have to be human: an autonomous approval counts, provided the decision ran against a written check and the record names the check it followed.
Watch for
"let me check with the team." What comes back later is memory, not a record. And watch the other direction too: one human name approving every agent change at machine volume is a ceremony, not a control.

Which system did you get

Three of the ten we published with the variance experiment, aimed at anyone evaluating AI-built software, ours included.

3. Run the same input twice. Show me the diff.

If they cannot produce two runs, that is the answer. This is the experiment question, and it can be run from a chair: when we froze one page of specification and ran it six times, the systems came back three authentication architectures wide. The diff between two runs of this vendor's process shows you their version of that spread, in their own output, with no slideware in between.

A good answer
two runs and a diff they can walk you through, including what they promise stays fixed between draws.
Watch for
"our output is deterministic." From a stochastic system, that claim is either untrue or unexamined.
Bonus
the diff, the method, and the runs are published.

4. Which run is in production for me, and what evidence attaches to that run, not to the tool that made it?

Every review your team runs attaches to a specific system. The threat model describes one architecture. The data-protection assessment describes one set of storage decisions. The supply-chain review covers one dependency list, and in our experiment that list went from six dependencies to zero between two draws of the same agent. Evidence about the tool, however polished, tells you what the tool can do. It does not tell you what you got.

A good answer
the evidence set for your run: its threat model, its data locations, its dependency list.
Watch for
benchmarks, certifications, or scan results about the tool.

5. When you regenerate the system, what is guaranteed to stay the same, in writing?

Bit-identical output from a stochastic process is not on offer, from us or anyone, whatever the slide deck implies. What is on offer, from a serious vendor, is a scope: a written line around what holds when the system is regenerated. Regeneration is cheap now, so it will happen, and every regeneration is a new draw. Without that line, the system you reviewed and the system you run drift apart with nobody deciding it.

A good answer
the scope in writing: which artifacts, which checks, which interfaces hold between runs. Narrow is fine; unwritten is not.
Watch for
"the output will be equivalent," with nothing signed.

The bar

6. What must every delivery pass before it reaches me, and can I read it before I sign?

Somewhere in this conversation the vendor will say "quality." This question converts the word into a document. A bar you cannot read is a bar you cannot hold anyone to, and an unpublished bar can move between deliveries without anyone deciding that, either.

A good answer
a fixed, published, versioned list of checks where failure blocks the delivery, readable before the contract.
Watch for
"industry best practices." Not readable, so not enforceable, on the vendor or by you.
Bonus
the bar is public on their website, not shipped as a PDF on request.

7. Show me one delivery that failed a check, and what happened next.

The counterintuitive one, and the fastest. A vendor operating a real gate accumulates failures on the record, because catching failures is what a gate is for. A gate that has never failed anything is a decoration, and a vendor reluctant to show you a failure is telling you which of the two they run.

A good answer
one failure, its fix, and its re-verification, all recorded at the time.
Watch for
"our process prevents failures." That is a description of not having a gate.
Bonus
failures propagate backward: when one teaches the vendor something, deliveries that predate the lesson get re-checked, not just the next one.

The last three

8. Where does my data live during the build, and what enforces that: a policy or a control?

A policy is a document. A control is what happens when an engineer tries anyway, and who finds out. Your own infrastructure team learned that distinction the hard way years ago; this question applies it to the vendor's.

A good answer
a named mechanism, enforced at the level of the whole organization, provable to your security team.
Watch for
"our policy prohibits transfers." Ask the follow-up in the header.

9. If you disappeared tomorrow, could my team rebuild and run this from the handover alone?

Procurement already asks this about licenses; point it at the artifact. An AI development vendor is young by definition. Your system should not be, and the handover is where that gets decided, not the day the vendor's phone stops answering.

A good answer
the environment rebuilds from the repository alone, the documentation matches what shipped, and the rollback path is tested and versioned with the delivery.
Watch for
"we offer support contracts." That answers a different question.

10. Which model providers sit underneath you, and what of mine reaches them?

Your vendor runs on someone else's models, and everything the agents read on your engagement travels toward those models at inference time: your specification, your code, your data. That puts the model provider in your processing chain, a sub-processor wherever personal data flows and a confidentiality exposure everywhere else. Under DORA this is subcontracting, and it belongs in your register of information. The stack can stay private. The terms governing your data cannot.

A good answer
the chain named in writing: which providers, what reaches them at inference time, under what terms, and whether any of it is retained or used for training.
Watch for
a platform name with no terms attached, or "we cannot disclose our stack."

Our answers, on the record

We sell governed AI software development to regulated enterprises, so these ten will be pointed at us first, and they should be. Here is our exam, taken in public:

Nine artifacts on every delivery, dated before the build: specification, user flows, architecture, technology decisions, data flow, threat model, DPIA, risk register, service blueprint.
Every change is in an audit trail kept immutable for seven years.
Published: the frozen spec, the run logs, and the per-system results are in the experiment bundle.
Evidence attaches per run: each delivery ships with its own artifact set from answer one, checked on that run.
In writing: every run produces the same artifact set, checked against the same published bar.
The bar is 22 checks across governance, security, build, and run, public at /gate before any contract.
On the record: initiative INI-021 in project XRM, a build that failed, was fixed, and passed re-verification; we show it in the demo.
A control: cloud regions outside the EU are denied at the organization level; the system description is available to your security team on request.
Checked on every delivery by the BUILD section of the same 22-check bar: rebuild from the repository, documentation matching what shipped, a tested rollback path.
Model calls run through AWS Bedrock, inside the same EU perimeter where non-EU regions are denied. Under the platform's published terms, nothing we send is used to train the models and the model providers never receive it (AWS, Bedrock FAQ). The model list lives in the system description, because it changes.

The first question your existing questionnaire would ask us, we answer badly: we hold no certifications today. ISO 27001 is in progress and SOC 2 Type II is planned, dated June 2026 on our receipts page. An impressive certificate would only have answered the organization question; the ten above are where the delivery gets audited. A vendor who answers the certification question honestly and loses the deal is cheaper than one who answers it optimistically and meets your auditor.

Some questions belong on a different list: IP indemnity for generated code, incident notification terms above the statutory floor, liability, insurance. That list is the contract lane, and your legal team owns it; nothing above substitutes for it.

Copy the ten into your next RFP; no attribution needed. There is a plain-text version made for pasting. And if you want to know how your own last AI project would score first, the self-test takes six answers. The checklist's own rule applies here too: not sure counts as no. An auditor counts it the same way.

Sources

Frequently asked questions

What questions should you ask an AI software development vendor?+

Ask for evidence about the delivery, not the organization: dated pre-build artifacts (threat model, DPIA input), the audit trail for a randomly chosen change with its approval, a diff between two runs of the same input, the evidence attached to the run in production, a written scope of what regeneration keeps stable, a readable pass/fail bar, one recorded failure, the mechanism enforcing data residency, whether the system survives the vendor's disappearance, and which model providers sit underneath the vendor.

Do ISO 27001 or SOC 2 certificates cover AI-generated code?+

No. They certify that an organization's management system exists and is followed. They attach to the company, not to a specific deliverable. Evidence about the system an AI agent produced for you, its threat model, its audit trail, its pass/fail record, has to be asked for separately, per delivery.

What is the biggest red flag in AI development vendor due diligence?+

A perfect record. A vendor operating a real quality gate accumulates recorded failures, because catching failures is the gate's job. A vendor who cannot produce two runs of the same input, or whose bar cannot be read before signing, is telling you the evidence does not exist.

Why ask an AI vendor to run the same input twice?+

Because coding agents sample from a distribution of possible systems. In a published July 2026 experiment, one frozen specification produced six working systems with three different authentication architectures. The diff between two runs shows what actually varies in the vendor's process, and reviews attach to a specific run, not to the tool.

Are AI model providers sub-processors of an AI development vendor?+

Treat them as part of the chain: at inference time the vendor's agents send your specification, code, and context to whatever sits underneath the vendor, a model provider directly or a platform in front of one. Under DORA that chain is subcontracting and belongs in the register of information. Ask which providers are in the chain, what reaches them, under what terms, and whether anything is retained or used for training.

Run the six-question self-test

Bring the project this article made you think about.

Book a demo