How do you know when it is safe to let an AI system run on its own?

You do not know in advance, and no amount of specification will tell you. You find out by running the system in three stages — first watching it, then approving it case by case, then letting it go — and by measuring one thing throughout: of everything the system would have let through unsupervised, how much a person had to change. That number, read next to two others, is what turns "are we comfortable with this?" into a decision someone can sign.

The bottleneck moved. Most projects did not.

Before LLMs, the arithmetic of a software project was easy. Specifying took two months, building took twelve. Spending those two months on a real PRD — user stories, acceptance criteria, edge cases — was obviously worth it, because a mistake found in month ten cost ten times what it cost in month two.

That arithmetic is gone. Implementation now takes a fraction of what it used to, which is mostly good news and has one consequence people are slow to absorb: the expensive part of a project is no longer building the thing. It is knowing what the thing should do.

Which would be fine if that part had got easier too. It has not.

Why the hardest thing to describe is work people already do

Writing a complete specification for an ordinary application is already hard — anyone who has tried to enumerate every screen and every failure state knows how much you miss. Now change the job. You are not designing something new. You are replacing work that a person, or a coordinated team of people, does today: a back office that has run for fifteen years, an inbox someone opens every morning, a supplier-invoice desk.

Ask the person who does that job to write down the rules and you will get a page and a half. Then sit next to them for a day and count forty decisions that were not on the page. Not because anything was hidden — because they do not experience those as decisions. The supplier who always sends the delivery note separately. The customer who puts the PO number in the wrong field, and everyone knows it. The email marked urgent, from the person whose emails never are.

That knowledge is real, it is load-bearing, and it is written nowhere. It lives in hands, not documents. The processes worth automating are precisely the ones people run on autopilot — and autopilot is exactly the state in which humans are worst at describing what they do.

Why more discovery does not fix it

The instinct is to buy more analysis: more interviews, more workshops, a longer document. But you would spend more eliciting the specification than building the system, and the answers that matter only surface when someone is looking at a real case with a real deadline, not at a whiteboard.

And you would still be wrong, without knowing which parts. A specification produced entirely by interview is a hypothesis wearing the clothes of a requirement. So we stopped trying — not by skipping the analysis, but by moving where the analysis happens.

What we ship first: a walking skeleton

We treat the project like a startup idea rather than a construction contract. Study enough to be confident about two things — the logical flow, meaning what happens in what order on what trigger, and the architecture, meaning which components exist and which systems they touch. Both at high level. Enough to be right about the shape, not the details.

Then build the thinnest version that runs end to end. The term comes from Alistair Cockburn, and in its strict sense a walking skeleton is a technical proof rather than a product: every architectural component linked and talking — interface, logic, integrations, data, and the deployment pipeline that puts it there — carrying almost no features. It answers "does this chain hold together at all?" before anyone has spent six months assuming it does.

We push it one step further, because it costs almost nothing to. The narrow slice we run through the skeleton is a real case type, from the real source, to a real output — so the same artefact that proves the architecture is also the first thing operators can look at and argue with. Everything outside the slice fails loudly and goes to a human.

Note what the table does not claim. The skeleton does not make the project shorter or the team faster. It moves the first contact with reality from month six to month two, because it converts an unanswerable question into an answerable one: "what should the system do?" has no good answer in a meeting room, while "look at what it did with this case — is that right?" gets a precise answer in four seconds, from someone who could never have stated the rule they are applying.

Why the extra correction points matter more here than elsewhere

The obvious objection is that this is just iterative development, and people have been arguing for it since long before anyone had an LLM. Fair. The argument is older than the technology. What changed is how much now rides on it.

Iterating is how you steer, and steering has a hard limit that has nothing to do with effort: you cannot correct a course more often than you can observe it. Three observations clustered in the last quarter of a project is not steering, it is a post-mortem with budget left for two patches. And the cost of finding out late is not linear — a rule you got wrong in month one and discover in month six has had five months of work laid on top of it. The correction is cheap; everything that assumed it is not.

What changes with AI is that you now have two unknowns instead of one. In ordinary software the uncertainty sits in the requirements: once the code is written it does what you wrote, so you may have built the wrong thing but you know what you built. Put a model in the loop and the second half stops being true. How the system behaves on the long tail of real inputs is not derivable from the specification, and not from the code either. It is only observable. A perfectly specified system will still do something you did not predict on the input nobody thought of.

So you are navigating toward a target you cannot see, using an instrument whose behaviour you also cannot fully see. Both unknowns shrink the same way — by running the thing and looking at what came out — and a process that hands you three looks is not underpowered by a margin. It is the wrong shape for the problem.

Shadow, human in the loop, full auto: the three modes

Shipping early into a live process is also an excellent way to break a business. So the skeleton does not go live connected. Every system we build runs in one of three modes, and moves through them in order.

Shadow mode. The system reads live inputs and produces real outputs, and the outputs go nowhere. It does not write to the ERP — it writes to staging, or only to the panel. It does not send the email — it drafts it. It does not open the ticket — it records the ticket it would have opened, with the fields it would have used. Operators work as before and review the shadow output once a day. This is where the forty undocumented decisions come out, one at a time, each attached to a real case.

Human in the loop. Now the system is connected to the real destination, but it waits. "I am about to do this — here is the data, here is why." A person confirms, edits or cancels; nothing lands in production without a click. This is where the workload actually starts to shrink, because a job that took forty minutes of gathering and typing becomes a review that takes forty seconds.

Full auto. The system acts on its own. It is the mode everyone wants and the one that needs the most infrastructure behind it, because "on its own" never means unattended. Two things stay in place: the validator, and a sample that keeps going to humans forever. Both below.

What a validator is, and whether it should use AI

Every system we build has a validator: a component whose only job is to answer one binary question about each output — can this go out without a human?

It is deterministic wherever it can be. Ordinary code, ordinary rules. That is deliberate: the thing it checks is already non-deterministic, and putting a second probabilistic component in charge of judging the first leaves you with two things to distrust instead of one.

Plain code checks more than people expect. That every required field is populated. That the total matches the sum of the lines. That the supplier the model picked exists in the master data and matches the VAT number on the document. That the amount sits under the ceiling this process is allowed to touch. That the account in the reply is the one the request came from. That no policy rule was triggered. That nothing in the output contradicts the source it was built from.

Where it stops, honestly: on structured outputs — a posting, a ticket, a record write — deterministic checks cover essentially everything that matters. On generative outputs, such as a reply written to a customer, they cover the verifiable invariants (no facts outside the source, no commitment the system is not allowed to make, no forbidden content) but not the whole of "is this a good answer". Where the invariants do not reach, we say so, and that class of output stays supervised for longer. A validator that pretends to judge quality it cannot measure is worse than one with a stated blind spot.

Human in the loop is where the numbers come from

All three modes matter, but they do not matter equally, and the middle one is doing something the other two cannot. This is the hinge of the whole method, so it is worth being exact about it.

Every system we build ships with an operator panel from day one. It is where the system shows what it did — or would have done — with its reasoning and the source data beside it, and where a person corrects it. In shadow mode that panel is a review queue someone opens when they have a spare half hour: useful, but the coverage is partial and the feedback is closer to opinion than to record.

In human-in-the-loop the panel stops being a review queue and becomes the job. Every invoice, every ticket, every reply goes through it, because that is now how the work gets done — nothing reaches production without someone clicking. Which means that for the entire length of the phase, every single execution carries two verdicts: what the validator said about it, and what the person then did. Full coverage, at production volume, from the people accountable for the outcome. That is not a survey of a sample. It is a census.

And two verdicts on every case is exactly, and only, what the matrix in the rest of this article is made of. The supervised phases are not just a safety ramp you tolerate before the good part — human-in-the-loop is the measurement, and the panel is the instrument that takes it. The operators are not doing extra work to help you evaluate the system. They are doing their job, and the evaluation is the by-product.

This is also why the panel cannot be phase two. Teams that defer it end up with a system nobody can evaluate, which in practice means a system nobody will authorise to run. Budget for it as part of the system, because it is part of the system.

A validator is a screening test

Before the measurement, the thing being measured. The validator's threshold is a dial, not a fact. Turn it permissive and more work goes out untouched: more automation, and more errors reaching production. Turn it strict and almost nothing reaches production unreviewed, but your operators are back to reviewing everything and the automation you paid for delivers a fraction of what it could.

The useful way to think about that dial is not a metaphor. It is the same mathematics. The validator is a screening test; the condition it screens for is "this output is wrong"; a positive result means "send it to a human". And everyone's intuition about screening tests is already correct. A test that tells everybody they are ill catches every case — perfect sensitivity — and is worthless, because it has no specificity and you have just referred the entire population to a specialist. A test so strict that it only fires on the obvious cases has excellent specificity and misses most of the ill. Neither is a good test.

And no test is perfect. You can improve one, you can never finish it. The work is not eliminating errors: it is deciding where the threshold sits, knowing what each kind of error costs. Most teams set that by feel, after a demo that went well. It is fixable, because the dial has numbers behind it.

The four outcomes

Two binary facts about every case. What the validator said: cleared, or needs review. What the person then did: accepted it, or changed it. Cross them and you get four cells — and everything you need in order to decide.

Two notes that keep this honest. A cancelled output counts as changed: if a person had to stop it, the system got it wrong. And "changed" is not always "wrong" — sometimes an operator rewrites a perfectly correct sentence because they would have phrased it differently. Have the panel distinguish substantive edits from cosmetic ones, or you will end up tuning the dial against your own house style.

Three numbers, not two

Automation risk — cleared & changed ÷ everything cleared. Of everything the validator waved through, how much turned out to be wrong. This is the number you are betting on the day you switch to full auto. In screening terms it is the complement of negative predictive value: when the test says clear, how much can you trust it?

Catch rate — flagged & changed ÷ everything changed. Of everything that turned out to be wrong, how much the validator flagged. This is sensitivity, and it is the honest measure of the validator on its own — it does not improve just because the model underneath got better.

Review rate — everything flagged ÷ everything. How much human time you are still paying for. It says nothing about quality, and without it the other two will lie to you. The next section is entirely about why.

Worth pausing on one property of the first number: it does not depend on the validator alone. Negative predictive value moves with prevalence — in plain terms, automation risk depends both on how good the validator is at flagging and on how often the system underneath is wrong in the first place. They are two different levers on two different numbers. Improving the validator raises the catch rate. Improving the system itself shrinks the whole review row. Teams that treat them as interchangeable spend months tuning the wrong one.

One good matrix, one useless one, and one that looks bad

Three shapes you will recognise once you start collecting this. The numbers below are illustrative — 600 cases each, so they can be compared directly.

A — most work goes through untouched, and when something is wrong the validator usually catches it. Whether 4% risk is acceptable is not an engineering question — but the question can now be asked seriously.

B — the validator sends everything to review. On the two headline numbers this is the best system on the page. It has automated nothing: your operators are doing exactly what they did before. Without the review rate, you cannot see it.

C — four cases in five still reach a person, but 390 of those 476 reviews are confirm-and-move-on. If the job used to take an hour, you have not automated 21% of the work: you have automated most of the work in every case, and kept a person accountable. Plenty of systems should stay here on purpose.

B is the one to be able to spot. On automation risk and catch rate it is the best system on the page, and it has automated nothing at all. That is what a validator tuned by anxiety looks like, and the third number is the only thing that shows it.

Working the matrix on one process, end to end

Abstract shapes only get you so far. Here is the whole loop on a single process: supplier invoices arriving by email. The shape is what we see in this kind of work; the numbers stay illustrative.

Today a person opens each invoice, finds the matching purchase order and delivery note, checks quantities and prices against them, and posts it to the ERP. Six hundred invoices a month, about six minutes each. The system does the same steps, and the validator checks what plain code can check: every required field present, line items summing to the stated total, the supplier in master data with a VAT number matching the document, a purchase order found and a delivery note referencing it, quantities and unit prices inside the contracted tolerance, bank details identical to the ones on file, total under the ceiling this process may post. Any failure sends the invoice to a person.

Eight weeks of human-in-the-loop later — every invoice through the panel, every invoice carrying two verdicts — the matrix looks like this.

The move between those two rows is the part people skip. You do not read the 18 as a score. You read the eighteen invoices. Eleven of them turn out to be the same thing: partial deliveries, where the delivery note covers fewer units than the invoice bills, because the tolerance rule was comparing the invoice against the purchase order rather than against what actually arrived. That is not a model problem. It is a missing rule, and it takes an afternoon. The bad cell is a backlog, not a verdict — and it is only legible because the cases were kept, not just the totals.

Then the decision, which is not ours to make. Is one wrong posting in sixty acceptable? For a payment to a supplier, no: a wrong one costs the payment, the recovery and the relationship. So the process splits. Invoices under €2,000, from suppliers with six months of clean history and an exact three-way match, go full auto — and before switching, you recompute the same three numbers on just that slice, which is the other reason to keep the raw cases. Everything above the threshold, everything from a supplier seen for the first time, and everything where the bank details changed since the last invoice stays supervised, permanently.

The finance team still sees roughly 40% of the invoices — but as a review in the panel, with the matching already done, not as six minutes of gathering. Nobody switched anything on because a demo went well.

What happens to the numbers once you switch to full auto

Here is the awkward part, and it is usually hand-waved. The matrix needs both columns, and the second one exists only if a person actually looked. The moment you switch to full auto, nobody looks at the cleared row any more. Catch rate and review rate survive. Automation risk — the one number you are actually betting on — becomes unobservable exactly when you have started betting on it. It does not get worse; you just stop being able to see it. Given that models get updated, suppliers change their formats and the mix of incoming cases drifts, that is not a state to be in.

The fix is a permanent random sample:

Everything else is downstream evidence: records corrected by hand afterwards, complaints, reopened tickets. Worth watching, but it only ever finds the errors somebody happens to notice, and it finds them late. The sample is the measurement; the downstream signals are the backstop.

Who decides when to go full auto?

Not us. We can tell you what each setting buys: at this threshold you automate 70% of the volume and one in fifty of those is wrong; at this one you automate 40% and it is one in five hundred. Which of those you want is not an engineering question. It is a question about what a wrong output costs you, and you are the only one who knows. A wrong FAQ answer on your website costs a mildly annoyed visitor; the invoice above costs a payment. Same architecture, same validator, entirely different dial.

And as that example showed, a single threshold for a whole process is rare. What usually happens is that the process splits, and automation turns out to be not a switch you flip on the system but a set of taps you open one case class at a time, starting where being wrong is cheap. It is also the honest answer to "when will it be fully automated": for most processes, never entirely, and that is the right outcome rather than a shortfall.

What the method costs

The fair objection: a panel, a validator and weeks of supervised running is more work than just building the thing. It is. It is also considerably less work than specifying it correctly first, which you could not have done anyway. And the comparison most people miss: the supervised phases do not merely protect you while you learn, they produce the specification. The forty undocumented decisions arrive attached to real cases, with the right answer beside them, in order of how often they occur — while the system is already running.

The panel and the validator are permanent assets, too. They are what lets you swap a model, add a case type or widen the scope next year and know within a week whether it still holds. Systems built without them do not get extended. They get replaced.

Which of your processes could survive this?

Start with the free AI Audit: five minutes of questions and you will know how AI-ready your processes are and which ones are worth putting through this loop first. Then book a call and we will walk through the ones that matter.