Why most AI projects never make it out of the demo
The demo works, the pilot goes well, and then it quietly stops being used. The failure almost never happens where people expect it to.
· 9 min read
Short answer: AI projects rarely fail on the model. They fail in the distance between something that works on a chosen example and something that runs unattended against everything, every day, at a cost somebody has to approve. The six recurring causes are below, roughly in the order they bite.
The shape is familiar enough to be a genre. Someone builds a prototype in an afternoon and it is genuinely impressive. A pilot gets approved. The pilot goes fine. Six months later the tool is still technically running, and three people are still using it, and nobody wants to be the one to say so out loud.
What is interesting is where the failure happens. Almost never in the model — the models are good, and getting better at a rate that makes betting against them unwise. It happens in the distance between something that works on a good example and something that runs unattended against everything, including the bad examples, every day, at a cost somebody has to approve.
Six reasons, roughly in the order they tend to bite.
| Cause | What fixes it |
|---|---|
| Workflow never measured first | Count the hours before building anything |
| Quality is a feeling | An evaluation set of real cases with correct answers |
| Data is not where you assumed | Find the source of truth before building retrieval |
| Nobody owns the wrong answer | A human checkpoint wherever an error is expensive |
| Running cost found too late | Model per-run cost at realistic volume, before rollout |
| It was a rules problem | If a person can write the rule down, write the rule |
1. Nobody measured the workflow before touching it
The most common one, and the one that quietly poisons everything downstream. The project starts from “we should use AI here” instead of “this specific task takes eleven hours a week and here is how we know”.
Without the before number there is no after number, so the question of whether it worked has no factual answer and gets settled by whoever is most enthusiastic in the room. That works while the sponsor is paying attention. It does not survive the sponsor moving on.
Spend a week counting first. Sometimes the counting alone kills the project, and that is a cheap way to find out — it is much better to discover in week one that the workflow costs four hours a month than to discover it after building for it.
2. Quality is a feeling instead of a number
A demo shows you the model doing something once, on an example chosen because it worked. Production is the thousandth document, including the one that was scanned at an angle, the one in a second language, and the one where a human wrote a note in the margin that changes the meaning entirely.
What closes that gap is an evaluation set: a few hundred real cases from your own history with the correct answer recorded next to each one. Then accuracy is a percentage. You can see which cases fail and whether they share a shape. You can change the prompt, or the model, or the retrieval, and know within minutes whether you made it better or merely different.
Building this is unglamorous and it is the single highest-return thing in the entire project. Without it, every change after launch is a guess, and the people making the guesses eventually stop because guessing is exhausting.
3. The data is not where everyone assumed
An assistant grounded in company knowledge assumes there is company knowledge to ground it in. What there usually is: a wiki last updated two reorganisations ago, four contradictory versions of the same policy in different inboxes, and the actual answer in somebody’s head.
A retrieval system built over that will confidently return the two-year-old version, because it has no way to know it is stale. Then someone acts on it, and trust in the tool goes to zero in a single afternoon — and trust, once gone, does not come back with a patch release.
Before building the assistant, find out what the source of truth actually is and whether it is written down. Sometimes the project that should happen first is not an AI project at all.
4. Nobody owns the wrong answer
Every one of these systems is wrong sometimes. That is a property of the technology, not a bug to be patched out, and a design that pretends otherwise has one failure mode: surprise.
So the questions have to be asked before rollout rather than after an incident. What happens when it is wrong? Who notices, and how quickly? What does it cost — an awkward email, or a payment sent to the wrong account? Is there a human checkpoint, and does that person have enough context to catch it, or are they clicking approve on things they cannot actually verify?
Where a mistake is cheap and visible, let it run unattended. Where a mistake is expensive or silent, keep a person in the loop and accept the smaller saving. Most disappointing projects picked the wrong one of those two.
5. The running cost showed up after the decision
Prototypes are cheap because they run a handful of times. The same design running against every incoming document, several times per document because it retries and validates, can cost something nobody modelled — and the invoice arrives a month after the decision that caused it.
Model the per-run cost early and multiply it by realistic volume, including retries and the growth you are hoping for. Sometimes the honest conclusion is that the automation costs more than the hours it saves. That is a fine answer to reach in week one and an expensive one to reach in month five.
6. It was a rules problem wearing an AI costume
A surprising share of these projects are trying to do something a lookup table, a form and a database would do — correctly, every time, for almost nothing.
The test is simple: could a competent person write down the rule? If yes, write down the rule. A deterministic system that is right every time beats a probabilistic one that is right most of the time, and it is cheaper to run, easier to explain to an auditor, and it does not need an evaluation set.
Save the models for the part that genuinely requires judgement — reading messy language, handling formats you cannot enumerate, deciding what an unstructured thing is. That part is real, and it is where the value has been all along.
What to settle before you start
None of this argues against the technology. It argues for treating an AI feature as a system that has to be deployed, monitored, paid for and maintained — because that is what it is, and the demo is the part of the work that was already easy.
Five things worth having answers to on day one:
- The hours this workflow costs today, counted rather than estimated.
- A set of real cases with correct answers, to measure against.
- What the source of truth is, and whether it is written down.
- What happens when the system is wrong, and who catches it.
- The cost per run at realistic volume, before rollout.
Projects that can answer those five tend to reach production. Projects that cannot tend to produce an excellent demo, which is not the same thing and never has been.
More on how we approach this on the AI integration page.
Got a pilot that stalled?
Tell us what it does and where it stopped. Usually the fix is one of the six above, and usually it is cheaper than starting again.