There is a specific kind of confidence that precedes an undefined success criterion. The team is experienced. The problem is understood. The data is available. The technology is proven. What success looks like is so obvious to everyone in the room that writing it down feels like unnecessary formality — the kind of thing that slows projects down rather than moving them forward.
This is the condition Joyce Weiner identifies as the single most common precursor to the first failure mode in AI and data science projects: the scope was too big to define, or the definition was deferred until the team had something to show. Either way, the result is the same — a project that proceeds without a formal, agreed, measurable definition of what it is trying to achieve, and therefore without any mechanism to know when it has arrived or why it hasn't.
The UK Home Office's Visa Streaming Tool is the documented form of this failure at government scale.
What the Visa Streaming Tool review found
The Visa Streaming Tool was deployed by the Home Office to assist caseworkers in prioritising visa applications for manual review. It assigned a stream — a priority category — to each application based on risk signals in the data. The tool was in use for several years before an independent review was commissioned in 2020 following concerns about its impact on applicants from certain nationalities.
The review, conducted by independent legal and technical reviewers and published in 2020, found multiple structural problems. Among the most significant: the system's success criteria had never been formally defined. The tool had been built, deployed, and used by caseworkers without a documented, agreed statement of what good performance looked like, what an incorrect output looked like, or what threshold of error would constitute grounds for suspension or review.
The system was ultimately suspended pending further review. The suspension was not triggered by measured underperformance against a defined standard. It was triggered by external scrutiny — because no internal standard existed against which performance could be measured.
"The review found that the tool had been introduced without a formal equality impact assessment and without a documented specification of the outcomes it was intended to produce. In the absence of a defined success criterion, it was not possible to evaluate whether the tool was performing as intended, or to identify the threshold at which its outputs would be considered sufficiently unreliable to warrant suspension."
The Weiner framework identifies this pattern as the first failure condition: scope too large, or scope undefined. The practical expression of undefined scope in an AI project is a system that cannot be evaluated — not because evaluation is technically difficult, but because no one agreed on what evaluation would be measuring.
Why "we'll know it when we see it" is a deployment block
The phrase does not appear in meeting notes. It appears in the structure of the project plan — in the absence of a formally agreed acceptance criterion, in the deferral of the success definition to the demo, in the assumption that the review board will recognise good performance when it sees the system running.
The problem is not that experienced practitioners cannot recognise good performance. They can, in general. The problem is that recognition is not the same as a defined standard — and a defined standard is what allows a system to be evaluated, challenged, improved, and sustained by anyone other than the people who built it. Without a defined standard, the system's ongoing performance is assessed by impression, not by measurement. And assessment by impression is not auditable, not transferable, and not compliant with Article 9 of the EU AI Act, which requires that risk management measures be defined in terms that allow the system's conformity to be assessed throughout its lifecycle.
The three-sentence success definition
Defining success criteria for an AI project does not require a specification document. It requires three sentences, written before architecture selection begins, that answer three questions no reasonable stakeholder can refuse to answer.
The first question: what is the system intended to produce? Not in general — specifically. For the Visa Streaming Tool, a one-sentence answer would have been: the system is intended to assign a priority stream to each visa application such that the stream assignment correlates with the caseworker's subsequent assessment in at least X% of cases. That sentence does not exist in the public record of the project. Its absence is the failure.
The second question: how will that be measured? The measurement does not need to be complex. It needs to be defined. A metric name, a measurement method, and a data source. Written down before the first training run.
The third question: what threshold constitutes failure? The number below which the system will be reviewed, adjusted, or suspended. Not the number below which someone will feel uncomfortable — the number below which a documented decision will be taken.
Before any AI project proceeds to architecture selection, write three sentences in the project brief. They are not optional addenda — they are the gate that separates a project with a definition of done from a project that will be evaluated by impression.
Sentence 1 (intended output): "This system is intended to produce [specific output] such that [measurable criterion] is met [in what context]."
Sentence 2 (measurement method): "Performance will be measured by [named metric], calculated as [method], using [data source]."
Sentence 3 (failure threshold): "If performance falls below [specific value], the system will be [suspended / reviewed / retrained], decided by [named role]."
These three sentences take thirty minutes to write when the project is well understood. When they take longer than thirty minutes — or when there is disagreement about the answers — that disagreement is the signal that the project is not yet ready to proceed. Not because the team is unprepared. Because the problem has not been defined at the level of precision that production requires.