There is a specific kind of confidence that follows a clean model evaluation. The metrics are good. The test set is held out. The evaluation methodology is documented. The team has done the work correctly. The confidence is legitimate — and it is also incomplete, for a reason that is structural rather than methodological: the team that trained the model also decided what the evaluation would test.
This is not a criticism of any particular team's rigour. It is a description of a systematic blind spot. The evaluation set reflects the inputs and edge cases the team thought of during development. It does not reflect the inputs and edge cases that will arrive from real users in the deployment environment — inputs shaped by usage patterns, user demographics, and operational contexts the development team was not in the room for. The model passes evaluation because the evaluation was designed by the same mental model that designed the model.
The MHRA's 2022 guidance on software and AI as a medical device addresses this directly — and does so in terms that apply to any AI system whose outputs will be acted on by someone who was not in the room when the model was built.
What the MHRA guidance establishes
The MHRA's guidance, published in 2022 as part of the UK's post-Brexit regulatory framework for medical devices including AI-enabled software, requires that validation of AI systems used in clinical decision support be conducted independently of the development team. The specific requirement is that the validation dataset — the data used to test final performance before deployment — must be collected and curated by a team that was not involved in model development, from a population and clinical setting representative of the intended deployment context.
The rationale the MHRA sets out is precise: a development team's choices about training data, feature engineering, and model architecture create systematic assumptions about what the deployment context looks like. An evaluation set built by the same team will reflect those assumptions. An independent validation dataset collected from the actual deployment context will not — and the gap between the two evaluations reveals the assumptions that were baked into the development process.
"The validation dataset should be independent of the training and tuning datasets and should be representative of the intended use population and clinical setting. Where possible, validation should be conducted by a team independent of the development team, using data that was not accessible during model development. This independence is necessary to ensure that the validation reflects the performance of the system in the intended deployment context, rather than the performance of the system on data that reflects the development team's assumptions about that context."
The EU AI Act's Annex IV, which specifies the technical documentation requirements for high-risk AI systems, includes a similar requirement: the documentation must describe the validation methodology and confirm that validation was conducted on data representative of the intended deployment population — not the training population. The distinction is subtle in documentation and significant in practice.
The three evaluation gaps this creates
When a development team builds its own evaluation set, three systematic gaps open between the evaluation result and the production performance. None of them are visible in the evaluation metrics. All of them are detectable if the evaluation is conducted independently.
Selection bias in edge cases. The development team tests the edge cases it imagined. Production will generate edge cases it did not imagine — unusual input combinations, atypical user phrasing, inputs from demographic groups underrepresented in the team's mental model of the user population. An independent evaluation team, given access to actual user data from the deployment context, will generate a different edge case set. The gap between the two is the selection bias.
Population mismatch. The development team's test set is drawn from the training distribution or from data the team had access to. The deployment population may have different characteristics — age distribution, language patterns, comorbidity profiles in clinical settings, transaction patterns in financial systems. An independent evaluation using data from the deployment population will surface performance differences invisible in the development evaluation.
Confirmation in metric choice. The development team chooses evaluation metrics. The choice of metrics reflects what the team believes is important to measure. An independent evaluator — particularly one who represents the clinical, operational, or regulatory perspective of the deployment context — will often identify metrics the development team did not include because those metrics were not salient to the development process.
When this applies outside regulated industries
The MHRA guidance applies to medical devices. The EU AI Act Annex IV applies to high-risk AI systems as defined by the Act. Neither applies directly to a recommendation engine or a customer churn model. But the structural reasoning applies universally: any model whose evaluation was designed by the same team that designed the model has a systematic gap between its evaluation results and its production performance, proportional to the difference between what the development team imagined and what the deployment context actually produces.
The practical question is not whether independent evaluation is legally required in a given context. It is whether the organisation is willing to discover the gap in evaluation — where it is cheap to act on — or in production, where it is not.
For the next model evaluation your team runs, make one structural change: the final validation dataset must include a minimum of 20% inputs that were not generated or curated by the development team. The inputs can come from a business analyst who was not in the development process, from a sample of actual user queries logged in production, or from a subject matter expert who represents the deployment context.
The 20% is not a regulatory threshold. It is a practical minimum that forces the evaluation to encounter inputs the development team's mental model did not generate. What the model does on those inputs — compared to what it does on the team's inputs — is the signal. If the performance difference is small, the team's mental model was a good proxy for the deployment context. If the performance difference is large, the evaluation was confirming a design assumption rather than testing a production reality.
The gap, if it exists, is better found before deployment than after it.