Every architecture diagram is a model of reality. The useful question about any diagram is not whether it is accurate — it is what it has chosen to leave out, and whether what it has left out is the thing that will cause the failure.
Most AI system architecture diagrams show the same things: the model serving layer, the feature pipeline, the data store, the API surface, the external dependencies. These elements are present because they are the things the team decided to build. What they do not show — in almost every case — is the operational context the system will encounter after it is built: the users who will misuse it, the inputs it was not trained on, the load conditions it will face at peak, and the regulatory obligations that apply to every inference it produces.
The diagram is accurate. The architecture it represents is incomplete. That distinction is where 87% of AI projects lose the gap between design and production.
What 87% actually means
The figure comes from a 2019 VentureBeat report on a panel session at Transform 2019, featuring Deborah Leff, then CTO for data science and AI at IBM, and Chris Chapo, then SVP of data and analytics at Gap. It was subsequently analysed by Joyce Weiner in Why AI/Data Science Projects Fail (Morgan & Claypool, 2021), which identifies five systematic failure conditions — none of which are technical failures in the conventional sense.
The failure Weiner documents most consistently is this: teams proceed through the design and build phase having modelled what the system will do, without modelling the conditions under which it will actually operate. The system that is built is a correct implementation of the design. The design was an incomplete model of the problem.
This is not an argument against architecture diagrams. It is an argument about what they need to contain. The system boundary on a C4 context diagram shows what the system communicates with. It does not show the conditions under which those communications will fail, the users who will interact with the system in ways it was not designed for, or the regulatory context in which every output it produces will be evaluated.
What the EU AI Act requires that most diagrams don't show
Article 9 of the EU AI Act requires a risk management system that, among other things, identifies and analyses known and foreseeable risks arising from the system's use — including risks arising from reasonably foreseeable misuse. The operative word is foreseeable. A risk that was foreseeable but not documented is not a risk that was managed. It is a risk that was ignored.
This is not a documentation obligation that can be satisfied by adding a risk register after the architecture is approved. It is a design obligation — the foreseeable risks must be identified before the architecture is finalised, because the architecture is what determines whether those risks are structurally managed or operationally improvised.
"The risk management system shall consist of a continuous iterative process run throughout the entire lifecycle of a high-risk AI system, requiring regular systematic review and updating. It shall identify and analyse known and foreseeable risks associated with each high-risk AI system, estimate and evaluate the risks that may emerge when the high-risk AI system is used in accordance with its intended purpose and under conditions of reasonably foreseeable misuse."
The three elements Article 9 requires that most architecture diagrams do not represent are: the operational context the system will encounter beyond the happy path, the foreseeable misuse conditions, and the risk management measures that are structurally embedded in the design rather than documented alongside it.
The gap between what the diagram shows and what Article 9 requires
Three elements to add before the architecture is approved
The three additions are not new diagrams. They are annotations to the diagram that already exists — three questions whose answers must appear in the architecture documentation before the design is submitted for review.
Operational context beyond the happy path. What conditions will this system encounter in production that are not present in the test environment? Specifically: what happens under peak load, with degraded inputs, with users who are not the intended user persona, and with data that does not match the training distribution? Each answer is a sentence. The absence of an answer is the gap.
Foreseeable misuse. Under Article 9, this is not optional. For any system that produces outputs that affect people — decisions, recommendations, information — the foreseeable ways in which the system can be misused, misunderstood, or produce harmful outputs must be documented before deployment. This is not the same as the list of things the system is designed to do. It is the list of things it is not designed to do but will be asked to do anyway.
Structural risk management. For each foreseeable risk, is the management measure structural — embedded in the design — or operational — relying on a human to intervene? A risk management measure that relies on human judgment at scale is not a structural measure. The EU AI Act's Article 9 requires measures that are part of the system, not procedures that sit alongside it.
Add these three fields to every AI system architecture document before it goes to review. Not as a compliance appendix — as part of the design record, in the same section as the system boundary and the integration contracts.
1. What production conditions are not represented in the test environment, and what does the system do under each of them?
2. What are the three most foreseeable ways this system will be used outside its intended purpose, and what is the output in each case?
3. For each identified risk, is the management measure structural (embedded in the design) or operational (dependent on human intervention)?
These three questions take thirty minutes to answer for a system you know well. They take longer for a system you don't — which is precisely why the time it takes to answer them is a signal about the design's maturity, not a cost of documentation.