The Three Roles That Keep Failing Each Other — And What Changes When They Stop

Six Patients. Three Roles. Zero Shared Context.

On 3 June 1985, a 40-year-old woman attended a radiation therapy session at a clinic in Marietta, Georgia. She was receiving treatment for breast cancer on a Therac-25 linear accelerator — a machine manufactured by Atomic Energy of Canada Limited (AECL) and widely deployed in North American radiotherapy clinics throughout the 1980s. The treatment ended. The machine displayed the message it was programmed to display when a session completed without incident: no malfunction. The patient left the clinic with what staff assumed was a mild skin reaction. She died of radiation injuries.

What the Therac-25 had actually delivered was approximately 100 times the intended radiation dose. The machine had malfunctioned in the most precise sense of the term: it had performed exactly as programmed, and the programme was wrong.

Over the next eighteen months, the same failure mode — a race condition in the machine's control software — produced at least five further overdoses. Three patients died. Several others suffered severe and permanent radiation injuries. The failure sequence was documented in forensic detail by Nancy Leveson and Clark Turner in their 1993 paper published in IEEE Computer, which remains the most rigorous case study in software engineering safety literature for the failure mode the Therac-25 exemplifies. What makes the case enduring is not the severity of the outcome — though that severity makes it impossible to treat abstractly — but the precision with which it identifies how the outcome was produced. Not by one decision, and not by one role.

The Therac-25 was the successor to the Therac-20, which had operated safely because it incorporated hardware interlocks — physical safety mechanisms that prevented the machine from operating in dangerous configurations regardless of what the software instructed. When AECL developed the Therac-25, the engineering team made a decision that was technically defensible given its stated rationale: they removed the hardware interlocks on the grounds that the software was sufficiently reliable to carry the safety function alone. This was a confident claim, and it was not examined with the rigour that confidence required.

That decision — taken at the engineering level — had an implication that was never surfaced at the design level: it transferred the entire safety burden to the software layer without anyone verifying that the software layer had been designed and tested to carry it. The engineer who made the interlocks redundant in code was not the person responsible for verifying the safety architecture of the system as a whole. The person responsible for the safety architecture proceeded on the assumption that the software team was handling it. Neither assumption was wrong in isolation. Together, they were fatal.

When the first incident occurred, the operator's report was reviewed by AECL and classified as operator error. This classification was not unreasonable given the information available at that level of the organisation. The machine had reported no malfunction. The software logs showed no fault. The organisation's incident review process — such as it was — had no mechanism to route the operator's report to the engineering team that would have recognised the race condition pattern. The three roles — engineer, system designer, and the organisational authority responsible for incident response — had each done something locally defensible. The compound outcome was not defensible at all.

Leveson and Turner identify the failure mode with clinical precision: the Therac-25 accidents were not the result of incompetent engineers, inadequate hardware, or unusual operating conditions. They were the result of safety-critical decisions being made in isolation by roles that had no structural mechanism for sharing the information that would have made those decisions safer. The engineer who removed the interlocks did not know what the system designer would do with that decision. The system designer did not know what the engineer had removed. The organisation reviewing incident reports did not know what either had decided. The machine killed people because three roles, each operating within the scope of what they were responsible for, had never been in the same conversation about what they were collectively building.

That is the failure class this article is about. Not individual incompetence. Structural disconnection. And it is not a failure class that belongs to 1985.

The Pattern Did Not End in 1987

The Therac-25 is 1985. The pattern is current. Three AI cases — each drawn from primary institutional sources, each involving systems built by competent organisations with sufficient resources — demonstrate the same three failure modes in the same three roles, with consequences that are less acute but far more pervasive than a recalled radiotherapy machine.

The Engineer's failure: optimising the right metric for the wrong context

In 2019, Ziad Obermeyer and colleagues published research in Science examining a commercial healthcare algorithm used by US health systems to identify high-risk patients for enrolment in care management programmes. The algorithm was built by Optum, a large health services company, and was used by approximately 200 million people's worth of managed care decisions annually. Its stated objective — predicting which patients would incur the highest future healthcare costs — was achieved with high accuracy. The model performed well. The problem was that healthcare cost and healthcare need are not the same variable.

For Black patients, structural barriers to healthcare access produce systematically lower historical costs for the same level of underlying medical need. A patient who cannot access care does not generate the claims data that would make a cost-prediction algorithm flag them as high-risk. The algorithm therefore consistently underestimated the health needs of Black patients — by a substantial margin. Obermeyer et al. estimated that at the same level of algorithm-predicted risk, Black patients were significantly sicker than white patients: they had more chronic conditions and more active comorbidities. The model was accurate on its stated metric. It was producing systemically biased outputs on the metric that the healthcare context and the regulatory obligation actually required.

This is The Engineer's failure in its canonical form. The objective was specified. The model was built to the specification. The specification was wrong for the compliance context — and the compliance context was never a constraint at the model design stage, because the conversation between the engineering function and the compliance function had not happened before the model was trained. It had not happened because no organisational mechanism required it to happen. Under EU AI Act Article 10, training, validation, and testing data must be examined for possible biases. That examination is not a data science task that The Engineer performs in isolation. It requires the compliance function to specify what the examination must cover and which protected characteristics are active constraints. Without that conversation — without the compliance function in the room before the first line of training code is written — The Engineer has no specification to work to. They build the best model they know how to build. The model fails on a dimension they were never asked to measure.

The Architect's failure: designing the system boundary without designing the constraints

Amazon's internal AI recruiting tool, reported by Reuters in October 2018 and subsequently confirmed by Amazon, was trained on historical hiring data from a period in which Amazon, in common with the technology industry, hired predominantly male software engineers. The system learned to penalise CVs that contained signals associated with female identity — the word "women's" in a club name, attendance at women's colleges, certain formulations of accomplishment that appeared more frequently in female applicants' CVs than male ones. The team that discovered the pattern attempted to correct it, and found the corrections insufficient to prevent the bias from appearing in other forms. Amazon dissolved the team.

The architectural failure here is not the bias itself. Bias in training data is a known failure mode, and the available tools for identifying and correcting it have been documented extensively since at least 2016. The architectural failure is that the training data provenance question — what does this data encode beyond what we intend, and what are the compliance implications of training a hiring decision system on it — was never elevated to an architectural constraint. It was a data science problem at the model level. It should have been an architectural constraint at the system design level, enforced before the first model was specified. The Architect who designed the system boundary for Amazon's recruiting tool either did not ask the question or asked it and had no mechanism to make the answer binding. EU AI Act Article 9 requires that risk be identified, analysed, and mitigated as part of the system design process — not as a post-deployment correction exercise. That is an architectural obligation. It requires the Architect to bring the compliance function into the design before the boundary is drawn, not after the model has been trained and the bias has been discovered in production.

The Principal's failure: approving decisions without building the signal path for countervailing evidence

The US House Committee on Transportation and Infrastructure published its final report on the Boeing 737 MAX in September 2020. The report is specific about the organisational mechanism of failure. Boeing's management structure, following the 1997 merger with McDonnell Douglas, had shifted primary accountability from engineering safety to programme schedule and financial performance. The MCAS system — the Manoeuvring Characteristics Augmentation System — was developed to compensate for the changed aerodynamics of the re-engined MAX variant. In its final design, MCAS relied on a single angle-of-attack sensor. If that sensor provided erroneous input, MCAS would activate and push the aircraft's nose down, with sufficient authority to overpower pilot correction. The activation was designed to be continuous, not a one-time adjustment.

Pilots were not informed of MCAS's existence in their type-rating training materials. Boeing's rationale was that including it would have triggered an expanded differences training requirement — which would have reduced the MAX's competitive advantage over its rival, the Airbus A320neo, which required no additional type rating for A320 pilots. The House Committee's report finds that Boeing's management actively influenced the FAA's oversight assessment of MCAS, that safety engineers within Boeing who raised concerns about single-sensor dependency were not heard at the level where the decision could have been changed, and that the organisational authority structure had been deliberately configured so that schedule and cost signals reached decision-makers faster and more reliably than safety signals did.

Lion Air Flight 610, 29 October 2018. Ethiopian Airlines Flight 302, 10 March 2019. 346 people. The House Committee's conclusion is precise: "The history of the 737 MAX is a story of the intersection of corporate pressure, failed oversight, and deliberately concealed design decisions." The Principal — in this case Boeing's executive leadership and the certification regime it had shaped — had approved a system whose failure mode was unknown to the people who needed to respond to it, because the organisational structure ensured that the engineering signals that would have made the failure mode visible never reached the level where they could have been acted on. This is not a safety engineering failure. It is a cross-role communication architecture failure that was institutionalised by management decision.

The reader who arrived at this article having read Articles 1 and 2 will recognise the pattern. The Knight Capital deployment failed because organisational authority, technical state management, and operational procedure were never in the same conversation. The Amazon and COMPAS cases failed because the system boundary was drawn around the model, not around the problem the model was being deployed to solve. The Therac-25, the Boeing 737 MAX, and the Optum healthcare algorithm are the same failure class at three different scales and in three different industries. The pattern is not coincidental. It is structural. And it is reproducible until the structural cause is addressed.

The Three Conversations That Change the Outcome

What follows is not a methodology. It is not a framework. It is a description of three specific moments in the lifecycle of an AI system where the cross-role conversation must happen — what each role needs to bring to it, what the output must be, and what the compounding cost is when the conversation is deferred or does not happen at all. These three moments are not arbitrary. They are the precise points in the lifecycle where locally reasonable decisions become collectively consequential — where a decision made by one role, without the information held by the other two, creates a failure mode that cannot be corrected without redesigning something that has already been built.

EU AI Act Article 14 requires that providers of high-risk AI systems implement human oversight measures that enable those who deploy or use the system to understand, monitor, and intervene in its operation. The NIST AI Risk Management Framework's GOVERN function establishes that organisational roles, responsibilities, and cultures must actively support AI risk management — not as a compliance exercise performed after the system is operational, but as a structural feature of how the system is built. Neither obligation is satisfied by good intentions and verbal briefings. Both require documented, structural mechanisms that ensure the right information reaches the right people at the right time. The three conversation triggers described below are the minimum expression of what those obligations require in practice.

Trigger 1 — The Pre-Sprint Architecture Conversation

This conversation happens before the first sprint begins. Before the model is specified. Before the data is sourced. Before the infrastructure is chosen. It is the conversation that establishes the constraint register — the documented set of constraints within which every subsequent decision will be made. Most AI projects do not have this conversation. They have a requirements meeting, a sprint planning session, and a kickoff. None of those three events is the Pre-Sprint Architecture Conversation, and none of them produces what the Pre-Sprint Architecture Conversation must produce.

The conversation is initiated by The Principal. Not because The Principal has the most relevant technical context — in most cases, they have the least — but because only The Principal has the organisational authority to stop a sprint that begins with inadequate architectural constraints. A sprint that begins without a documented constraint register is a sprint that will produce technical work that is locally correct and collectively fragile. The Principal who initiates this conversation is not inserting themselves into a technical process. They are exercising the one form of authority that the technical process cannot supply for itself.

What The Architect brings to this conversation is the system boundary diagram — not as a final proposal, but as a starting point for constraint identification. The diagram shows what the system will do, what it will interact with, and what it will not do. More importantly, it shows what assumptions the proposed boundary encodes, and which of those assumptions require validation before the first sprint begins. The Architect also brings the three production-ready dimensions — technical, organisational, and compliance — documented as constraints, not aspirations. Not "we will aim for explainability" but "the inference pipeline will produce a SHAP explanation for every decision that can be reviewed by a human operator before the decision is acted upon." The distinction is the difference between an intention and a design requirement.

What The Engineer brings is the data provenance statement. This is the document that answers, precisely and with sources: where will the training data come from; what does that data encode beyond its stated variables; what demographic subgroups are represented in it and in what proportions; what biases have been examined and what hasn't been examined yet and why; and what the implications of those gaps are for the compliance obligations that EU AI Act Article 10 imposes. This is not a document that The Engineer can produce in isolation from the compliance function. It requires the compliance function to specify which protected characteristics are active constraints for this system's risk classification. But it is The Engineer's document — The Engineer is accountable for its accuracy, and The Engineer is the person who will make the training decisions that the document constrains.

What The Principal brings is the regulatory context. Which EU AI Act risk classification applies to this system. Which Article 9 risk management obligations are active. What the audit trail requirements are. What the human oversight mechanism must look like, given Article 14. This is not information The Principal produces independently. It comes from the compliance function and from external legal counsel if the system operates in multiple jurisdictions. But The Principal is the person who must present it at this conversation — because the authority to make it a binding constraint on the sprint comes from The Principal, not from the compliance team.

The output of this conversation is a constraint register. Not a Confluence page. Not a slide deck. A versioned, signed artefact that specifies: the compliance obligations active for this system; the data governance requirements that apply to training, validation, and testing data; the explainability and audit trail requirements that the production architecture must satisfy; and the human oversight mechanism that must be built into the inference pipeline before the system is deployed. Three authors. Three signatures. The sprint begins when the constraint register exists. Not before.

The cost of not having this conversation is not that the sprint fails. The sprint usually succeeds. The model is trained. The performance metrics are achieved. The architecture is built. The system is deployed. The cost appears six months later, when the compliance team reviews the system and discovers that the inference pipeline produces no SHAP explanations, that the human oversight mechanism is a UI button that can be bypassed, and that the training data was never examined for the demographic biases that Article 10 required it to be examined for. The remediation cost is not the cost of a conversation that didn't happen. It is the cost of redesigning a production system that was built to the wrong specification — because the specification was never documented, and every role optimised for the specification they assumed was correct.

Trigger 2 — The Model Handoff Conversation

The Model Handoff Conversation happens at the point of handoff from model development to production architecture — not after the model is trained, and not after the production architecture is specified. It happens before the production architecture is specified, because the decisions the production architecture must make are determined by what the model can and cannot do. An architecture designed around a model's average performance will fail when the model encounters conditions that deviate from the average. An architecture designed around a model's worst-case behaviour — its performance under distribution shift, its confidence intervals at the boundary of its training distribution, its behaviour when inputs are degraded or anomalous — is an architecture that can be defended to a regulator when the failure mode eventually manifests.

The Model Handoff Conversation is initiated by The Engineer. This is the moment in the lifecycle where The Engineer has the most information about what the model can and cannot do — and where that information is most likely to be lost, because the handoff from model development to production engineering is typically a verbal briefing with a performance summary. A performance summary is not a handoff. A performance summary tells you that the model achieved 91% accuracy on the test set. It does not tell you what the accuracy was on the demographic subgroup most represented in the production user population. It does not tell you what happens to the model's confidence scores when it encounters inputs that are outside its training distribution. It does not tell you where the model fails, at what rate, and in what direction — whether it fails conservatively or optimistically, and what the implications of each failure direction are for the users who will be affected by those failures.

The output of the Model Handoff Conversation that The Engineer initiates is a model card — not a performance summary, but a structured document that specifies, at minimum: the training data provenance and the examination performed for demographic biases; the model's performance on each demographic subgroup represented in the production population; the confidence intervals for model outputs under nominal and degraded input conditions; the known failure modes and the conditions under which they manifest; the recommended monitoring thresholds for detecting distribution shift in production; and the explainability mechanism — the SHAP values or equivalent — that The Architect must build into the production pipeline to satisfy EU AI Act Article 13's transparency obligations. This document is not optional under Article 13. The Act requires that high-risk AI systems provide output interpretability to the users and operators who rely on them. That requirement cannot be satisfied by a production architecture that was designed without knowing what the model's outputs mean in edge cases.

What The Architect brings to this conversation is the deployment constraint map: the HITL requirements that the production pipeline must implement; the latency constraints within which the explainability mechanism must operate; the audit trail architecture that will record every inference decision with the inputs that produced it; and the rollback architecture — the documented, tested procedure for removing the model from production if a monitoring threshold is breached. The rollback architecture is not an operational detail. Under EU AI Act Article 9, risk management must include the ability to stop or reverse a system whose behaviour has become unsafe. A rollback procedure that exists in someone's head is not a rollback architecture. A rollback procedure that is documented, tested, and executable by someone who was not involved in building the system is.

The output of this conversation is a signed handoff checklist with named acceptance criteria. The Architect accepts the model into the production architecture design only when every item on the checklist is satisfied. The criteria are specific: not "explainability is implemented" but "SHAP explanations are generated per inference, stored with a reference to the inference decision record, and retrievable by the human oversight operator within 30 seconds of any decision being flagged for review." The Engineer signs to confirm the model meets the criteria. The Architect signs to confirm the architecture is designed to receive a model that meets those criteria. The Principal signs to confirm that the criteria are consistent with the regulatory obligations that the constraint register identified.

What is the cost of this conversation not happening? The cost is the €40 million warranty reserve from Article 7 of this series. It is the Amazon recruiting tool that was discovered to be systematically biased in production and had to be discontinued. It is the NHS diagnostic tool described in the Lancet Digital Health literature whose failure was discovered not in observation but in the statistical pattern of outcomes that the monitoring system wasn't designed to detect. The common feature of all of these failures is that the information needed to design the monitoring correctly existed at model development time and was not transferred to the team that built the production architecture. The model card is the artefact that transfers it. The Model Handoff Conversation is the mechanism that ensures the artefact is produced and received.

Trigger 3 — The Pre-Finalisation Compliance Review

The third conversation is the one most organisations hold in the wrong place at the wrong time. The compliance review, in most organisations, is a deployment gate — a check performed after the system is built to verify that it satisfies the relevant regulatory obligations. This is not a compliance review. It is a compliance recovery exercise, and it is consistently more expensive than compliance design because the finding that emerges from it is not "this documentation needs to be completed" but "this architecture needs to be redesigned."

The Pre-Finalisation Compliance Review happens before the design is finalised. It is a design input, not a deployment gate. Its purpose is to ensure that the compliance obligations that EU AI Act Articles 9, 10, 13, and 14 impose are expressed as structural constraints in the design — not as documents written after the system is built that describe a compliance state the system does not actually satisfy.

The conversation is initiated by The Architect. The compliance function does not have the authority to initiate a conversation that changes a system design — only The Architect can call the design into question. But The Architect who understands what production-ready means knows that a design that has not been reviewed against the compliance obligations before it is finalised is not a production-ready design. It is a design that will require revision at the worst possible time.

What the compliance function brings to this conversation is not a checklist of documentation requirements. It is the specific structural obligations that each active Article imposes on the system design. Article 9 requires that a risk management system be established, implemented, and documented for the system's entire lifecycle. That is not a document. It is a process with named owners, documented triggers, and a versioned record that proves the process was followed. Article 10 requires that training, validation, and testing data be subject to data governance practices that include examination for possible biases. That is a constraint on the data pipeline architecture, not an annotation in a compliance document. Article 14 requires that human oversight measures be implemented such that the people who deploy or use the system can understand and intervene in its operation. That is a state machine in the inference pipeline, not a button in a UI.

What The Architect brings to this conversation is the ADR log — every architectural decision, with the compliance obligation it satisfies or the compliance risk it creates. The ADR log is not the product of the compliance review. It is the input to it. The Architect who has been maintaining an ADR log throughout the design process arrives at the compliance review with a documented record of every decision and its rationale. The compliance function reviews that record against the active Articles and identifies the gaps. The gaps become design changes before the design is finalised.

What The Engineer brings is the explainability documentation: the SHAP architecture, the audit trail design, the specification of what information is recorded per inference decision and how it can be retrieved. Under Article 13, the transparency obligation requires that the logic of high-risk AI system decisions can be understood by the users and operators who are affected by them. The audit trail is the technical expression of that obligation. If The Engineer has not designed it, it does not exist — and no compliance document can substitute for a system that does not actually record the information the regulation requires it to record.

The output of this conversation is a compliance mapping document. One row per active EU AI Act obligation. Three columns: the obligation as stated; how the current design satisfies it structurally; and what change is required if the current design does not satisfy it. This document is not produced for the regulator. It is produced for the design team, to be used as a constraint on the finalised design. The regulator will ask for evidence of compliance when the audit happens. The compliance mapping document is the evidence. Producing it before the design is finalised means the design can be adjusted to satisfy the obligations it does not yet satisfy. Producing it after the system is deployed means the evidence does not exist and the system cannot be defended.

The NIST AI Risk Management Framework's GOVERN function identifies this requirement in institutional terms: organisations that govern AI risk effectively have established policies, processes, and accountability structures that ensure risk management is integrated into AI development from the beginning. The compliance review as a deployment gate is the structural expression of an organisation that has not yet established those policies. The Pre-Finalisation Compliance Review as a design input is the structural expression of an organisation that has.

What the System Looks Like When the Conversation Has Happened

The system that emerges from three correctly executed cross-role conversations is not architecturally different in its technical components. The models are the same. The infrastructure is comparable. The performance metrics are equivalent. What is different is the relationship between the system's technical decisions and the organisational and compliance context in which those decisions must survive.

The ARB presentation changes. In the typical AI ARB presentation, The Architect presents the technical architecture, fields questions from engineering peers about infrastructure choices, and receives approval with a list of conditions. The compliance function is either absent or present as an observer. The business case is asserted rather than demonstrated. When the three conversations have happened correctly, the ARB presentation is structured around the artefacts those conversations produced. The constraint register is tabled first — not as a slide, but as a versioned document that the board can reference. The ADR log maps every architectural decision to the constraint it satisfies or the constraint it introduces. The compliance mapping document shows, row by row, how the design satisfies each active Article. The Principal who chairs the ARB can ask specific questions because there are specific answers. The board does not approve a system that seems production-ready. It approves a system for which production-readiness is demonstrated.

The sprint dynamic changes. The sprint that begins with a documented constraint register has a different character from the sprint that begins with requirements and good intentions. Scope changes are evaluated against the constraint register, not against the sprint backlog. The PM who wants to defer the HITL mechanism to a post-launch iteration encounters a constraint register that specifies the HITL as an architectural requirement, not a feature. The Engineer who discovers a data quality issue escalates it against the data provenance statement in the constraint register, not against their own judgment about what is significant enough to escalate. The constraints are not opinions. They are a signed artefact, and changing them requires the same three-role conversation that produced them.

The handoff to production operations changes. A production system that was built against a signed handoff checklist has a different operational character from a system that was handed off verbally. The on-call engineer who is paged at 2 a.m. because a monitoring threshold has been breached has a documented runbook that specifies, in terms of the model card's failure mode documentation, what the threshold breach likely indicates and what the response options are. The rollback procedure is documented, tested, and executable by someone who was not involved in building the system. The audit trail is structured, retrievable, and sufficient for a regulatory inspection. These are not aspirations. They are the conditions under which the handoff was accepted.

The compliance audit changes. The organisation that has been through three correctly executed cross-role conversations arrives at a regulatory audit with a evidence portfolio — not a compliance narrative assembled post-hoc to justify decisions that were made on other grounds, but a contemporaneous record of the decisions that were made, the obligations those decisions were designed to satisfy, and the structural mechanisms that enforce them in the running system. The auditor asks: "How is human oversight implemented?" The answer is not a description of the system's intent. It is a reference to the HITL state machine specification in the design record, the inference pipeline implementation that enforces it, and the Chronicle audit log entries that prove it has been operating as specified since deployment. The auditor asks: "How were training data biases assessed?" The answer is a reference to the data provenance statement in the constraint register, the Trigger 2 model card's demographic subgroup performance analysis, and the Article 10 data governance documentation produced in the Trigger 3 compliance review. The documentation exists because the conversations that produced it were structured requirements, not optional additions.

Most importantly: the failure mode changes. The system that was built through three correctly executed cross-role conversations does not fail silently. When it fails — and it will fail, because all production systems do — the failure is caught by monitoring thresholds that were specified against the model's documented failure modes, escalated through a signal path that was designed to carry it to the right decision-makers, and responded to with a runbook that was written before the failure occurred. The Therac-25 killed people because the organisation had no mechanism to route the incident signal to the people who could have recognised the failure class. The Boeing 737 MAX crashed twice because the organisational structure was designed to ensure that the safety signal did not reach the level where it could be acted on. The organisations that built both systems were technically competent. The failure was structural, not technical. Production-ready is not a technical standard. It is a standard for the relationship between the technical system and the organisational and compliance context in which it must survive.

Three free articles. One failure class. Three roles. The same structural disconnection at the £440 million trading floor in 2012, in the radiotherapy clinic in 1985, at Boeing's engineering offices in 2017, and in the model training pipelines of organisations building AI systems in regulated industries today. The diagnosis is specific. The mechanism is known. The three conversation triggers described in this article are not a complete solution — a complete solution requires the framework within which those conversations are structured and the discipline to maintain that structure across every sprint and every design decision. But the three conversations are the minimum expression of what production-ready requires from the cross-role relationship. If they are not happening in your organisation, the failure class described in these three articles is not a historical curiosity. It is a forecast.