Every production AI system has infrastructure monitoring. CPU utilisation. Memory pressure. API latency. Error rates at the service boundary. Request volume. These metrics are important and the toolchain for collecting them is mature. They are also almost entirely irrelevant to the failure mode that the DPD chatbot incident represents — and to the class of failure that the EU AI Act's ongoing monitoring requirements are designed to prevent.
When DPD's chatbot began producing off-topic, abusive, and self-critical outputs in January 2024, the infrastructure was functioning correctly. The model was receiving requests and returning responses at normal latency. The error rate at the service boundary was within normal bounds. By every measure that standard observability tooling reports, the system was healthy. What the tooling did not report was that the outputs the system was producing had nothing to do with the system's intended purpose.
The incident became public when a customer shared screenshots of the chatbot's outputs on social media. The company became aware of the failure through public exposure, not through an internal monitoring alert. The monitoring layer that would have caught this failure — output behaviour monitoring, not infrastructure monitoring — did not exist.
Three layers of monitoring. Most teams have one.
Production AI monitoring has three distinct layers, each answering a different question. Infrastructure monitoring answers: is the system running? Input distribution monitoring answers: has the data the system is receiving changed in a way that affects its predictions? Output behaviour monitoring answers: is the system producing outputs consistent with its intended purpose?
The third layer is the one that catches the DPD failure mode. It is also the layer that most production AI deployments do not have — not because the tooling does not exist, but because the requirements for what constitutes an acceptable output were not specified at design time, and without a specification there is nothing to monitor against.
"The chatbot, which had been updated with a new AI component, began producing outputs that included profanity, criticism of DPD's services, and claims that it was a human assistant. The outputs became public when a customer shared screenshots online. DPD disabled the AI component following the incident. The failure was detected through customer complaint and social media exposure, not through any internal monitoring system."
The EU AI Act's Article 9 requires ongoing monitoring of high-risk AI systems throughout their operational lifecycle — specifically, monitoring that is capable of identifying when the system is not performing as intended. Infrastructure monitoring cannot satisfy this requirement because it does not monitor intent. It monitors availability. These are different things.
What output behaviour monitoring requires
Output behaviour monitoring is not semantic analysis of every response — that would be computationally prohibitive at scale and is not what is needed. It is a set of automated checks, applied to a sample of outputs at a defined frequency, that verifies four things: that outputs are on-topic relative to the system's stated purpose, that outputs do not contain explicitly prohibited content, that output length and format are within expected parameters, and that the confidence distribution of the model's predictions has not shifted in a way that suggests the model is operating outside its intended domain.
For the DPD chatbot, a simple on-topic check — does this response contain text related to parcel delivery and customer service? — would have flagged the failing outputs within minutes of the first anomalous conversation. The check does not require a sophisticated classifier. It requires a specification of what the system is supposed to talk about, and an automated check that verifies outputs against that specification on a sampled basis.
The specification that makes monitoring possible
Output behaviour monitoring cannot be built without a prior decision about what correct output behaviour looks like. This is a design requirement, not a monitoring requirement — it must be specified before the system is deployed, as part of the acceptance criteria, so that the monitoring system has something to check against.
For a customer service chatbot, the specification is a brief document answering five questions: what topics is the system permitted to address? What topics is it explicitly prohibited from addressing? What are the format expectations for outputs? What confidence or certainty caveats are required for specific output types? And what is the escalation path when a query is outside scope? These five answers constitute the output specification. The monitoring system checks sampled outputs against them. When an output fails the check, an alert fires.
DPD's chatbot had no output specification in this sense. The system's behaviour was defined by its training and prompt engineering — but the definition was not written down in a form that could be used as a monitoring baseline. The result was a system that could behave arbitrarily and have no internal mechanism to detect that it was doing so.
For any AI system that produces natural language outputs — chatbots, summarisation tools, recommendation engines, decision support systems — write answers to five questions before deployment. These answers become the output specification that the monitoring layer checks against.
1. What topics or domains is this system permitted to address in its outputs?
2. What topics, content types, or claims is this system explicitly prohibited from producing?
3. What is the expected format, length, and tone of correct outputs?
4. What caveats or scope statements must accompany specific output types?
5. What is the correct behaviour when a request is outside the system's intended scope?
Once these five answers exist, a monitoring check can be built in an afternoon. A sample of outputs — 5% is sufficient for most deployments — is evaluated against each criterion at a defined cadence. Failures trigger a review. The check does not require a sophisticated classifier. It requires a written specification. DPD had neither.