When an ML engineer says a model is explainable, they usually mean one of several things: the feature importances are documented, a SHAP analysis has been run, the model card describes the training data and evaluation metrics, or the architecture is simple enough that the reasoning is traceable. These are all legitimate forms of technical explainability. None of them are what Article 13 of the EU AI Act requires.
Article 13 requires that high-risk AI systems be designed and developed so that their operation is sufficiently transparent to enable deployers to interpret the system's output and use it appropriately. The standard is not the developer's understanding. It is the operational user's ability to act on the output in a way that is informed and defensible. That is a different thing — and the gap between the two is where the Dutch childcare benefits scandal lived for seven years.
What happened in the Netherlands
Between 2012 and 2019, the Dutch Tax Authority (Belastingdienst) used a risk-scoring algorithm to flag applications for childcare benefit subsidies as potentially fraudulent. Approximately 26,000 families were flagged. Many had their benefits clawed back in full, were left with debts they could not repay, and faced cascading consequences: homelessness, divorce, loss of employment.
The parliamentary inquiry that followed, and the Donner Report published in 2020, identified the structural cause with precision. The algorithm produced a score. The score was acted on by caseworkers. The caseworkers had no way to understand why a specific family had been scored as high-risk, what inputs had driven the score, or what would constitute a valid challenge to it. They were operating on the output of a system they could not interpret — and in the absence of interpretability, they defaulted to the output as though it were a finding rather than a signal.
"The Tax Authority used a risk model whose outcomes were treated as conclusions rather than as starting points for further investigation. Officials could not explain to citizens why they were under suspicion, because the officials themselves did not understand the model's reasoning. The system was not designed to be used the way it was used — and the design provided no mechanism to prevent that misuse."
The model was documented in the technical sense. The documentation existed. What did not exist was the operational layer between the model's output and the human decision: a structured explanation of what the score meant, what the inputs were that drove it, and what the caseworker should do with that information before acting. The model was explainable to the team that built it. It was opaque to everyone it affected.
Two explainability standards — and only one that matters operationally
The distinction is not subtle. Developer explainability and operational explainability are answers to different questions. Developer explainability answers: how does this model work? Operational explainability answers: can the person acting on this output understand it well enough to act on it responsibly, challenge it if it is wrong, and explain their decision to the person it affects?
For a risk-scoring model in a benefit system, operational explainability means: the caseworker receives not just the score but the three inputs that drove it, a plain-language description of what each input represents, and a clear statement of what the score is — and is not — a finding of. That is not a feature request. Under Article 13 of the EU AI Act, for systems used to make or inform decisions that affect individuals, it is a design requirement.
What this means for how models are built
Operational explainability is not a reporting layer added after the model is built. It is a design decision made before training begins, because the explanation format constrains the model choice. A model that cannot produce a per-decision explanation at inference time in terms that are meaningful to a caseworker is a model that cannot be deployed in a context where Article 13 applies — regardless of how accurate it is.
This is not an argument for simpler models as a matter of principle. It is an argument for specifying the operational explainability requirement before selecting the model architecture, because the requirement determines whether a given architecture is deployable in a given context. An ensemble model with 200 features can satisfy Article 13 if it is designed to surface the top three drivers at output time in plain language. A simpler linear model can fail Article 13 if no one has designed the output to communicate what it means to the person acting on it.
Before selecting a model architecture for any system that informs decisions affecting individuals, write three sentences that describe the output explanation in operational terms — not technical terms.
Sentence one: what the output score or classification means in plain language, and what it is not a finding of.
Sentence two: which inputs, expressed in terms the operational user understands, drove the output for this specific case.
Sentence three: what the operational user must do before acting on the output, and what recourse exists for the person the decision affects.
If you cannot write these three sentences for your proposed model architecture, the architecture cannot satisfy Article 13. That is the finding to bring to the design review — not after the model is trained, but before the architecture is chosen.