Drift detection has become a standard component of production ML systems. Input distribution monitoring, data quality checks, prediction distribution tracking — the toolchain is mature and the practice is well established. The failure mode that drift detection does not catch is the one that does the most quiet damage: the case where the model is performing correctly on its optimisation target, and the optimisation target has stopped being a reliable proxy for the outcome the business actually needs.
This is not model drift. The model has not changed. The world has not changed in a way the model's input monitoring would surface. What has changed is the relationship between the thing being measured and the thing that matters. Standard drift detection cannot see it because it is not monitoring the right relationship.
The structure of the problem
Almost every production ML system is optimising a proxy. The actual business objective — reliable transport access, accurate fraud detection, equitable credit decisions — is difficult to measure directly in the moment a prediction is made. So a proxy is chosen: a metric that is correlated with the business objective, measurable in real time, and usable as a training signal.
The proxy works as long as the correlation holds. It fails silently when the correlation breaks down in specific conditions — conditions that may not manifest as a change in the input distribution, but that produce a systematic gap between what the model is optimising and what the business needs it to do.
The FTC's 2022 report on algorithmic pricing practices — which examined demand-based pricing models across multiple industries, including ridesharing — identified this pattern as the dominant failure mode in consumer-facing algorithmic systems: models optimising legitimate proxies that diverge from consumer welfare outcomes under specific market conditions. The specific conditions named in the report were emergencies, adverse weather events, and major public gatherings — exactly the conditions under which the proxy (price elasticity of driver supply) and the outcome (reliable transport access) are most likely to diverge.
"Algorithmic pricing systems optimise for measurable proxies of firm-level objectives. When those proxies diverge from consumer welfare outcomes — particularly during high-demand events where the two objectives are structurally in tension — the system performs as designed while producing outcomes inconsistent with the objective it was understood to serve. This divergence is often not visible in standard model performance monitoring."
Joyce Weiner's analysis in Why AI/Data Science Projects Fail identifies this as the fifth and most costly failure condition: solving the wrong problem — which in this context means instrumenting for the proxy while the outcome goes unmeasured. The model is not wrong. The measurement framework is incomplete.
Why standard drift detection misses it
Standard drift detection monitors the relationship between today's input distribution and the training distribution. It asks: has the world changed in a way that makes the model's learned mapping unreliable? If the input distribution is stable, the alert does not fire.
Objective drift asks a different question: has the relationship between the proxy metric and the business outcome changed, regardless of whether the input distribution has shifted? These are independent conditions. A model can exhibit stable input distributions and severe objective drift simultaneously — which is precisely the scenario the FTC report describes.
The fix is not more monitoring. It is the right measurement.
The resolution is not to build a more sophisticated drift detection system. It is to define, before the model is deployed, the business outcome metric that the proxy is intended to represent — and to log both, separately, so the correlation between them can be tracked over time.
For a rideshare pricing model, the proxy is price elasticity of driver supply. The business outcome metric is reliable transport completion rate under high-demand conditions. For a credit scoring model, the proxy is default probability. The outcome metric is loan portfolio performance adjusted for actual repayment conditions in the deployed population. For a fraud detection model, the proxy is transaction anomaly score. The outcome metric is confirmed fraud rate in the flagged population, measured against the base rate.
In each case, the proxy and the outcome are different measurements. They are correlated under the conditions the model was trained on. Whether that correlation holds under all production conditions is an empirical question — and it cannot be answered without logging both.
Before any model is deployed to production, write two metric definitions in the model card: the optimisation target (what the model maximises) and the outcome validator (the business metric that the optimisation target is intended to serve).
They are distinct measurements. They are logged separately. The correlation between them is computed and reviewed on a defined cadence — weekly or monthly depending on the deployment context. When the correlation falls below a defined threshold, the model is reviewed — not because it has drifted in the conventional sense, but because the ground it was designed to stand on has shifted.
This is not a new instrumentation layer. It is the addition of one row to the model card and one field to the inference log. The field that the FTC found absent in the systems it reviewed — and the one that, if present, would have surfaced the divergence before it became a regulatory matter.