Feature pipelines are tested. The unit tests pass. The integration tests pass. The staging environment runs the pipeline without errors. The data quality checks confirm that features are within expected ranges. The evaluation metrics on the validation set are good. The team is confident.
What the tests did not run is the pipeline under the conditions it will encounter in production: the full data volume, the full user graph, the concurrency of real-time feature computation competing with batch jobs, the edge cases in production data that staging data does not replicate, and — most critically for graph-based ML systems — the behaviour of the pipeline when every node in the graph updates simultaneously.
The LinkedIn People You May Know (PYMK) incident of 2012 is the most publicly documented case of this gap causing a production failure at scale. It is also one of the most analytically useful, because LinkedIn's engineering team published a detailed retrospective describing exactly what the staging environment failed to simulate — and why the failure was invisible until it reached production.
What the LinkedIn PYMK incident established
LinkedIn's engineering blog post from 2012, authored by members of the PYMK team, described a feature update that caused a large volume of unexpected recommendation emails to be sent to users. The cause was a combination of factors, but the structural root was consistent: the feature pipeline had been tested in a staging environment that used a subset of the production data and did not replicate the full interaction patterns of the production social graph.
Specifically, the staging environment did not replicate what happens when a graph update — a new connection, a profile change, an activity signal — propagates through a dense social network where many users share many connections. In staging, the graph was sparse enough that a feature update affected a bounded set of users. In production, the same update propagated through a much denser graph and triggered downstream computation for a far larger population than the staging test had suggested. The pipeline ran correctly. The scale assumption was wrong.
"The staging environment did not reflect the density of the production social graph. When the feature update was deployed to production, the propagation of graph updates through the full network triggered downstream computation at a scale that was not predicted by staging tests. The pipeline behaviour was correct — the scale assumption embedded in the pipeline's design was not. We had tested what the pipeline did. We had not tested what it did at production scale on production data."
The EU AI Act's Article 9 requires that risk management identify risks arising from the reasonably foreseeable conditions of use, including conditions that differ from the test environment. A feature pipeline that has only been tested in staging — on a subset of production data, with a fraction of production concurrency — has a reasonably foreseeable condition of use that it has never encountered: full production load on production data. That condition is not exotic. It is the default condition of operation.
Four production conditions staging environments routinely fail to replicate
The LinkedIn case is not exceptional. It is representative of a structural gap that exists in every ML system whose feature pipeline has been validated only in staging. Staging environments are built for functional correctness testing, not production condition simulation. They systematically underrepresent four conditions that produce feature pipeline failures in production.
Data volume. Staging datasets are subsets of production. Feature computation behaviour that is acceptable at 10% of production volume may become unacceptable at 100% — not because the computation is wrong, but because the performance characteristics, cache hit rates, and query execution plans change at production scale.
Graph density. For any ML system that computes features from a graph — social graphs, transaction networks, supply chain relationships, knowledge graphs — staging graphs are sparser than production graphs. The propagation behaviour of updates through a sparse graph does not generalise to a dense graph. The LinkedIn incident is the canonical case.
Concurrency. Staging pipelines run with far less concurrency than production. Race conditions, lock contention, and resource competition that are invisible at staging concurrency levels emerge at production concurrency. Feature pipelines with shared state — shared feature stores, shared caches, shared databases — are particularly exposed.
Data distribution tail. Staging datasets are typically drawn from the central distribution of production data. Production data has a tail — unusual values, rare combinations, extreme inputs — that staging data underrepresents. Feature pipelines that handle the central distribution correctly may compute incorrect feature values for tail inputs, because tail inputs were not present in sufficient volume during staging validation.
The production simulation test
The resolution is not to make staging environments more production-like — that is expensive and often impractical for systems with large graphs or high concurrency requirements. The resolution is to add one explicit test to the deployment gate: a production simulation test that runs the feature pipeline against a snapshot of production data at a representative fraction of production concurrency, for a defined duration, and compares the feature output distribution to the last known-good production run.
The test does not need to run at 100% production scale. It needs to run at a scale that exposes the failure modes that staging scale conceals — which in practice means at least 50% of production data volume and graph density. The test result is not pass/fail on a single metric. It is a comparison: if the feature output distribution from the simulation test differs significantly from the last known-good production run, the difference is the signal that something in the pipeline has changed in a way that staging did not surface.
Before any feature pipeline change is promoted to production, run the following test: execute the pipeline against a snapshot of production data at a minimum of 50% production graph density and 50% production data volume. Log the feature output distribution for a representative sample of entities. Compare the distribution to the last known-good production run using a defined statistical threshold.
The test has three pass conditions: the pipeline completes without errors at the simulation scale; the feature output distribution falls within the expected range relative to the last known-good run; and the pipeline completes within an acceptable time bound at simulation scale.
If any condition fails, the change does not proceed to production. Not because the functional tests failed — they passed. Because the production conditions test surfaced a behaviour that the staging environment was too small to reveal. That is the test LinkedIn did not have. It is the test that would have caught the PYMK incident before it became a production failure.