How Uber built its ML data supply chain: lessons from the Michelangelo feature store

By 2017, Uber's machine learning ambitions had outpaced its data infrastructure. The company was running models for surge pricingsurge pricingAutomatically adjusting prices in real time based on demand, competition or user behaviour to optimise revenue, margin or conversion.View full definition →, ETA prediction, fraud detection, and driver churn, each built by a different team, each pulling data from overlapping sources, and each computing features independently. An estimated 40 to 50 percent of feature engineering work was being duplicated across teams, according to Uber Engineering's own published accounts. More critically, a feature computed one way during training was sometimes computed differently at inference time, producing what the industry calls training-serving skew. Models that looked solid in offline evaluation performed worse in production, and debugging the gap was expensive and slow.

The organisation had also grown fast enough that institutional knowledge about which features worked well was scattered. A team working on food delivery recommendations at Uber Eats had no reliable way to discover that the fraud team had already developed strong temporal features around user session patterns that might transfer. The data supply chain was, in practice, a collection of disconnected local pipelines rather than a shared asset.

What Uber actually built

The response was Michelangelo, an end-to-end ML platform that Uber Engineering documented publicly in a 2017 post. The feature storefeature storeA centralised repository managing ML features, ensuring consistency between training and serving environments.View full definition → component sat at the centre of it. The core engineering decision was to separate feature computation from model training and serving entirely, treating features as first-class, versioned, shareable artefacts rather than code embedded inside individual model pipelines.

Concretely, this meant building two storage layers. A batch layer used Hive and Spark to compute features at scale and persist them in a Cassandra-backed offline store. An online layer served those same pre-computed features at low latency (typically under 10 milliseconds for Uber's use case) via Cassandra lookups during real-time inference. The same feature definition governed both paths, which is what eliminated training-serving skew: there was one canonical computation, run at different times and against different data volumes, but producing values that were structurally identical.

Teams contributed features to a shared catalogue with documentation and ownership metadata. Before building anything new, engineers were expected to check the catalogue first. This sounds simple, but it required both tooling and organisational incentives to work. Uber introduced a feature discovery interface so that searching the catalogue was faster than rebuilding from scratch.

The platform also handled point-in-time correct joins automatically. In fraud and churn modelling particularly, using a feature value that was not available at the time the training label was generated produces data leakage. Michelangelo's offline store tracked feature values with timestamps, and training pipelines could retrieve the state of any entity's features as of a specific historical moment. This capability is technically straightforward to describe and genuinely difficult to implement reliably at scale.

The results, stated carefully

Uber's engineering blog (a vendor source in the sense that it is Uber's own account, worth reading critically but with no obvious incentive to fabricate operational details) reported that by 2018, Michelangelo was supporting over 100 teams and thousands of features in the catalogue. Model deployment cycles that had previously taken weeks dropped significantly, though Uber did not publish a precise before-and-after figure for that metric.

The more concrete outcome was architectural: teams that would previously have spent two to four weeks building a feature pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → could instead build on existing catalogue entries and ship a first model version in days. The duplicate work problem did not disappear entirely, but the overhead shifted from computation to governance, which is a better problem to have.

Training-serving skew was reduced measurably, though again Uber did not publish a quantified figure on production model degradation rates. What the organisation did say publicly was that debugging production model behaviour became substantially faster because the feature layer was no longer a suspect: the computation was deterministic and auditable.

The pattern proved influential. LinkedIn, Twitter, and later Google (with Feast, an open-source feature store spun out of Gojek and now a CNCF project) all built on similar principles. Tecton, Hopsworks, and Databricks Feature Store each commercialise variants of this architecture, and their marketing materials frequently reference Uber's published work as a conceptual origin point. Those are vendor sources and should be read accordingly, but the architectural convergence across independent implementations is itself evidence that Uber's approach solved a real structural problem.

What transfers and what does not

The core lesson is organisational before it is technical. Uber's feature store only delivered value because the company enforced a norm that engineers would check the catalogue before building. Without that norm, a feature store becomes an archive that nobody maintains. A CDO implementing this pattern needs to define who owns feature definitions, what the review process looks like before a feature enters the catalogue, and how stale or deprecated features get retired. The tooling is secondary.

The point-in-time correct join capability matters most in domains where labels are retrospective: credit risk, churn prediction, fraud. If your primary use case is real-time recommendation with near-instant feedback loops, this is less critical and the implementation complexity may not be worth it.

Uber's situation also involved a large, well-resourced engineering organisation. A team of 10 data scientists does not need Cassandra and Spark. Several open-source options, including Feast and Hopsworks Community Edition, allow smaller organisations to implement the catalogue and lineage principles with lighter infrastructure. The principle scales down; the original implementation does not.

One thing that often gets lost in discussions of feature stores is monitoring. Features drift over time as upstream data distributions shift. A feature store without monitoring on the statistical properties of features entering production is an incomplete data supply chain. Uber built monitoring into Michelangelo; many organisations adopt a feature store and treat monitoring as a later problem, which it rarely is.

The feature store is not primarily a way to make ML faster, though it often does that. It is a way to make ML auditable, reusable, and consistent across the gap between training and serving. Those properties matter most when models are in regulated domains or when production failures are expensive. Start with the catalogue and governance structure, and let the infrastructure follow the actual scale of your use case.