+180 XP

The lakehouse: unifying analytics & ML

The $200 million copy job

In 2019, a large European bank ran the analytics setup that most enterprises still run today: a Teradata warehouse feeding the regulatory and BI teams, and a Hadoop lake feeding the data scientists. Between them sat 40 nightly ETL jobs whose only purpose was to copy data from one system into the other so both sides could see roughly the same numbers. The reconciliation team existed *solely* to explain why the warehouse's "customer count" and the lake's "customer count" disagreed by 3%.

When the bank finally tallied the cost of that duplication, storage paid twice, pipelines maintained twice, two governance regimes, and a headcount whose job was reconciliation, the figure crossed nine digits over the platform's life. None of it produced a single insight. It was the tax you pay for maintaining two copies of the truth.

The lakehouse is, at its core, an architectural bet that you can stop paying that tax. This lesson is about whether that bet pays off for *your* organization, because for a meaningful minority of enterprises, the two-system split is still the right call.

What the lakehouse actually changes

You already know the warehouse-plus-lake pattern. The lakehouse doesn't invent new user needs; it re-draws the boundary between storage and compute so that one governed copy of data can serve both BI and ML.

The mechanism that makes this possible is the open table format, Delta Lake, Apache Iceberg, or Apache Hudi. Understand this layer and you understand the whole architecture; miss it and every vendor pitch will sound identical.

A raw data lake was just files (Parquet, ORC) sitting in object storage. Cheap, scalable, and dumb: no transactions, no schema enforcement, no way to update a single row without rewriting a whole partition. That's why you couldn't run serious BI on it. The table format adds a metadata layer *on top of* those files that provides the things a warehouse always had:

  • ACID transactions, concurrent writes don't corrupt reads.
  • Schema enforcement and evolution, bad data gets rejected at write time; columns can be added safely.
  • Time travel, query the table as it existed at a prior version, which transforms ML reproducibility and audit.
  • Upserts and deletes, critical for GDPR "right to be forgotten" and CDC ingestion.

The strategic consequence: your data sits once, in open-format files in your own object storage, and *multiple engines* point at it. A SQL engine serves the BI analyst. A Spark or Python cluster serves the data scientist. A streaming engine writes fresh events into the same tables. Nobody copies anything.

sql
-- Same governed Delta table, two very different consumers
-- BI analyst:
SELECT region, SUM(revenue) FROM sales.transactions
WHERE order_date >= '2024-01-01' GROUP BY region;

-- Data scientist, same table, reading a reproducible snapshot:
SELECT * FROM sales.transactions VERSION AS OF 142;

That VERSION AS OF clause is worth pausing on. In the two-system world, a data scientist who trained a model in March could rarely reconstruct the exact training data six months later when a regulator or a model-risk committee asked. Time travel makes "show me the data this model saw" a one-line query. For any CDO in a regulated industry, that single capability often justifies the architecture on its own.

Separating storage and compute is the real shift

The deeper shift isn't the file format, it's that storage and compute are now independently scalable and independently priced. In a classic warehouse, you bought a coupled appliance: more storage meant a bigger box, whether or not you needed the horsepower. In the lakehouse, storage is commodity object storage at a few cents per gigabyte, and you spin compute up and down against it on demand.

This changes the CDO's cost conversation from "how big is our warehouse" to "what workloads are we running and when." It's a better conversation, but it introduces a new failure mode we'll return to: without discipline, decoupled compute is a bill that scales with carelessness.

The honest tradeoff table

The vendor narrative is that the lakehouse dominates the old split on every axis. It does not. Here is the judgement a CDO actually has to make.

Where the lakehouse wins decisively:

  • Single source of truth. One copy, one governance model, one lineage graph. The reconciliation team disappears. This is the biggest structural win and the hardest to quantify until you've lived the pain of duplication.
  • ML and unstructured data. Data scientists work against production-grade governed tables instead of stale extracts, and the same store holds images, text, and audio that a warehouse can't. If your strategy is genuinely AI-forward, this is not optional.
  • Cost at scale. Object storage plus elastic compute is dramatically cheaper than warehouse storage for large, cold, or infrequently queried data.
  • No vendor lock-in at the storage layer. Your data is open-format files you own. You can change query engines without a migration project. This is real leverage in vendor negotiations, use it.

Where the warehouse-plus-lake still wins:

  • Low-latency, high-concurrency BI. A mature warehouse (Snowflake, BigQuery, Teradata) serving thousands of concurrent dashboard users with sub-second response is still, in many cases, faster and more predictable than a lakehouse tuned for the same job. The gap is closing fast, but it exists.
  • Operational maturity. Warehouses have 30 years of tooling, DBA expertise, and battle-tested optimizers. The lakehouse stack demands scarcer skills, people who understand file compaction, small-file problems, and partition design. If you don't have that talent, the "cheaper" architecture gets expensive.
  • Simplicity for pure-BI shops. If you have no serious ML ambition and modest data volumes, a warehouse alone is simpler than either the lakehouse *or* the two-system split. Don't adopt a lakehouse to solve a problem you don't have.

The decision rule I give CDOs: the lakehouse's advantage grows with the size of your data, the seriousness of your ML agenda, and the pain of your current duplication. If all three are high, the case is overwhelming. If your world is small-data, BI-only, and running smoothly, the lakehouse is a solution shopping for a problem.

Data Warehouse vs Data Lake vs Data Lakehouse

Watch on YouTube

The hidden costs nobody demos

Two operational realities never appear in the sales deck, and both land on the CDO's desk.

The small-file problem. Streaming ingestion writes many tiny files. Left unmanaged, query performance degrades badly because the engine spends its time opening thousands of files instead of reading data. This is manageable, Delta and Iceberg have compaction and clustering, but it's a *permanent operational responsibility*, not a one-time setup. Budget for the platform engineering to own it.

Compute cost sprawl. Because compute is elastic and easy to spin up, ungoverned lakehouses produce shocking bills. A single analyst running an unoptimized query against a petabyte table can burn thousands of dollars in minutes. The two-system world had a natural cost ceiling in its fixed appliance; the lakehouse trades that ceiling for flexibility, and flexibility without FinOps discipline is just an uncapped credit card. Workload tagging, cost attribution by team, and query governance are not optional, they're the price of admission.

Migration judgement: how to actually get there

Assume you've decided the lakehouse is right. The mistake I see most often is treating this as a lift-and-shift infrastructure project. It isn't. It's a governance and workload-migration program that happens to involve infrastructure. Here's how to sequence it.

Start with the medallion architecture as your organizing discipline. You've likely heard the bronze/silver/gold framing; the point for a CDO is that it maps cleanly to ownership and trust:

  • Bronze, raw ingested data, owned by platform/ingestion teams. Nobody builds on it directly.
  • Silver, cleaned, conformed, deduplicated. Owned by data engineering. This is where your single source of truth actually lives.
  • Gold, business-level aggregates and ML feature tables, owned by domain teams and analysts.

Why this matters strategically: it prevents the lakehouse from decaying into the "data swamp" that killed the first generation of data lakes. The medallion layers are governance boundaries, not just processing stages. Make ownership of each layer explicit in writing before you migrate anything.

Migrate by workload, not by dataset. Don't try to boil the ocean by moving all your data first. Pick a bounded, high-pain workload, ideally one that currently *requires* copying data between your warehouse and lake, and move that end-to-end. The reconciliation-heavy analytics the European bank suffered from is the ideal first candidate: migrating it proves the single-source-of-truth thesis and delivers a visible headcount saving.

Run in parallel and reconcile before you cut over. For the first migrated workloads, run old and new side by side and prove the numbers match. This is tedious and unglamorous, and it is the single most important trust-building activity in the entire program. The first time a migrated dashboard shows a different number than the legacy one and you *can't* explain why, you lose the business's confidence for a year.

Vérification des acquis

1. What is the core architectural change that the lakehouse introduces relative to the traditional warehouse-plus-lake pattern?

2. Why is the open table format described as the layer you must understand to grasp the entire lakehouse architecture?

3. The 'reconciliation team' story illustrates which underlying problem that the lakehouse aims to solve?

CHOIX MULTIPLES

4. Select ALL capabilities that an open table format adds on top of raw files in object storage.

Sélectionnez toutes les réponses correctes.

CHOIX MULTIPLES

5. Select ALL statements that correctly describe why a raw data lake alone could not support serious BI.

Sélectionnez toutes les réponses correctes.

Decide your table format deliberately, it's a strategic, not a technical, choice. Delta Lake is tightly integrated with Databricks and has the most mature tooling; if you're a Databricks shop, it's the path of least resistance but couples you more tightly to that ecosystem. Apache Iceberg has become the neutral, multi-engine standard, Snowflake, BigQuery, AWS, and others all support it, making it the stronger choice when your explicit goal is engine independence and negotiating leverage. Make this decision at the CDO level *with* your architects, because it determines how much lock-in leverage you retain for the next decade. Don't let it be made implicitly by whoever runs the first proof-of-concept.

Don't dissolve your warehouse reflexively. The most sophisticated enterprises run a hybrid: lakehouse as the governed single source of truth, with a subset of gold-layer data still served through a specialized warehouse or serving layer for the highest-concurrency BI. The open table format makes this viable, an Iceberg table can be queried by both your Spark jobs and Snowflake. Purity is not the goal; eliminating duplicated *truth* is. You can have one governed copy and still push it into a fast serving engine, because that's a performance projection, not a second source of truth.

The organizational change that decides success

The architecture unifies storage. It does not automatically unify your *people*, and that's where lakehouse programs quietly fail. In the two-system world, the BI team and the ML team had separate platforms, separate governance, and separate rituals. Put them on one store and you've created a shared dependency between two groups that may never have collaborated.

The CDO's job here is to make the shared platform a shared *responsibility*. Someone must own each medallion layer. Someone must own compaction and cost governance. The definition of a "certified" gold table must be agreed across BI and ML, because they will now be drinking from the same well. If you migrate the technology but leave the org in its old silos, you'll rebuild the two-system split inside a single platform, same duplication, now hidden.

Key Takeaways

1. The table format is the whole game. Delta, Iceberg, or Hudi is what turns a dumb lake into a governed store serving both BI and ML. Evaluate lakehouse proposals on their table-format strategy, and choose Iceberg when engine-independence and vendor leverage are priorities.

2. Score the decision on three axes: data scale, ML seriousness, and duplication pain. High on all three makes the lakehouse overwhelming; low on all three means a plain warehouse beats both the lakehouse and the old split. Don't adopt the architecture for prestige.

3. Budget for the operational tax the demo hides. The small-file problem and elastic-compute cost sprawl are permanent responsibilities. Stand up FinOps discipline, workload tagging, cost attribution, query governance, *before* go-live, not after the first shocking bill.

4. Migrate by high-pain workload, run in parallel, and reconcile obsessively. The first unexplained number discrepancy costs you a year of business trust. Prove the single-source-of-truth thesis on one reconciliation-heavy workload before scaling.

5. Unify the org, not just the storage. Assign explicit ownership to each medallion layer and agree a shared definition of a certified table across BI and ML, otherwise you'll rebuild your silos inside one platform.

À faire, tiré de cette leçon

Ces actions sont compilées dans le plan d'action du rôle.

  • Choose architecture using workload, team skills, volume, and lock-in tolerance
  • Standardize pipelines on the Medallion bronze-silver-gold layering pattern
Voir le plan d'action complet →

Précédent

Event-driven architecture & streaming platforms

Retour au parcours