In 2019, a large European bank ran the analytics setup that most enterprises still run today: a Teradata warehouse feeding the regulatory and BIBITechnologies and processes that turn raw data into actionable insights via reporting, dashboards and analysis, so teams can decide based on facts rather than intuition.Voir la définition complète → teams, and a Hadoop lake feeding the data scientists. Between them sat 40 nightly ETLETLETL (Extract, Transform, Load) is a data integration process that pulls data from sources, reshapes it into a consistent format, and writes it into a target system.Voir la définition complète → jobs whose only purpose was to copy data from one system into the other so both sides could see roughly the same numbers. The reconciliation team existed *solely* to explain why the warehouse's "customer count" and the lake's "customer count" disagreed by 3%.
When the bank finally tallied the cost of that duplication—storage paid twice, pipelines maintained twice, two governance regimes, a headcount whose job was reconciliation—the figure crossed nine digits over the platform's life. None of it produced a single insight. It was the tax you pay for maintaining two copies of the truth.
The lakehouselakehouseA hybrid architecture combining the flexibility of a data lake with the analytical capabilities of a data warehouse, on a single storage layer.Voir la définition complète → is, at its core, an architectural bet that you can stop paying that tax. This lesson is about whether that bet pays off for *your* organization—because for a meaningful minority of enterprises, the two-system split is still the right call.
You already know the warehouse-plus-lake pattern. The lakehouselakehouseA hybrid architecture combining the flexibility of a data lake with the analytical capabilities of a data warehouse, on a single storage layer.Voir la définition complète → doesn't invent new user needs; it re-draws the boundary between storage and compute so that one governed copy of data can serve both BIBITechnologies and processes that turn raw data into actionable insights via reporting, dashboards and analysis, so teams can decide based on facts rather than intuition.Voir la définition complète → and ML.
The mechanism that makes this possible is the open table format—Delta Lake, Apache Iceberg, or Apache Hudi. Understand this layer and you understand the whole architecture; miss it and every vendor pitch will sound identical.
A raw data lakedata lakeA data lake is a centralized repository that stores large volumes of raw data in its native format, from structured tables to unstructured files, until needed.Voir la définition complète → was just files (Parquet, ORC) sitting in object storage. Cheap, scalable, and dumb: no transactions, no schemaschemaA schema is the formal blueprint that defines how data is structured, named, typed, and related within a database, file, or message.Voir la définition complète → enforcement, no way to update a single row without rewriting a whole partition. That's why you couldn't run serious BIBITechnologies and processes that turn raw data into actionable insights via reporting, dashboards and analysis, so teams can decide based on facts rather than intuition.Voir la définition complète → on it. The table format adds a metadatametadataDonnées sur les données, informations décrivant le contexte, la structure, la provenance et les caractéristiques d'un asset de données (auteur, date, format, source, définition). layer *on top of* those files that provides the things a warehouse always had:
The strategic consequence: your data sits once, in open-format files in your own object storage, and *multiple engines* point at it. A SQLSQLSales Qualified Lead: a prospect the sales team has validated as ready for direct outreach and a proposal, having passed clear qualification criteria.Voir la définition complète → engine serves the BIBITechnologies and processes that turn raw data into actionable insights via reporting, dashboards and analysis, so teams can decide based on facts rather than intuition.Voir la définition complète → analyst. A Spark or Python cluster serves the data scientist. A streaming engine writes fresh events into the same tables. Nobody copies anything.
-- Same governed Delta table, two very different consumers
-- BI analyst:
SELECT region, SUM(revenue) FROM sales.transactions
WHERE order_date >= '2024-01-01' GROUP BY region;
-- Data scientist, same table, reading a reproducible snapshot:
SELECT * FROM sales.transactions VERSION AS OF 142;That VERSION AS OF clause is worth pausing on. In the two-system world, a data scientist who trained a model in March could rarely reconstruct the exact training data six months later when a regulator or a model-risk committee asked. Time travel makes "show me the data this model saw" a one-line query. For any CDO in a regulated industry, that single capability often justifies the architecture on its own.
The deeper shift isn't the file format—it's that storage and compute are now independently scalable and independently priced. In a classic warehouse, you bought a coupled appliance: more storage meant a bigger box, whether or not you needed the horsepower. In the lakehouselakehouseA hybrid architecture combining the flexibility of a data lake with the analytical capabilities of a data warehouse, on a single storage layer.Voir la définition complète →, storage is commodity object storage at a few cents per gigabyte, and you spin compute up and down against it on demand.
This changes the CDO's cost conversation from "how big is our warehouse" to "what workloads are we running and when." It's a better conversation, but it introduces a new failure mode we'll return to: without discipline, decoupled compute is a bill that scales with carelessness.
The vendor narrative is that the lakehouselakehouseA hybrid architecture combining the flexibility of a data lake with the analytical capabilities of a data warehouse, on a single storage layer.Voir la définition complète → dominates the old split on every axis. It does not. Here is the judgement a CDO actually has to make.
Where the lakehouse wins decisively:
Where the warehouse-plus-lake still wins:
The decision rule I give CDOs: the lakehouse's advantage grows with the size of your data, the seriousness of your ML agenda, and the pain of your current duplication. If all three are high, the case is overwhelming. If your world is small-data, BIBITechnologies and processes that turn raw data into actionable insights via reporting, dashboards and analysis, so teams can decide based on facts rather than intuition.Voir la définition complète →-only, and running smoothly, the lakehouselakehouseA hybrid architecture combining the flexibility of a data lake with the analytical capabilities of a data warehouse, on a single storage layer.Voir la définition complète → is a solution shopping for a problem.
Two operational realities never appear in the sales deck, and both land on the CDO's desk.
The small-file problem. Streaming ingestion writes many tiny files. Left unmanaged, query performance degrades badly because the engine spends its time opening thousands of files instead of reading data. This is manageable—Delta and Iceberg have compaction and clustering—but it's a *permanent operational responsibility*, not a one-time setup. Budget for the platform engineering to own it.
Compute cost sprawl. Because compute is elastic and easy to spin up, ungoverned lakehouses produce shocking bills. A single analyst running an unoptimized query against a petabyte table can burn thousands of dollars in minutes. The two-system world had a natural cost ceiling in its fixed appliance; the lakehouselakehouseA hybrid architecture combining the flexibility of a data lake with the analytical capabilities of a data warehouse, on a single storage layer.Voir la définition complète → trades that ceiling for flexibility, and flexibility without FinOps discipline is just an uncapped credit card. Instituting workload tagging, cost attributionattributionA framework for assigning credit to the touchpoints that contributed to a conversion, so you can measure which channels and interactions actually drive results.Voir la définition complète → by team, and query governance is not optional—it's the price of admission.
Assume you've decided the lakehouselakehouseA hybrid architecture combining the flexibility of a data lake with the analytical capabilities of a data warehouse, on a single storage layer.Voir la définition complète → is right. The mistake I see most often is treating this as a lift-and-shift infrastructure project. It isn't. It's a governance and workload-migration program that happens to involve infrastructure. Here's how to sequence it.
Start with the medallion architecture as your organizing discipline. You've likely heard the bronze/silver/gold framing; the point for a CDO is that it maps cleanly to ownership and trust:
The reason this matters strategically: it prevents the lakehouselakehouseA hybrid architecture combining the flexibility of a data lake with the analytical capabilities of a data warehouse, on a single storage layer.Voir la définition complète → from decaying into the "data swamp" that killed the first generation of data lakes. The medallion layers are governance boundaries, not just processing stages. Make ownership of each layer explicit in writing before you migrate anything.
Migrate by workload, not by dataset. Don't try to boil the ocean by moving all your data first. Pick a bounded, high-pain workload—ideally one that currently *requires* copying data between your warehouse and lake—and move that end-to-end. The reconciliation-heavy analytics the European bank suffered from is the ideal first candidate: migrating it proves the single-source-of-truth thesis and delivers a visible headcount saving.
Run in parallel and reconcile before you cut over. For the first migrated workloads, run old and new side by side and prove the numbers match. This is tedious and unglamorous, and it is the single most important trust-building activity in the entire program. The first time a migrated dashboard shows a different number than the legacy one and you *can't* explain why, you lose the business's confidence for a year.
Vérification des acquis
1. In the story of the European bank, what was the fundamental problem that the nine-digit cost actually represented?
2. According to the lesson, what does the lakehouse fundamentally re-draw or change?
3. Why does the lesson single out the open table format as the layer you must understand to understand the whole architecture?
4. Select ALL correct answers about what a raw data lake (files in object storage) lacked before table formats.
Sélectionnez toutes les réponses correctes.
5. Select ALL correct answers about the reasoning the lesson uses to frame the lakehouse decision.
Sélectionnez toutes les réponses correctes.
Decide your table format deliberately—it's a strategic, not a technical, choice. Delta Lake is deeply integrated with Databricks and has the most mature tooling; if you're a Databricks shop, it's the path of least resistance but couples you more tightly to that ecosystem. Apache Iceberg has become the neutral, multi-engine standard—Snowflake, BigQuery, AWS, and others all support it—making it the stronger choice when your explicit goal is engine independence and negotiating leverage. This is a decision to make at the CDO level *with* your architects, because it determines how much lock-in leverage you retain for the next decade. Do not let it be made implicitly by whoever runs the first proof-of-concept.
Don't dissolve your warehouse reflexively. The most sophisticated enterprises run a hybrid: lakehouselakehouseA hybrid architecture combining the flexibility of a data lake with the analytical capabilities of a data warehouse, on a single storage layer.Voir la définition complète → as the governed single source of truth, with a subset of gold-layer data still served through a specialized warehouse or serving layer for the highest-concurrency BIBITechnologies and processes that turn raw data into actionable insights via reporting, dashboards and analysis, so teams can decide based on facts rather than intuition.Voir la définition complète →. The open table format makes this viable—an Iceberg table can be queried by both your Spark jobs and Snowflake. Purity is not the goal; eliminating duplicated *truth* is. You can have one governed copy and still push it into a fast serving engine, because that's a performance projection, not a second source of truth.
The architecture unifies storage. It does not automatically unify your *people*, and that's where lakehouselakehouseA hybrid architecture combining the flexibility of a data lake with the analytical capabilities of a data warehouse, on a single storage layer.Voir la définition complète → programs quietly fail. In the two-system world, the BIBITechnologies and processes that turn raw data into actionable insights via reporting, dashboards and analysis, so teams can decide based on facts rather than intuition.Voir la définition complète → team and the ML team had separate platforms, separate governance, and separate rituals. Put them on one store and you've created a shared dependency between two groups that may never have collaborated.
The CDO's job here is to make the shared platform a shared *responsibility*. Someone must own each medallion layer. Someone must own compaction and cost governance. The definition of a "certified" gold table must be agreed across BIBITechnologies and processes that turn raw data into actionable insights via reporting, dashboards and analysis, so teams can decide based on facts rather than intuition.Voir la définition complète → and ML, because they will now be drinking from the same well. If you migrate the technology but leave the org in its old silos, you'll rebuild the two-system split inside a single platform—same duplication, now hidden.
1. The table format is the whole game. Delta, Iceberg, or Hudi is what turns a dumb lake into a governed store serving both BIBITechnologies and processes that turn raw data into actionable insights via reporting, dashboards and analysis, so teams can decide based on facts rather than intuition.Voir la définition complète → and ML. Evaluate lakehouselakehouseA hybrid architecture combining the flexibility of a data lake with the analytical capabilities of a data warehouse, on a single storage layer.Voir la définition complète → proposals on their table-format strategy, and choose Iceberg when engine-independence and vendor leverage are priorities.
2. Score the decision on three axes: data scale, ML seriousness, and duplication pain. High on all three makes the lakehouselakehouseA hybrid architecture combining the flexibility of a data lake with the analytical capabilities of a data warehouse, on a single storage layer.Voir la définition complète → overwhelming; low on all three means a plain warehouse beats both the lakehouselakehouseA hybrid architecture combining the flexibility of a data lake with the analytical capabilities of a data warehouse, on a single storage layer.Voir la définition complète → and the old split. Don't adopt the architecture for prestige.
3. Budget for the operational tax the demo hides. The small-file problem and elastic-compute cost sprawl are permanent responsibilities. Stand up FinOps discipline—workload tagging, cost attributionattributionA framework for assigning credit to the touchpoints that contributed to a conversion, so you can measure which channels and interactions actually drive results.Voir la définition complète →, query governance—*before* go-live, not after the first shocking bill.
4. Migrate by high-pain workload, run in parallel, and reconcile obsessively. The first unexplained number discrepancy costs you a year of business trust. Prove the single-source-of-truth thesis on one reconciliation-heavy workload before scaling.
5. Unify the org, not just the storage. Assign explicit ownership to each medallion layer and agree a shared definition of a certified table across BIBITechnologies and processes that turn raw data into actionable insights via reporting, dashboards and analysis, so teams can decide based on facts rather than intuition.Voir la définition complète → and ML—otherwise you'll rebuild your silos inside one platform.