How Cloudflare rebuilt its data stack around dbt, Fivetran, and Airflow

By mid-2022, Cloudflare's data engineering team was maintaining hundreds of bespoke Python scripts that moved data from product databases, Salesforce, Stripe, and a dozen other sources into Snowflake. The scripts had accumulated over years of fast hiring, each written to slightly different conventions, with no shared testing framework and no consistent way to understand what ran when. When pipelines broke, the failure mode was typically silent: a downstream dashboard would go stale and someone in finance would notice three days later. For a company processing trillions of DNS queries per day and selling to enterprise customers on usage-based contracts, stale revenue data was not an abstract problem.

The team faced a choice common in fast-scaling B2B SaaS companies: continue patching the existing system or commit to a deliberate architectural reset. They chose the latter, and the mechanics of how they did it carry lessons that generalise well beyond Cloudflare's specific context.

What they did

The architectural bet was on three components working together: Fivetran for managed ingestion, dbt (data build tool) for transformation, and Apache Airflow for orchestration. Each layer had a specific rationale.

Fivetran (a commercial vendor, so these characterisations should be read alongside independent benchmarks) replaced the bespoke ingestion scripts. The logic was straightforward: maintaining CDC (change data capture) connectors for Salesforce, Zendesk, and Stripe is not a source of competitive advantagecompetitive advantageA lasting edge over competitors: a resource, capability or position they cannot easily replicate, letting a firm earn above-average returns over time.View full definition → for a network infrastructure company. Buying that maintenance contract freed engineers from connector upkeep and moved ingestion failure handling to a vendor SLA. The tradeoff is cost and lock-in, which Cloudflare accepted because their engineering time had a higher opportunity cost than the subscription fee.

The dbt layer is where the more interesting architectural decisions happened. Cloudflare structured their dbt project around a staging/intermediate/mart pattern that has since become a widely adopted convention, partly because dbt Labs (the commercial company behind dbt Core and dbt Cloud) has promoted it, though independent practitioner communities have validated it as genuinely useful. Staging models did nothing except rename and lightly clean columns from raw sources. Intermediate models joined and filtered. Mart models were the only layer that business users and BIBITechnologies and processes that turn raw data into actionable insights via reporting, dashboards and analysis, so teams can decide based on facts rather than intuition.View full definition → tools queried directly. This separation meant that when Salesforce changed a field name, the fix was isolated to one staging model rather than scattered across dozens of downstream queries.

The team also invested heavily in dbt's testing framework, writing not-null, uniqueness, and referential integrity tests on every model. This sounds basic but was the mechanism by which silent failures became loud ones. When a Fivetran sync delivered a malformed batch, the dbt tests caught it before the mart layer ran, and Airflow surfaced the failure as a broken DAG step with a Slack alert. The loop from failure to detection went from days to minutes.

Airflow handled orchestration because Cloudflare already had Kubernetes infrastructure and the internal Python expertise to manage it. This is worth flagging: Airflow is operationally expensive to run well. Teams without existing Kubernetes familiarity often find that managed alternatives like Prefect, Dagster, or dbt Cloud's own scheduler reduce the total operational burden significantly. Cloudflare's choice was context-dependent, not universally correct.

The data contract layer they added later

One element that is underreported in most ELTELTELT (Extract, Load, Transform) is a data integration pattern where raw data is loaded into a target system first, then transformed inside it using the platform's compute power.View full definition → case studies is what Cloudflare's team built on top of the standard stack. By late 2023 they had introduced lightweight data contracts at the staging layer: explicit schemaschemaA schema is the formal blueprint that defines how data is structured, named, typed, and related within a database, file, or message.View full definition → definitions that upstream producing teams were expected to honour, with automated validation in the dbt pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition →. When a backend engineering team changed the events schema in their Kafka topic, the contract check failed before any downstream model ran, and the producing team received the alert rather than the data team absorbing the breakage silently. This organisational shift, making producers responsible for schema stability, was as consequential as any tooling choice.

The results

Cloudflare has not published a detailed public postmortem with specific percentage improvements, so some of what follows is drawn from conference talks and engineering blog posts rather than audited figures. The claims should be read with that caveat.

Their engineering blog noted that the number of data incidents requiring manual intervention dropped materially in the six months after the dbt testing framework was fully deployed. Pipeline development time for new data sources fell, by the team's own estimate, from several weeks (including connector maintenance) to days using Fivetran plus a new dbt staging model. The business impact they cited most frequently was on revenue recognition: finance teams were able to close monthly reporting on a consistent, repeatable schedule rather than waiting for ad hoc data fixes.

What is verifiable from public information is that Cloudflare's data team headcount did not scale linearly with the company's data volume growth between 2022 and 2025. That is meaningful: it suggests the architectural investment delivered leverage, though isolating the ELT stack's contribution from other factors (hiring quality, tooling elsewhere) is not straightforward.

What transfers

The pattern that works across contexts is the strict separation between ingestion, transformation, and orchestration as distinct concerns with distinct ownership. When those three functions are collapsed into monolithic scripts, every change carries risk across the entire pipeline. Separating them creates a system where a connector failure does not corrupt transformation logic, and a transformation bug does not mask an ingestion problem.

The data contract approach transfers particularly well to organisations where multiple product teams feed a central data warehousedata warehouseA central repository that consolidates data from many source systems into a structured, query-optimized store designed for analytics, reporting, and business intelligence.View full definition →. Without contracts, the data team becomes a permanent intermediary absorbing every upstream change. With contracts, schema stability becomes a shared engineering responsibility.

Where Cloudflare's context differs from most: they had strong internal Kubernetes infrastructure and a data engineering team with senior Python talent. Smaller organisations or those without that infrastructure should seriously evaluate managed orchestration before defaulting to self-hosted Airflow. The tooling is not the point. The architectural separation is.

The practical starting point for most teams is not to replicate Cloudflare's full stack at once. Deploy dbt tests on your most critical mart models first, surface failures to a Slack channel, and watch how quickly the organisation's relationship with data reliability changes. That feedback loop is the mechanism. Everything else is implementation detail.