Cloud data infrastructure: services, costs & migration
Cloud has changed data architecture permanently. On-premise data infrastructure, the owned servers, the SAN storage, the proprietary appliances, is increasingly a legacy choice. But cloud data infrastructure isn't a single thing. It's a spectrum of services and trade-offs that require deliberate architectural decisions.
Cloud-native data services
Every major cloud provider offers a complete data stack. Here's what each one includes:
AWS: S3 (storage), Glue (ETLETLETL (Extract, Transform, Load) is a data integration process that pulls data from sources, reshapes it into a consistent format, and writes it into a target system.View full definition →/catalog), Redshift (warehouse), Athena (serverless SQLSQLSales Qualified Lead: a prospect the sales team has validated as ready for direct outreach and a proposal, having passed clear qualification criteria.View full definition → on S3), EMR (Spark), SageMaker (ML), Kinesis (streaming), Lake Formation (governance).
Google Cloud: GCS (storage), BigQuery (warehouse + lake hybrid), Dataflow (streaming/batch ETL), Vertex AI (ML), Pub/Sub (messaging), Dataplex (governance).
Azure: ADLS Gen2 (storage), Synapse Analytics (warehouse + lake), Data Factory (ETL), Azure ML, Event Hubs (streaming), Microsoft Purview (governance).
Cloud Data Architecture Patterns
Knowledge check
1. According to the lesson, what is the recommended practical answer to the build vs. buy decision?
2. Why does the lesson suggest that open table formats (Iceberg, Delta Lake) and open compute frameworks (Spark) are valuable?
3. According to the lesson, what is the most common actual reason organizations end up running multi-cloud architectures?
4. Select ALL statements that reflect the case for BUYING vendor data products (as opposed to building).
Select all the correct answers.
5. Select ALL correct statements about cloud data infrastructure as presented in the lesson.
Select all the correct answers.
The build vs. buy decision
Every CDO faces this decision repeatedly: build custom solutions or buy vendor products?
The case for buying: Speed to value. A Snowflake implementation takes weeks, not years. The vendor manages infrastructure, performance tuning, and upgrades. Your team focuses on data problems, not infrastructure problems.
The case for building: Control. You own the code, the schemaschemaA schema is the formal blueprint that defines how data is structured, named, typed, and related within a database, file, or message.View full definition →, the optimization. No vendor lock-in. No per-query pricing surprises. Open source (Spark, Iceberg, Airflow) gives you full flexibility.
The practical answer: Buy the commodity components (storage, compute, orchestration SaaS), build what differentiates your business (proprietary data models, custom ML pipelines, domain-specific features).
Multi-cloud and cloud-agnostic architecture
Multi-cloud sounds appealing (no vendor lock-in, best-of-breed services) but comes with real costs: operational complexity, data transfer costs, team skill fragmentation. Most organizations using multi-cloud do so for regulatory reasons (data residency requirements) or as the result of M&A activity, not by deliberate choice.
Open table formats (Apache Iceberg, Delta Lake) and open compute frameworks (Apache Spark) reduce lock-in without requiring multi-cloud complexity. Use them as your hedge.
Cost architecture in the cloud
Cloud data costs are non-trivial and easy to underestimate. The main cost drivers:
- Storage: Cheap but scales with retention. Implement lifecycle policies, move data to cheaper tiers after 90 days, archive after 1 year.
- Compute: The main variable cost. Warehouse query costs scale with data scanned. Clustering and partitioning reduce data scanned, which cuts cost directly.
- Egress: Moving data between regions or to on-premise is expensive. Design to minimize cross-region data movement.
- Tools and licenses: Snowflake, Databricks, and similar tools layer pricing on top of cloud costs. Model total cost before committing.
At Spotify, a dedicated "data platform cost optimization" team cut cloud spend by 30% without reducing functionality, through query optimization rules, data retention policies, and compute scheduling. A CDO who doesn't own cost architecture doesn't really own the data platform.
Migration strategy: on-prem to cloud
Most CDOs inherit some on-premise infrastructure. Cloud migration is almost always the right direction but requires sequencing:
- Assess and inventory: What exists, what it costs, what it supports
- Identify low-risk candidates: Historical data, archival data, reporting workloads
- Lift-and-shift first: Move workloads before optimizing them, proving cloud works reduces organizational resistance
- Refactor incrementally: Optimize for cloud-native patterns after stabilization
- Decommission progressively: Kill on-premise only after cloud workloads are stable
Avoid the trap of re-architecting everything at once. Many cloud migrations fail not because of technical complexity but because the scope is too ambitious.
Quiz Questions
- Quelle est la principale raison pour laquelle la plupart des organisations adoptent le multi-cloud ?
A) Pour réduire les coûts
B) Pour des raisons réglementaires ou suite à des M&A
C) Pour améliorer les performances
D) Pour simplifier l'architecture
Réponse: B
- Quel est le principal avantage d'utiliser des formats de tables ouverts comme Apache Iceberg ?
A) Ils sont plus rapides que les solutions propriétaires
B) Ils réduisent les coûts de stockage
C) Ils réduisent le vendor lock-in sans nécessiter une architecture multi-cloud
D) Ils sont maintenus par les fournisseurs cloud
Réponse: C
- Dans la stratégie de migration cloud, quelle est la bonne séquence ?
A) Refactoriser d'abord, puis migrer
B) Tout migrer et optimiser simultanément
C) Inventorier → migrer les cas simples → lift-and-shift → refactoriser → décommissionner
D) Décommissionner l'on-premise avant de migrer
Réponse: C
What to do, from this lesson
These actions are compiled in the role's Playbook.
- Migrate to cloud by inventory, lift-and-shift, then refactor incrementally
Related articles
Recent articles from the blog that build on this lesson.
- Datalag_tolerance is a budget decision now, and most teams have not made itdbt State went generally available in September 2026, turning "rebuild everything on a schedule" into "rebuild only what changed." The savings are real, but they only land if you decide, model by model, how stale your data is allowed to be.
- DataCutting cloud data costs without breaking analyticsCloud bills for data infrastructure have become one of the fastest-growing line items in enterprise IT budgets, and most organizations are overpaying without realizing it. This playbook gives CDOs a concrete sequence of moves to reduce spend significantly while keeping analytical capability intact.
- DataWhy your data architecture is lying to you, and what modern CDOs are doing about itMost enterprises believe they have a data architecture. What they actually have is a collection of historical accidents held together by good intentions and expensive middleware. Here's how the CDOs redefining competitive advantage are building differently.