Event-driven architecture & streaming platforms

# Event-driven architecture and streaming platforms

In 2017, a batch-oriented retailer discovered that its "fraud detection system" was catching fraud an average of nine hours after the transaction cleared. The model was excellent. The pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → was the problem: fraud signals sat in a nightly ETLETLETL (Extract, Transform, Load) is a data integration process that pulls data from sources, reshapes it into a consistent format, and writes it into a target system.View full definition → job, waiting for the 2 a.m. run. By the time the alert fired, the goods had shipped and the card was maxed. The fix was not a better model. It was a different *shape* of data movement, one where the transaction itself announced its own existence the instant it happened, and any number of systems could react in milliseconds.

That shift, from asking systems "what's the latest state?" to letting systems *announce that something changed*, is the whole of event-driven architecture. As a CDO, you already own the data strategy and the governance around it. What most data leaders under-appreciate is that the *architecture of data movement* is a strategic lever, not a plumbing detail. Get it wrong and every real-time ambition your CEO has, personalization, dynamic pricingdynamic pricingAutomatically adjusting prices in real time based on demand, competition or user behaviour to optimise revenue, margin or conversion.View full definition →, operational AI, dies in a nightly batch window.

What an event actually is (and why it changes everything)

An event is an immutable fact about something that happened, stamped with a time. OrderPlaced. PaymentFailed. SensorReadingExceededThreshold. CustomerLoggedIn. Note the past tense, this is not incidental. An event is not a command ("charge this card") and not a query ("what's the balance?"). It is a record that *the world moved*, and it cannot be un-happened.

This distinction matters more than it sounds. In your traditional world, the source of truth is a table, a snapshot of *current* state. A customer's address is whatever the row says today. In an event-driven world, the source of truth becomes the *log of everything that ever happened*, and current state is just a projection you compute from that log. The customer's address is the result of replaying every AddressChanged event.

This is the mental model shift most executives miss, so let me make it concrete with the practical consequences:

• State becomes reconstructable. If a downstream system corrupts its data, you replay the event log and rebuild it. Batch pipelines can't do this; the source rows have already been overwritten.
• New consumers can join late and still get history. A team spins up an ML feature storeML feature storeA centralised repository managing ML features, ensuring consistency between training and serving environments.View full definition → six months after launch and replays two years of events to backfill. No data engineering ticket, no source-system load.
• Time becomes a first-class citizen. You can ask "what did we know at 3:14 p.m.?", critical for audit, model debugging, and regulatory reconstruction.

The retailer's fraud problem was, at root, that they treated transactions as *state to be queried later* rather than *events to be reacted to now*.

The event as a contract

Here's where the CDO's governance instincts must extend into architecture. An event is a published contract between whoever produces it and everyone who consumes it. The schemaschemaA schema is the formal blueprint that defines how data is structured, named, typed, and related within a database, file, or message.View full definition → of OrderPlaced, its fields, types, and meaning, becomes a shared APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition → across the enterprise. Change it carelessly and you break systems you've never heard of.

This is why mature streaming organizations run a schema registry with enforced compatibility rules. You do not get to rename a field because it was convenient for your team. Events are strategic assets governed exactly like the data products in your monetization portfolio, versioned, owned, documented, and backward-compatible by policy.

Why decoupling producers from consumers is the real prize

The technical benefit everyone cites is "scale." That's true but shallow. The deeper prize is *organizational decoupling*, and that's a CDO concern, because your bottleneck is rarely compute. It's coordination between teams.

In a request-driven world, integration is point-to-point. The order system calls the inventory system, which calls the loyalty system, which calls the analytics system. Every new consumer means a new integration, a new dependency, a new meeting. With *N* systems you march toward *N²* connections, and every one is a place your architecture can seize up. This is the "integration spaghetti" that quietly consumes 40% of your engineering capacity.

Event-driven architecture inverts this. The producer publishes OrderPlaced to a stream and does not know or care who consumes it. The inventory team subscribes. Later the loyalty team subscribes. Later still, a data science team subscribes to build a churn feature, without ever talking to the order team. The producer's job ended when it published the fact.

Producer  →  [ OrderPlaced topic ]  →  Inventory service
                                    →  Loyalty service
                                    →  Fraud scoring
                                    →  Analytics / feature store

The organizational payoff is this: teams ship without cross-team negotiation. Your data platform stops being a queue of integration requests and becomes a marketplace where teams publish and subscribe on their own schedule. For a CDO trying to scale data usage across a company that outgrows your central team, this is the only model that works. Centralized point-to-point integration does not scale past a certain org size, it just converts your best engineers into full-time integration brokers.

The trade-offs you own, not delegate

Decoupling is not free, and the costs are exactly the ones a CDO must adjudicate because they cross team boundaries:

• Eventual consistency. Consumers process at their own pace. For a few hundred milliseconds, sometimes seconds, the inventory count and the order count disagree. Your business partners must understand that "real-time" means "eventually, quickly, and asynchronously," not "instantly consistent everywhere." This is a business conversation, not a technical one.
• Debugging becomes distributed. When something goes wrong, the story is spread across a dozen independently-moving consumers. You need distributed tracing and strong observability *from day one*, not bolted on after the first incident.
• Ordering and duplicates. Streaming platforms typically guarantee *at-least-once* delivery, meaning a consumer may see the same event twice. Every consumer must be idempotent, processing the same event twice produces the same result. This is a design discipline you must mandate as policy, because a single non-idempotent consumer that double-charges a customer becomes *your* headline.

The platforms: kafka vs. cloud pub/sub, and how to choose

You do not need to configure a broker. You do need to make the platform *decision* correctly, because it's a five-to-ten-year commitment with real cost and lock-in implications.

Apache Kafka (and its managed forms, Confluent Cloud, AWS MSK) is a *distributed, durable log*. Events are written to topics, partitioned for parallelism, and, critically, *retained*. Kafka keeps events for days, weeks, or forever. This is what enables replay, backfill, and treating the log as the source of truth. Consumers track their own position (the "offset") and can rewind. Kafka is the right default when you want the event log to be a lasting asset, when you need very high throughput, and when replay is central to your use cases (feature stores, event sourcing, audit).

Cloud Pub/Sub (Google Pub/Sub, AWS SNS/SQS, Azure Service Bus) is a *messaging service*. Its mental model is "deliver this message to subscribers, then move on." Retention is shorter and replay is more limited. In exchange, it is dramatically simpler to operate, fully serverless, near-zero ops overhead, scales elastically without you thinking about partitions. It's the right choice when you want event-driven decoupling *without* running a platform, and when replaying deep history is not a core requirement.

Here is the decision framework I'd put on one slide:

| If your priority is… | Lean toward… |

|---|---|

| Replay, backfill, log-as-source-of-truth | Kafka |

| Very high sustained throughput, stream processing | Kafka |

| Minimal ops, serverless, fast time-to-value | Cloud Pub/Sub |

| Simple fan-out notifications between services | Cloud Pub/Sub |

| Deep ecosystem (connectors, stream SQLSQLSales Qualified Lead: a prospect the sales team has validated as ready for direct outreach and a proposal, having passed clear qualification criteria.View full definition →, tooling) | Kafka / Confluent |

The mistake I see repeatedly: teams adopt Kafka for its prestige, then spend a year building operational muscle they didn't need for what was really a notification problem. Match the platform to the *use case's demand for history and throughput*, not to résumé-driven development.

One config idea worth internalizing

You don't write this, but you should recognize what it means when your architects debate it. In Kafka, a topic's partition count sets the ceiling on parallel consumption:

partitions = 12       # up to 12 consumers process this topic in parallel
retention.ms = 604800000   # keep every event for 7 days for replay

The judgment embedded here: partitions are hard to increase later without breaking ordering guarantees. Under-provision and you throttle throughput; over-provision and you waste resources and complicate rebalancing. When your team asks for a decision on partition strategy, they're really asking you to price the trade-off between future scale and present simplicity. That's a CDO call, not a junior engineer's.

Where a CDO actually applies this on monday morning

Frameworks are worthless without application. Here is where event-driven architecture earns its keep, and how to decide *whether it's warranted at all*.

1. Real-time operational analytics and AI. Your fraud example, dynamic pricingdynamic pricingAutomatically adjusting prices in real time based on demand, competition or user behaviour to optimise revenue, margin or conversion.View full definition →, recommendation engines, inventory reallocation, predictive maintenance. Any decision whose value *decays with time* is a candidate. The test: does a decision made in 200 milliseconds beat the same decision made in six hours? If yes, batch is leaving money on the table.

2. Feeding the feature store. This is the highest-leverage and most under-appreciated application. If your ML basics are already in place, the next maturity leap is *fresh features*. A model scoring on features that are 24 hours stale is guessing. Streaming events into your feature storefeature storeA centralised repository managing ML features, ensuring consistency between training and serving environments.View full definition → means the same event that drives operations also drives the model, eliminating the notorious training-serving skew where your batch pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → and your live pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → compute features differently.

3. Change Data Capture (CDC) as the on-ramp. You do not have to boil the ocean. CDC tools (Debezium is the common one) read the transaction log of your existing databases and *emit each row change as an event*, without touching the source application. This is the pragmatic entry point: you get an event stream out of a legacy system that has no concept of events, and you buy time to modernize gradually. This is how most enterprises actually start.

4. Breaking the monolith without a rewrite. When you need a new capability that would otherwise require modifying a fragile core system, you subscribe to its events instead of touching its code. Event-driven architecture is often the *least invasive* way to add value on top of systems you can't afford to replace.

The anti-patterns that will cost you

Say no to streaming when:

• The data is genuinely batch by nature. Monthly financial close, quarterly reporting, model retraining that runs weekly, forcing these into streaming adds cost and complexity for zero latency benefit. Not everything needs to be real-time; "real-time everywhere" is an architecture smell.
• You lack the operational maturity. Streaming platforms demand observability, on-call discipline, and idempotency rigor. Adopting Kafka before you have these produces a fragile system that fails silently and erodes trust in data, the one asset you cannot afford to lose.
• A single consumer will ever exist. The whole value is many independent consumers. One producer, one consumer, forever? A direct integration is simpler and you should use it.

The strategic framing to bring to your executive peers: event-driven architecture is an investment in *optionality*. You are paying a complexity premium now to make future use cases cheap to add. That's justified when you have a roadmap of real-time and AI ambitions. It's over-engineering when you don't.

Key Takeaways

1. Treat events as governed contracts, not messages. Extend your existing governance discipline into a schemaschemaA schema is the formal blueprint that defines how data is structured, named, typed, and related within a database, file, or message.View full definition → registry with enforced backward compatibility. A breaking schemaschemaA schema is the formal blueprint that defines how data is structured, named, typed, and related within a database, file, or message.View full definition → change is a breaking APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition → change, and you own the blast radius.

2. Buy decoupling, not just speed. The strategic prize is that teams publish and subscribe without cross-team negotiation. If your data platform is a queue of integration requests, event-driven architecture is how you dissolve the bottleneck as the org scales.

3. Choose the platform by its need for history. Default to Kafka when replay, backfill, and log-as-source-of-truth matter; default to cloud Pub/Sub when you want event-driven decoupling with near-zero operational burden. Reject résumé-driven Kafka adoption for notification-shaped problems.

4. Start with CDC, not a rewrite. Emit events from your existing databases via change data capture to get on-ramp value from legacy systems without touching their code, then modernize incrementally.

5. Mandate idempotency and observability as policy from day one. At-least-once delivery means every consumer must handle duplicates safely, and distributed debugging requires tracing before your first incident, not after it.