Real-time data is not a luxury feature. For a growing class of use cases, it's a business requirement.
Fraud detection that runs on yesterday's transactions is useless. Personalization based on last week's behavior misses the moment. Operational dashboards that update hourly can't drive real-time decisions. The business value of data degrades with time, and for some use cases, it degrades very fast.
Streaming data is a continuous, unbounded sequence of events. Unlike batch data (a bounded dataset processed at scheduled intervals), streaming data flows continuously and must be processed with low latency.
A real-world example: every click, page view, add-to-cart, and purchase on an e-commerce site is a streaming event. These events are produced by user actions in milliseconds. Processing them in real-time enables: real-time inventory updates, instant fraud scoring, live personalization, and operational dashboards.
Apache Kafka in 6 Minutes
Knowledge check
1. What fundamentally distinguishes streaming data from batch data?
2. Why is real-time processing described as a business requirement rather than a luxury for certain use cases?
3. How does Kafka's retention model differ from a traditional message queue?
4. Select ALL correct statements about Kafka's core architectural components.
Select all the correct answers.
5. Select ALL scenarios that genuinely benefit from real-time streaming rather than periodic batch processing.
Select all the correct answers.
Kafka is the de facto standard for streaming data infrastructure. Understanding its architecture:
Topics: Logical channels. Events are published to topics (e.g., "checkout-events", "user-clicks"). Topics can have many producers and many consumers.
Producers: Applications that publish events to topics. Your checkout service publishes a "purchase-completed" event whenever an order is placed.
Consumers: Applications that subscribe to topics and process events. Your fraud detection service consumes "purchase-completed" events and scores each transaction.
Consumer groups: Multiple consumers can form a group and divide work between them. If you have a topic with 10 partitions and 10 consumer instances, each instance handles one partition, horizontal scalability.
Retention: Kafka stores events for a configurable period (default: 7 days). Late consumers can replay history. This is different from a message queue where consumed messages are deleted.
LinkedIn, where Kafka was invented, processes over a trillion events per day through it. Uber, Netflix, and Airbnb each handle hundreds of billions of events daily through Kafka clusters.
Consuming events from Kafka is one thing. Enriching, joining, aggregating, and acting on them in real-time requires stream processing.
Apache Flink is the dominant stream processing engine. It processes events with sub-second latency, supports stateful computations (maintaining running totals, session windows), and handles late-arriving data gracefully.
Apache Spark Structured Streaming is the Spark-native approach, familiar to teams already using Spark for batch. Supports micro-batch (near-real-time) natively, true streaming with higher complexity.
Use case example: A streaming pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → for real-time fraud detection:
1. Kafka consumes purchase events from checkout service
2. Flink enriches each event with customer history (looked up from a feature storefeature storeA centralised repository managing ML features, ensuring consistency between training and serving environments.View full definition →)
3. Flink applies the fraud scoring model (sub-100ms)
4. If score > threshold, event is published to "flagged-transactions" topic
5. Downstream service triggers human review or card block
The entire flow completes in under 500ms from purchase event to fraud flag.
Streaming infrastructure is significantly more complex to build and operate than batch. Before committing:
Real-time is worth it when:
Batch is sufficient when:
The streaming tax, additional engineering complexity, operational overhead, debugging difficulty, is real. Pay it only when the business value justifies it.
1. Quelle est la principale différence entre le streaming et le batch processing ?
A) Le streaming est toujours plus coûteux
B) Le streaming traite des événements continus avec une faible latence, le batch traite des datasets bornés à intervalles planifiés
C) Le batch ne peut pas gérer de grandes quantités de données
D) Le streaming nécessite obligatoirement Apache Kafka
Réponse: B
2. Dans l'architecture Kafka, qu'est-ce qu'un "topic" ?
A) Un type de message d'erreur
B) Un canal logique auquel les producteurs publient des événements et que les consommateurs lisent
C) Une base de données de streaming
D) Un outil de monitoring
Réponse: B
3. Pour quel cas d'usage le streaming en temps réel est-il absolument nécessaire par rapport au batch ?
A) Rapports hebdomadaires de revenus
B) Modèles de prévision mensuels
C) Détection de fraude en temps réel lors des transactions
D) Consolidation des données comptables
Réponse: C
These actions are compiled in the role's Playbook.