Batch vs streaming: choosing the right paradigm

The $12 million refresh nobody watched

In 2019, a large North American retailer rebuilt its inventory analytics on a full streaming stack, Kafka, Flink, the works, so that store managers could see stock positions "in real time." Eighteen months and roughly $12 million later, an internal audit found the ugly truth: the median store manager checked inventory dashboards once per shift, and reorder decisions were locked to a nightly replenishment cycle that fired at 2 a.m. regardless. The business had bought sub-second latency for a decision that moved once a day. The streaming pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète → was a Ferrari idling in a parking lot.

This is the trap. Streaming has become a status symbol, proof that a data org is "modern." But latency is a cost, not a virtue. Your job as a CDO is not to minimize latency; it's to match latency to the decision that consumes it. Overbuild and you burn budget and headcount on operational complexity nobody asked for. Underbuild and you strangle a use case, fraud, dynamic pricingdynamic pricingAutomatically adjusting prices in real time based on demand, competition or user behaviour to optimise revenue, margin or conversion.Voir la définition complète →, fleet routing, that genuinely dies without freshness.

This lesson gives you the decision framework and the operating discipline to get that match right.

The decision latency framework

Stop asking "should this be real-time?" It's the wrong question because it's answered by engineering enthusiasm. Ask instead: what is the decision latency of the consuming process?

Decision latency is the time between when an event occurs and when a human or system can actually *act* on it in a way that changes an outcome. It has three components you must interrogate separately:

• Data freshness, how quickly the data can arrive.
• Decision cadence, how often the consuming process actually makes a choice.
• Action latency, how long it takes to execute once decided.

The binding constraint is the *slowest* of the three. The retailer above had sub-minute freshness, a once-per-day decision cadence, and an action latency (physical replenishment) measured in hours. Freshness was never the bottleneck. Spending to improve it was economically illiterate.

Here's the rule that follows: freshness is only worth buying up to the point where it stops being the binding constraint. Beyond that, every dollar of latency reduction produces zero decision improvement.

The value-decay curve

For any use case, plot how the value of a piece of data decays with age. The shape tells you the paradigm.

• Cliff decay, value drops to near zero within seconds to minutes. Fraud authorization, ad bidding, algorithmic trading, fleet dispatch, anomaly-triggered safety shutoffs. Here, streaming isn't a luxury; late data is *worthless* data. A fraud score delivered 400ms after the transaction cleared is a report, not a control.
• Linear decay, value erodes steadily over hours. Operational monitoring, same-day supply chain, customer service context. Micro-batch (1-15 minute intervals) usually wins: most of the value, a fraction of the complexity.
• Flat-then-drop, value is stable for a day or more, then matters at a fixed boundary. Financial close, regulatory reporting, weekly cohort analysiscohort analysisCohort analysis groups users by a shared starting trait or time (such as signup month) and tracks their behavior over time to reveal retention and lifecycle patterns.Voir la définition complète →, model retraining. Nightly batch is not a compromise here; it's the *correct* engineering.

The mistake CDOs make is treating the value-decay curve as a property of the *data* ("transactions are important, so they must be real-time"). It's a property of the *decision consuming the data*. The same transaction record feeds a cliff-decay fraud check *and* a flat-decay quarterly revenue report from the same source. One demands streaming; the other must not use it.

The total cost of latency

The Ferrari isn't just expensive to buy. It's expensive to *own*, and that ownership cost is where most CDO business cases quietly lie.

Batch systems have a forgiving failure mode: a job dies at 2 a.m., you get paged, you rerun it, and by 6 a.m. the world looks the same. Idempotent reruns, clear checkpoints, easy backfills. Streaming systems have no such mercy. They fail continuously and in flight, which introduces cost categories that never appear in the initial architecture slide:

• Exactly-once semantics. Guaranteeing each event is processed once, not zero times, not twice, under partial failure is genuinely hard. Getting it wrong means double-charged customers or double-counted metrics.
• Out-of-order and late-arriving events. Real event streams don't arrive in order. You need watermarking and windowing logic to decide how long to wait for stragglers before you close a window, a tradeoff between completeness and latency that has no free answer.
• Backpressure and replay. When a downstream consumer slows, the whole pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète → must degrade gracefully or you lose data. Recovering from a bug means replaying from an offset, which requires you to have retained the raw stream.
• On-call reality. Streaming is a 24/7 operational commitment. A nightly batch has a maintenance window; a stream never sleeps. Budget for the on-call rotation and the burnout, not just the cluster.

A useful heuristic from teams who've run both: a production streaming pipeline costs roughly 3-5x the total operating effort of an equivalent batch pipeline once you include on-call, testing complexity, and the specialized talent premium. That multiplier is your hurdle rate. The streaming use case must generate at least 3-5x the *decision value* of the batch alternative to break even, before you've earned a dollar of upside.

This reframes the business case. The question is not "can we afford to build streaming?" It's "does the marginal decision value from fresher data clear a 4x cost multiple?" For fraud, dynamic pricingdynamic pricingAutomatically adjusting prices in real time based on demand, competition or user behaviour to optimise revenue, margin or conversion.Voir la définition complète → at scale, or real-time personalization on a high-traffic property, easily yes. For an internal ops dashboard glanced at twice a day, almost never.

The micro-batch middle

Most CDOs frame this as a binary. It isn't. Micro-batch, running batch jobs on tight intervals (every 1, 5, or 15 minutes), captures the majority of streaming's freshness benefit while retaining batch's operational sanity: reruns are still easy, semantics are simpler, and you keep a normal maintenance posture.

The architectural signal is straightforward:

# Micro-batch: still batch semantics, near-real-time cadence.
# You get idempotent reruns and simple checkpointing.
spark.readStream \
    .format("delta") \
    .load("/events/transactions") \
    .writeStream \
    .trigger(processingTime="5 minutes") \
    .foreachBatch(upsert_to_warehouse) \
    .start()

Change processingTime to "1 minute" and you're tighter; set it to availableNow=True and you've collapsed back to scheduled batch, *same code*. This is the point: micro-batch lets you tune the freshness dial without committing to the full operational weight of continuous streaming. A large fraction of use cases labeled "real-time" in requirements docs are satisfied completely by a 5-minute micro-batch. ReachReachThe number of unique people exposed to your message in a given period. Unlike impressions, reach counts each person once, no matter how often they see it.Voir la définition complète → for true event-at-a-time streaming only when the value-decay curve is a genuine cliff *and* your action latency can actually exploit sub-minute freshness.

Running the decision on monday morning

Here is the sequence to walk when a stakeholder demands "real-time."

1. Force the decision-latency conversation. Ask: "When this data is fresher, what specific decision changes, who makes it, and how often?" If they can't name the decision, the requirement is aspirational, not real. Nine times out of ten, "real-time" means "I'm tired of stale dashboards", a data-quality or refresh-frequency problem, not a paradigm problem.

2. Find the binding constraint. MapMapUsing software to automate repetitive marketing tasks and campaigns, enabling personalisation at scale across channels like email, web, and social.Voir la définition complète → the three latency components. If action latency (a human approval, a physical process, a downstream nightly job) dominates, freshness upstream is wasted. The retailer's replenishment lock made this obvious in hindsight; make it obvious *in advance*.

3. Price the decision value. Estimate the incremental value of acting N minutes sooner. Fraud has a clean number: dollars of prevented loss per minute of faster detection. Dynamic pricingDynamic pricingAutomatically adjusting prices in real time based on demand, competition or user behaviour to optimise revenue, margin or conversion.Voir la définition complète →: revenue lift per pricing cycle. If the number is fuzzy or small, that's your answer.

4. Apply the multiplier. Does the decision value clear the 3-5x operating-cost hurdle over the micro-batch alternative? If not, ship micro-batch and move on.

5. Default to the simpler paradigm. When genuinely uncertain, choose the lower-complexity option. You can always tighten a micro-batch interval or promote a pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète → to streaming later. Unwinding an over-engineered streaming stack that three teams now depend on is a political and technical nightmare.

The portfolio view

You are not choosing *one* paradigm for the organization. You're managing a portfolio, and the winning architecture usually runs both against the same source data, the pattern you know as the two-track design where a fast serving layer handles cliff-decay decisions and a batch layer produces the authoritative, reconciled record.

The CDO discipline is placement: deciding which use case sits on which track, and refusing to let engineering enthusiasm or executive fashion pull flat-decay workloads onto expensive infrastructure. Maintain a simple register, for each major data productdata productA data asset managed like a product, with an owner, defined users, guaranteed quality, and measurable business value.Voir la définition complète →, record its decay shape, binding constraint, chosen paradigm, and the decision value justifying it. Review it when someone requests an upgrade. This turns "should this be streaming?" from a recurring emotional argument into a governed, evidence-based call.

One more trap to name: freshness theater. Executives love a dashboard tile that ticks every second. It *feels* modern. It drives streaming investment untethered from any decision. When you see a real-time visualization whose underlying decision moves daily, you're looking at cost dressed as capability. Kill it, or downgrade it to micro-batch, and redeploy the savings to a use case that lives on the cliff.

When batch is the braver choice

There's a cultural dimension senior leaders underestimate. In many data orgs, choosing batch is read as choosing *behind*. Ambitious engineers want streaming on their résumés; vendors sell it as table stakes; the board read a McKinsey piece about real-time enterprises. The pressure runs one direction.

Defending a batch decision, or downgrading a streaming aspiration to micro-batch, requires more organizational courage than approving the shiny thing. But it's frequently the higher-judgment call. The retailer's $12M lesson wasn't a technology failure; every component worked as designed. It was a placement failure and a courage failure: nobody senior enough asked whether the decision the pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète → fed actually moved fast enough to justify it.

Your credibility as a CDO is built partly on saying no to sophisticated things for unsophisticated reasons. When you decline to stream a workload, document the decision-latency logic and the cost multiplier. That paper trail converts a "conservative" choice into a defensible, quantified one, and protects you when the fashion cycle turns and someone asks why the company isn't "real-time everywhere."

The inverse courage matters too. When a use case genuinely lives on the cliff, fraud, safety, real-time bidding, and someone tries to save money by forcing it onto nightly batch, you must fight the other direction just as hard. Starving a cliff-decay use case of freshness isn't frugality; it's shipping a control system that arrives after the event it was meant to prevent. That's worse than not building it, because it creates false confidence.

Key Takeaways

• Match latency to decision latency, not to ambition. Buy freshness only until it stops being the binding constraint among data freshness, decision cadence, and action latency. Beyond that point, every latency dollar returns zero.

• Classify by value-decay shape, per decision, not per dataset. Cliff decay → streaming. Linear decay → micro-batch. Flat-then-drop → batch. The same source feeds different decisions on different tracks; treat placement as your core CDO discipline.

• Apply the 3-5x cost multiplier as a hurdle rate. Streaming's true cost, exactly-once semantics, out-of-order handling, replay, 24/7 on-call, runs several times a batch equivalent. The use case must clear that multiple in incremental decision value before you commit.

• Default to micro-batch when uncertain. A 5-minute interval satisfies most "real-time" requirements with batch-grade operational sanity, and the same code tunes from scheduled batch to near-real-time. Promote to true streaming only for genuine cliffs.

• Kill freshness theater, and have the courage to defend batch. A real-time dashboard feeding a daily decision is cost dressed as capability. Document the decision-latency logic behind every paradigm choice so a conservative call becomes a quantified, defensible one.