“Real-time” is one of the most requested and least examined features in data engineering. Stakeholders ask for it because it sounds better than the alternative, and engineers build it because the request is clear. The cost shows up later, in on-call burden and complicated code. This article offers a way to decide between batch and streaming by starting from the decision the data supports, not from the technology.

Define the Terms Without the Hype

Batch processing collects data over an interval and processes it together. Jobs run on a schedule, handle a bounded set of records and finish. The mental model is simple: input in, output out.

Stream processing handles records continuously as they arrive. Jobs run indefinitely, maintain state over time and produce results with low delay. The mental model includes ordering, late events and state that survives restarts.

There is also a spectrum between them. Micro-batch approaches process small intervals often, such as every few minutes, delivering fairly fresh data with batch-like simplicity. Many “real-time” requirements are well satisfied here.

Start With the Decision, Not the Data

The most useful question is: what will someone do differently if this data is fresher?

• If the answer is “look at a weekly report,” batch is plenty.
• If the answer is “a person reviews a dashboard during the day,” hourly or micro-batch likely suffices.
• If the answer is “an automated system must react within seconds, such as flagging a suspicious transaction,” streaming may be justified.

If nobody can name the action, the requirement is probably an assumption. Challenge it politely before building for it.

What Streaming Costs You

Streaming is powerful, but it changes the engineering problem.

State and ordering. Events can arrive late or out of order. You must decide how long to wait for stragglers and how to correct results when they appear.

Exactly-once expectations. Avoiding duplicates and losses end to end requires careful design of both processing and destinations. Delivery guarantees differ between systems, so read your tools’ current documentation rather than assuming.

Operations. Streaming jobs run continuously, so deployments, upgrades and failure recovery need planning. There is no “rerun tomorrow” fallback.

Cost. Always-on infrastructure can cost more than a periodic job, though this depends on workload and platform.

Testing. Reproducing time-dependent behavior in tests takes extra effort.

What Batch Costs You

Batch is not free of trade-offs either. Results are only as fresh as the last run. Large batch windows can create heavy peaks of load, and a failure late in a long job delays everything behind it. Late data requires reprocessing logic. Still, batch jobs are easier to reason about, rerun and backfill, and most teams have more experience operating them.

A Decision Table

Factor	Leans batch	Leans streaming
Freshness need	Minutes to days is fine	Seconds matter to an action
Consumer	People reading reports	Automated systems or alerts
Data volume pattern	Bulk, periodic	Continuous event flow
Team experience	Strong SQL and scheduling skills	Experience with event systems
Correctness needs	Easy to recompute and correct	Needs careful state handling
Operational appetite	Low, prefers simple reruns	Willing to run always-on services

Combining Both Sensibly

Many organizations use both. A streaming path serves a small number of latency-sensitive use cases, while batch handles the broader analytical load and acts as a source of corrected, authoritative numbers. When you run both, be explicit about which is the source of truth. Otherwise two systems produce slightly different totals and trust erodes.

One practical pattern is to reuse the same transformation logic where tooling allows, so the two paths do not drift apart. Another is to treat the fast path as provisional and reconcile it with the batch result later.

Questions to Ask Before You Commit

1. What is the cost of being late, in concrete terms?
2. Who will be on call for this pipeline?
3. How will we backfill or correct historical data?
4. What happens when the source sends duplicates or late events?
5. Can micro-batch meet the requirement with less risk?

A Realistic Example

Suppose a hypothetical retailer asks for “real-time inventory analytics.” On discussion, the merchandising team reviews stock positions twice a day, while the website needs to avoid selling items that are out of stock within moments of the last unit being sold. The team splits the requirement. The website-facing availability check uses an event-driven path, because it triggers an automated action. The analytics view is refreshed every fifteen minutes with a micro-batch job and reconciled nightly with a full batch run. The streaming surface area stays small, and the larger analytical workload stays easy to operate.

Frequently Asked Questions

Is streaming always more expensive than batch? Not always, but it commonly involves always-on resources and more operational effort. Compare for your own workload and platform.

What is micro-batch? Processing small batches at short intervals, such as every few minutes. It offers fairly fresh data with simpler operations than continuous streaming.

Can I convert a batch pipeline to streaming later? Often yes, but it is rarely a trivial change because state, ordering and delivery guarantees must be rethought. Designing clean, replayable inputs from the start makes either direction easier.

How do late events affect results? They can change values that were already reported. You need a policy for how long to wait and how to update outputs when stragglers arrive.

Do dashboards need streaming? Seldom. Most dashboards serve people who check them periodically, so frequent batch refreshes are usually enough.

Next Step

For each pipeline you own or plan, write one sentence stating the action taken on its output and how stale the data can be before that action suffers. Use those sentences, not the word “real-time,” as the requirement.