By John Apostolo, HEXstream full-stack data analyst

Consider a simple case. A row changes in some source system—a status flips from 2 to 3, a balance goes from 1000 to 1200. Something carries that change downstream. But what actually shows up on the other end?

There are two basic ways data moves. CDC (Change Data Capture) captures row-level changes—inserts, updates, deletes—and ships them downstream continuously or in near-real-time. Batch moves snapshots or time-bounded chunks on a schedule, producing a new output per run.

On the surface, that's the whole decision—stream it or batch it. But there's a second choice that rarely gets named: what shape the truth takes when it lands.

One useful way to think about it: the truth tends to arrive in one of three shapes. The raw ledger—every transition, uninterpreted, the auditor's dream. The domain narrative—OrderShipped, RateChanged, business meaning baked in. The snapshot—the whole table as of midnight.

Boring, simple, complete.

The shape decision rarely gets made explicitly. It arrives via a connector's default payload, a replication tool's reference model, a prototype that shipped before the design review. But defaults couple: the raw ledger forces every consumer to rebuild ordering and idempotency. The domain narrative bakes interpretation into the transport layer, making schema evolution a cross-team negotiation. The snapshot pushes late-arrival pain into the next batch window.

Each hop re-interprets the shape—ledger → narrative → snapshot—and every conversion is a seam where meaning leaks. The report renders. The values look plausible. The timezone context didn't survive the hop. No test catches it.

If you're shipping changes (CDC). The first question is where you capture: log-based CDC usually preserves the most detail with the least impact on the source, while triggers or application events can be simpler organizationally but risk gaps and performance overhead. The second question is the shape question.

Raw row deltas maximize auditability and replay—but they push deduping, ordering, and idempotent upserts onto every consumer. Deletes deserve special respect: a snapshot reflects them for free, but CDC has to propagate them explicitly, and a missed delete can sit invisible until someone queries a row that shouldn't exist. And because failures are inevitable, it pays to treat replay as normal: stable keys, clear sequencing, and a defined bootstrap/backfill path aren't nice-to-haves—they're often the difference between a pipeline you trust and one you babysit.

If you're shipping snapshots (batch). Batch optimizes for throughput and repeatability, and the core trade is freshness versus simplicity and cost. Full refreshes tend to be the most reliable but expensive; incremental batch is cheaper but buys you watermark logic, late-arriving data handling, and partition repair policies. Where batch shines: heavy joins, large aggregations, and reproducible "as-of" reporting—anywhere deterministic reruns matter more than minute-by-minute updates.

The shape follows you into storage. A store fed by CDC has to be comfortable with constant upserts and deletes—fewer indexes, partitioning for write locality, a deliberate merge-on-read versus merge-on-write choice (query-time cost vs. ingest-time cost), and compaction policies so tombstones don't quietly rot performance. A store fed by batch rewards the opposite: append-heavy loads, columnar storage, compression, partition pruning, sort keys for the common queries. Pick a shape and you've probably half-picked a database, too.

And here's a part that's easy to miss: the shape isn't picked once. It gets re-picked at every hop. A ledger lands in staging, gets interpreted into a narrative, gets flattened into a snapshot for reporting—and every conversion is a chance for meaning to leak. A report needs a field the model never mapped, so somewhere in the semantic layer it gets pointed at the closest lookalike column. The filter populates, the values look plausible, and nobody's the wiser. Or a timestamp crosses a hop and its timezone doesn't come with it—the value survives, the context doesn't. Nothing errors out.

The report renders fine. It's just wrong, in a way no schema check is likely to catch.

So the real failure usually isn't choosing badly—it's not noticing there was a choice.

A few things that seem to help: write down, for each hop, what shape arrives and what it guarantees—grain, completeness, replay semantics. Pick shapes based on what the consumer needs, not on what the tool emits. And put checks at the seams between layers, not just inside them, because the seams are where contracts tend to break.

Your pipeline is already making these promises. Somebody should probably read them.

CLICK HERE TO CONNECT WITH US ABOUT YOUR PIPELINE.