What Building Data Pipelines Taught Me About Reliability

The failures that taught me the most about building data pipelines were never the ones that crashed. A crash is almost a gift — it’s loud, it points roughly at its own cause, and it stops the bad output from going anywhere. The failures that actually cost time were the ones where the pipeline ran to completion, produced output that looked completely reasonable, and was wrong in some way nobody noticed until it had already propagated into a dashboard, a report, or a downstream model that quietly got a little worse.

That asymmetry — loud failures are cheap, silent failures are expensive — has ended up shaping most of what I now consider good practice in this area, more than any specific tool or framework has.

Idempotency is not a nice-to-have

The first lesson, and the one I resisted longest because it sounded like premature engineering, is that pipeline steps should be safe to rerun. Not “usually fine to rerun.” Safe, as in: running the same step twice with the same input produces the same result as running it once, with no duplicated rows, no double-counted aggregates, no silently corrupted state.

The reason this matters more than it seems to at design time is that reruns are not an edge case in a data pipeline’s life — they’re closer to a routine event. A job times out partway through. An upstream source gets updated after you already consumed it. Someone needs to backfill three weeks of history because a bug was fixed. Every one of these situations turns into a rerun, and if the pipeline wasn’t built to tolerate that from the start, the fix is never “just rerun it.” It’s “carefully figure out what state we’re actually in, clean up whatever the partial or duplicate run left behind, and then rerun it,” which is a much more expensive and much more error-prone operation, usually performed under time pressure.

Idempotency is cheap to build in from the start and expensive to retrofit, which is a common enough pattern in engineering that it should probably be trusted more than it usually is.

Validate at the boundary, not in the middle

The second lesson is about where to put your defenses. My early instinct was to scatter sanity checks throughout a pipeline — a little validation here, an assertion there, wherever something felt fragile at the time I was writing it. This produces an inconsistent safety net with gaps you can’t predict, because the coverage reflects what felt risky while you were writing the code, not what’s actually risky in the data.

A more effective pattern is validating aggressively at the boundaries — when data enters the system, and again when it’s about to leave a stage and be handed to something else that will trust it. Boundaries are where assumptions get made explicit: this column should never be null, this value should fall within this range, this count shouldn’t drop by more than some threshold from the last run without triggering a closer look. Concentrating validation there, rather than diffusing it throughout the internal logic, means you have a much clearer picture of what your pipeline is actually promising to whoever consumes its output — and a much easier time finding where a violated assumption first entered the system.

def validate_boundary(df, *, required_columns, max_null_fraction=0.01):
    missing = set(required_columns) - set(df.columns)
    if missing:
        raise ValueError(f"missing required columns: {missing}")

    null_fractions = df[list(required_columns)].isnull().mean()
    violations = null_fractions[null_fractions > max_null_fraction]
    if not violations.empty:
        raise ValueError(f"null fraction exceeded threshold: {violations.to_dict()}")

Nothing sophisticated is happening there. That’s the point — the validation logic doesn’t need to be clever, it needs to be consistently applied at every place data crosses a trust boundary, which turns out to be the harder discipline to maintain.

Silent failures come from optimistic defaults

A pattern I’ve seen repeatedly, and been guilty of myself, is a transformation step that has a reasonable-looking default for missing or malformed data — fill nulls with zero, drop rows that don’t parse, coerce an unexpected type — implemented because it lets the pipeline keep running instead of stopping on every minor data quality issue. This is defensible in the moment. It’s also exactly the mechanism by which a pipeline produces confidently wrong output instead of failing.

The problem isn’t defaults themselves; sometimes a sensible default really is the right call. The problem is defaults applied silently, without any record that they were used. A dropped row that isn’t logged, a null filled with zero that isn’t flagged, is a small lie the pipeline tells about the completeness of its own output, and the lie compounds every time the pipeline runs, because there’s no signal anywhere that would prompt anyone to go check.

The fix isn’t to remove defaults — an overly strict pipeline that halts on every minor anomaly creates its own operational burden. It’s to make every default’s usage visible: count how often it fired, log which rows it applied to, and treat a rising rate of “we had to use the fallback here” as a signal worth investigating, rather than a detail buried in a debug log nobody reads.

Monitoring the output, not just the process

The last lesson took the longest to actually adopt, because it required admitting that “the job succeeded” and “the job produced correct output” are different claims, and most monitoring only checks the first one. A pipeline can exit with status zero, log no errors, finish within its expected runtime, and still be quietly wrong, because none of those signals say anything about whether the data itself makes sense.

What eventually helped was treating the output data as something to monitor in its own right, separately from the process that produced it — tracking distributions, row counts, and key aggregates over time, and alerting on movement that’s large relative to normal variation, independent of whether the job technically “succeeded.” A pipeline that runs cleanly but produces a row count forty percent lower than every previous run is not a success, even though every process-level signal says it is.

The common thread

Looking back at all of this, the actual lesson isn’t really about pipelines specifically — it’s that reliability work is disproportionately about handling the boring, likely failure modes, not the dramatic unlikely ones. Nobody gets burned by a once-in-a-decade catastrophic failure nearly as often as they get burned by “the job silently used a fallback ten thousand times last month and nobody noticed,” which is a far less interesting story and a far more common one. Most of what I’d now call good pipeline design is just taking that boring category of failure seriously enough to build against it deliberately, instead of leaving it to be discovered downstream, by someone else, much later than it should have been.