The Day an Upstream Rename Broke My Pipeline Without a Single Error
The views I express here are mine alone and do not necessarily reflect the opinions or official position of my employer.
A source system renamed a column. Nobody told us. The pipeline didn't error, it just stopped finding the column it expected, filled the downstream field with NULLs, and kept running on schedule every day for a week before anyone noticed a metric had quietly gone to zero. That's the version of failure that scares me most: not a crash, a wrong number sitting in a dashboard looking perfectly normal.
Schema drift versus schema evolution
Schema drift is what happens when a source schema changes underneath you and nobody planned for it on your side: a column added, removed, renamed, retyped, or reordered without a heads-up. Schema evolution (sometimes loosely called schema shift, though that's not a formal or standardized term) is the same shape of change, but intentional and coordinated, someone tells you a column is being renamed next sprint, and you update your models to match before it happens.
What actually matters is whether you found out about the change before your pipeline did.
Catching drift before production does
The goal is to fail loud and early at the boundary, not downstream where a wrong number looks exactly like a right one. A few layers that actually catch this:
- dbt source freshness and contracts. A
not_nulloraccepted_valuestest on a source column catches a value going unexpectedly missing. A model contract enforces the column types you expect and fails the build the moment a source stops matching them, catching a type mismatch that would otherwise flow through undetected. - Column-level checks on ingestion. Compare the incoming column list against what you expected before you load anything. A new or missing column should stop the run, not get absorbed without a trace.
INFER_SCHEMAon staged files. Snowflake can inspect a staged file and report its actual column names and types before you commit to a table definition. Running this as a pre-load check surfaces a renamed or retyped column immediately, before it has a chance to land anywhere downstream.- Alert on unexpected new columns too, not only ones that vanish. A drift check that only watches for columns disappearing will miss a column appearing under a new name while the old one is still expected somewhere downstream.
Handling it once you know it's coming
A permissive, schema-on-read landing layer absorbs whatever the source sends without complaint. Everything downstream of that should be strict and typed, so if something genuinely changed shape, the failure happens at a model boundary you control, with a clear error, long before it reaches a report three joins downstream.
When a change is real and needs to land in your models, prefer additive over destructive. Add the new column, backfill it, and only retire the old one once everything downstream has actually migrated over. Renaming or dropping in place and hoping nothing depends on the old name skips that window entirely. The additive path is slower on purpose: it gives you a stretch of time where both the old and new shape work.
Why Iceberg makes some of this cheaper
Apache Iceberg tracks columns by a stable field ID instead of by name or position. That means a rename, an add, a drop, or a reorder is a metadata operation, not a data rewrite: the table's history and existing data files don't need to change just because a column got a new name. A rename upstream that would otherwise mean rewriting an entire table becomes a schema update that finishes in seconds. It doesn't eliminate the need to detect the drift in the first place, it just makes handling a legitimate schema evolution far less expensive once you know about it.
The gotcha that's easy to miss: a datatype change that still "works"
The most dangerous drift is a column that's still there, still populated, and just retyped. An integer ID that becomes a string ID will usually still load. Ingestion doesn't complain. The failure shows up two steps later, in a join that no longer matches because 123 and '123' aren't equal in every comparison context, or in an aggregate that coerces the value or errors outright depending on the engine. This is exactly why type contracts matter more than existence checks alone: a column being present tells you nothing about whether it still means what it used to mean.
| Drift type | How it breaks | How to handle |
|---|---|---|
| Column renamed | Old name missing, field reads as NULL downstream | Contract/schema test on expected columns, alert on unexpected new names |
| Column removed | Same as a rename, but permanently | Source freshness checks, fail the load instead of substituting a default |
| Column added | Usually harmless unless something downstream selects * | Explicit column lists in models, not SELECT * |
| Datatype changed | Loads fine, breaks joins or aggregates later | Model-level type contracts |
| Column reordered | Harmless if referenced by name, breaks if referenced by position | Reference columns by name so a reorder cannot remap them |
The Boundary Is Where I Check for Drift Now
Every one of these failures traces back to the same root cause: treating a source schema as something the pipeline can trust indefinitely instead of a contract that needs to be checked every time data crosses the boundary into your system. A schema test at the edge is cheap. A wrong number sitting undetected in a stakeholder's dashboard for a week is not, and explaining why nobody caught it sooner is the hard part that follows.