The most common cause of a broken integration is not a network failure or a bad deploy on your side. It is an application team adding, renaming or retyping a column in a database you replicate from, usually without telling anyone, because from their point of view nothing important changed. Downstream, a load fails at 3 a.m., or worse, succeeds and quietly puts nulls in a column the finance dashboard depends on.

You cannot stop sources from changing. You can decide in advance how each kind of change is handled.

Sort Changes by Risk

Not all schema changes are equal. Treat them in tiers.

Additive changes are a new column, or a new table. They are usually safe for consumers that select explicit columns, and they are the easiest to automate. The danger is consumers that select everything and then break on an unexpected field.

Widening changes are things like an integer becoming a larger integer, or a string gaining length. Often safe, but check the destination type can hold the new range.

Breaking changes are dropped columns, renamed columns, narrowed types, changed meaning, and changed units. A renamed column looks to a pipeline like one column vanishing and a new one appearing. Meaning changes are the nastiest: the column name and type stay the same, but “amount” now includes tax. No tool detects that automatically.

Writing these three tiers down and agreeing how each is handled is more valuable than any particular tool.

Decide the Default Behavior for Each Tier

For additive changes, many teams let the pipeline add the column to the raw layer automatically and alert, while the curated layer ignores the new column until someone chooses to use it. This keeps raw data complete without forcing immediate downstream work.

For breaking changes, the safer default is to stop loading that table and alert, rather than loading corrupt or partial data. Stopping feels disruptive, but a delayed table is a visible problem while a silently wrong one is not. Quarantining the affected records or batch lets other tables continue loading.

Your tooling may offer settings for schema drift handling, such as allow, ignore or fail. Check your integration tool’s current documentation for what it supports, and set the policy per table rather than globally, since a scratch table and a revenue table deserve different strictness.

Detect Changes Before They Hurt

Three practical detection points:

1. At ingestion: compare incoming column names and types against the last known schema, and log or alert on differences.
2. In tests on the raw layer: assert that expected columns exist, expected types hold, and critical columns are not suddenly all null.
3. At the boundary with the source team: subscribe to their change process. Even a shared channel where migrations are announced helps.

The null-rate check catches the meaning-and-rename cases that a pure schema comparison misses, because a renamed column typically shows up as an expected column that is now empty.

Build Layers That Absorb Change

Keep raw data close to the source and unmodified. Put a thin staging layer on top that selects explicit columns, renames them to your conventions and casts types. When a source column is renamed, you fix one staging definition and everything downstream continues working. Never let dashboards query raw tables directly, because then every source change is a dashboard change.

For high-stakes feeds, consider a data contract: a written agreement, ideally checked automatically, about which columns exist, their types and their meaning, plus a promise to notify before breaking changes. Contracts work best when the source team gets something back, such as a clear list of who depends on their data, rather than simply a new obligation.

Change Handling Matrix

Change	Typical risk	Reasonable default
New column	Low	Add to raw, alert, ignore in curated until reviewed
New table	Low	Alert, load only if explicitly approved
Type widened	Low to medium	Verify destination type, then allow
Type narrowed or changed	High	Stop that table, alert owner
Column renamed	High	Stop that table, update staging mapping
Column dropped	High	Stop, confirm whether consumers still need it
Meaning or unit changed	High and silent	Needs source-team notice and value-range checks

A Realistic Example

Imagine a replicated orders table where the application team renames cust_id to customer_id during a refactor. The ingestion step notices an unexpected column and a missing one, halts that table, and pages the data owner. Meanwhile other tables keep loading. The engineer updates the staging model to map the new name to the existing customer_key, reruns the halted load, and replays the missed batches in order. A message to the source team asks them to announce future migrations in the shared channel. Total impact: a late table for a few hours, instead of a week of wrong customer counts.

After the Fix, Close the Loop

Every schema incident teaches you something about your process. Add the check that would have caught it earlier, record the change in your runbook, and note who owns the relationship with that source. Over time the number of surprises falls, not because sources stop changing, but because changes become routine events with a known procedure.

Frequently Asked Questions

Should the pipeline automatically adapt to every schema change? No. Automatic handling is fine for additive changes in raw layers. Breaking changes should stop and alert, because adapting blindly can load wrong data.

How do I handle a column whose meaning changed? You need human communication plus value checks, such as expected ranges or distributions. Schema comparison alone cannot see it.

Is CDC more or less vulnerable to schema changes? Both are affected. Change-data-capture streams carry the schema from the source log, which can propagate changes quickly, so test how your tool behaves with each type of change before relying on it.

How often should we review source schemas? Continuously through automated checks, and periodically through a conversation with the source teams for your most important feeds.

Do data contracts replace tests? No. Contracts state the agreement, and tests verify it holds. You need both.

Next Step

List your five most important source tables and mark for each whether a column rename would stop the load, pass silently, or something else. The ones you cannot answer are where to add a check first.