Sooner or later you find a bug that has been quietly corrupting a column for six weeks. Or the business asks for a new metric computed over the last two years. Or you onboard a new source with a long history. All three end the same way: you need to rerun a pipeline across historical data. That is a backfill, and done carelessly it can duplicate rows, melt a source system, or leave dashboards half-updated during working hours.

A backfill is not special code. It is your normal pipeline run over a different time range, which is exactly why the design of the normal pipeline determines how painful it is.

What Makes a Pipeline Backfillable

Three properties matter, and you want them before you need the backfill.

Parameterized by time range. The job takes a start and end, or a partition date, as input, rather than calling “now” internally. A job that hardcodes “yesterday” cannot be pointed at March without editing code.

Idempotent. Running the same range twice produces the same result as running it once. The usual technique is to overwrite whole partitions, or to merge on a primary key, instead of blindly appending. If your job appends, every rerun duplicates data.

Deterministic. The same inputs give the same outputs. Logic that depends on the current time, on external lookups that have since changed, or on random ordering will give a backfill different answers than the original run. Sometimes that difference is the point, but you should know it is happening.

If your pipeline lacks these, fixing them is the first task, and the backfill becomes a good test of the fix.

Plan Before You Press Run

Answer these questions in writing:

• What exactly is the date range, and what tables are affected?
• Does the source still hold the old data? Some systems retain only recent history, and APIs may not return old versions of records.
• What downstream tables, dashboards or exports consume the output, and what happens to them mid-run?
• Does the transformation logic need to match old behavior, or the corrected current logic?
• How much load will this add to the source and to the warehouse, and when is a quiet time?
• How will you verify the result?

The source retention question surprises people most often. If the raw layer holds the original files, you are in good shape. If not, the backfill may be partial, and that needs to be known up front.

Run It in Small, Observable Pieces

Do not launch two years in one job. Split by partition, such as one day or one week at a time, so a failure costs a small retry rather than the whole run. Start with one partition and compare the output with expectations. Then run a month. Then the rest.

Control concurrency. Backfills tend to run many partitions in parallel, which can overwhelm a source database or exhaust warehouse capacity needed by production reporting. Most orchestrators let you cap concurrent runs; set a conservative cap and raise it if things look healthy.

Isolate the Output Until It Is Verified

The safest pattern is to write backfill results to a separate location or shadow table, validate there, and then swap or copy partitions into the live table. That way consumers never see a half-corrected mix. If your platform supports atomic partition replacement or table swaps, use it. If not, schedule the swap for a quiet window and communicate it.

Be careful of downstream cascades. When a base table changes, tables built on it are now stale. Make a list of dependent models and rerun them in dependency order afterward, otherwise numbers in the base layer will disagree with those in the summaries.

Verify Like You Mean It

Check at least:

• Row counts per partition against the source or the previous version.
• Key aggregates, such as total amount per day, against an independent number.
• Primary key uniqueness, to catch duplicates.
• A sample of individual records compared by eye.
• The specific field that motivated the fix, to confirm it is actually corrected.

Record the before and after figures. If people ask later why last quarter’s revenue changed in the dashboard, you can show them.

Backfill Runbook Outline

Step	Question to answer	Evidence to keep
Scope	Which tables and dates?	Written range and table list
Source check	Is the original data still available?	Retention confirmation
Dry run	Does one partition match expectations?	Comparison of counts and sums
Throttle	What concurrency is safe?	Chosen limit, source load notes
Isolate	Where do results land before release?	Shadow table or location
Verify	Do counts, keys and aggregates reconcile?	Before/after report
Release	How and when are partitions swapped?	Time of swap, owner
Cascade	Which dependent tables are rerun?	Ordered list, completion status
Communicate	Who is told about changed history?	Message to data consumers

A Realistic Example

Imagine a currency conversion bug that applied the wrong rate for six weeks. The pipeline is partitioned by date and overwrites each partition, so the fix is a code change plus a rerun for those forty-two days. The team runs one day into a shadow table and finds the converted totals match the finance team’s spreadsheet. They run the rest with a concurrency cap of three, verify row counts per day, swap the partitions during a quiet evening, then rerun the dependent monthly summary. A short note goes to finance explaining that figures for those weeks changed and why.

Tell People History Changed

Technically a backfill is finished when the data is right. Organizationally it is finished when people know. Someone has probably exported or screenshotted the old numbers. A brief message stating what changed, which dates, and the size of the correction preserves trust and prevents a week of confused questions.

Frequently Asked Questions

Can I backfill a streaming pipeline? Often yes, by replaying retained events or running the equivalent batch logic over stored raw data. It depends on how long your event log retains history, so check your platform’s retention.

Should a backfill use the old logic or the new? Usually the corrected logic, since fixing history is the goal. If you need the original behavior for audit, keep the old output archived before replacing it.

How do I prevent a backfill from blocking the daily run? Use separate resource pools or queues where available, limit concurrency, and avoid scheduling the heaviest partitions during the daily load window.

What if the job is not idempotent? Fix that first. Otherwise delete the target partitions explicitly before each rerun, understanding this is riskier because a failure can leave a gap.

How far back should I go? As far as the bug or the requirement reaches, not further. Smaller scope means less risk and less cost.

Next Step

Take your most important pipeline and ask: if I had to rerun last Tuesday only, could I do it with one command and no duplicates? If not, make that the next reliability improvement.