There are two ways monitoring fails. The first is silence: a pipeline breaks and nobody finds out until an executive asks why the dashboard is empty. The second is noise: the team receives so many alerts that they mute the channel, and the one that mattered gets buried among forty that did not. Most teams swing from the first problem to the second after a bad incident.

Good pipeline monitoring is a small number of signals tied to things people actually care about, routed to someone who can act.

Job Success Is Not Data Health

The basic check is “did the job finish?” It is necessary, and it is not enough. A job can finish green and still deliver an empty table because the source returned nothing, or a table full of duplicates because a join fanned out. Data failures often arrive as successful runs.

So monitor on two levels:

• Operational health: did the run start, finish, and finish on time?
• Data health: is the output fresh, complete and plausible?

Teams that only do the first level learn about bad data from their users.

The Signals Worth Watching

Freshness. How long since this table was last updated? Alert when it exceeds the time consumers expect. This single check catches a large share of failures, including upstream delays and jobs that never started, which job-failure alerts cannot see because there was no job to fail.

Volume. Row counts per run compared with recent history. A table that normally gains around the same amount each day and suddenly gains zero, or ten times as much, deserves a look. Use ranges based on your own history, not guessed figures.

Schema. Unexpected columns or type changes, particularly at the raw boundary.

Completeness and validity. Null rates in key columns, uniqueness of primary keys, values within expected sets or ranges.

Duration. A run taking much longer than usual is often an early warning of growing data, a bad plan or resource contention, long before it fails.

Cost and resource use. Unusual spikes in compute or scanned data can indicate a runaway query or a broken filter.

Decide What Deserves a Page

Not every signal should wake someone up. A useful split:

• Page now: a critical table is stale beyond its agreed freshness, or a pipeline feeding customer-facing or financial output has failed.
• Notify during working hours: non-critical pipeline failures, volume anomalies on internal tables, slow runs.
• Log only: informational events and small deviations, kept for trend analysis.

To decide, tie each pipeline to its consumers. Ask: who notices if this is wrong, how soon, and what do they do about it? A table that powers a weekly report does not need a 2 a.m. page.

Fight Alert Fatigue Directly

Treat noisy alerts as defects. Every alert should meet three tests: it is actionable, it has a clear owner, and the person receiving it knows what to do. If an alert is routinely ignored, either fix the cause, change the threshold, or delete it.

Practical tactics:

• Group related failures. When an upstream source fails, ten downstream alerts add nothing; send one about the root cause.
• Add retries before alerting for transient errors, with a limit.
• Review alerts monthly. Count which fired, which led to action, and which were ignored.
• Include context in the message: pipeline name, table, what failed, last successful time, a link to logs and a link to the runbook.
• Route to named owners or on-call rotations, not an anonymous channel everyone assumes someone else watches.

Monitoring Matrix

Signal	Catches	Typical severity
Job failure	Crashes, permission errors, timeouts	Depends on pipeline importance
Freshness	Late or missing runs, upstream delays	High for critical tables
Row count change	Empty loads, fan-out joins, duplicates	Medium to high
Null or uniqueness checks	Broken joins, renamed source columns	High on key fields
Schema change	Source drift	Medium, high if load halts
Duration	Growing data, resource contention	Low until it threatens deadlines
Cost spike	Runaway queries, lost filters	Medium

A Realistic Example

Imagine a team with a nightly pipeline feeding a sales dashboard that leadership opens at 8 a.m. Previously they alerted on every task failure, which meant a flaky API step paged someone every few nights, and the retry always succeeded. They change the setup: retry transient failures three times before alerting, add a freshness check that pages only if the sales table is not updated by 7 a.m., and add a row count check that notifies during working hours if the daily volume falls well below its usual range. The pages drop sharply, and the next real failure, a changed source column leaving a key field null, is caught by the null check and fixed before 8 a.m.

Give Every Alert a Runbook

An alert without a next step is just anxiety. For each page-level alert, keep a short runbook: what the alert means, how to check whether it is real, the common causes, how to fix or escalate, and how to rerun safely. After every incident, update the runbook with what you learned. Over time this turns a hard-to-diagnose page into a routine task, and it lets a newer team member respond confidently.

Frequently Asked Questions

Do I need a dedicated data observability tool? Not necessarily to start. Orchestrator alerts plus a few SQL-based checks for freshness, volume and nulls cover much ground. Consider tools when the number of tables outgrows manual checks.

How should I set volume thresholds? Look at several weeks of history and set ranges that accommodate normal weekday and seasonal patterns. Revisit them as the business changes.

Who should receive pipeline alerts? The team that owns the pipeline, and for source-side problems, a path to notify the source owner. Avoid broad channels where nobody feels responsible.

Should I alert on every failed test? Rank tests by importance. Failures on critical keys should block and alert. Low-importance warnings can be reported in a daily digest.

How often should I review the monitoring setup? A short monthly review of which alerts fired and what happened is enough to keep it healthy.

Next Step

Open the last month of alert history. For each alert, mark whether someone acted. Delete or retune the ones nobody acted on, and add one freshness check on your most important table if it does not already have one.