Data Monitoring / Pipeline Freshness & Data Quality
Data pipelines: Critical field completeness
A valid schema carries an increasing number of empty customer keys. Measure null, blank and invalid sentinel values separately for required fields.
Environment and inputs: Data pipelines and analytical datasets; Apache Airflow is an orchestration example. Quality acceptance rules must match the dataset contract.
Failure scenario
A valid schema carries an increasing number of empty customer keys. Confirm the cause before choosing a fix.
Evidence to collect
Measure null, blank and invalid sentinel values separately for required fields. Record the time, affected scope and a healthy comparison interval.
Worked diagnostic example
Illustrative SQL completeness check for a required key in a bounded batch.
SELECT COUNT(*) AS total_rows,
SUM(CASE WHEN customer_key IS NULL THEN 1 ELSE 0 END) AS missing_keys
FROM monitored_batch;Fix the confirmed cause
Quarantine invalid records or repair upstream production with an observable rejection count. Write the expected result first. Change one relevant setting at a time.
Check the result and hand over
Repeat the check for critical field completeness using the same scope and a representative interval. Check that the symptom is resolved, expected records are present and downstream results are correct. If coverage, correctness or latency worsens, restore the recorded configuration through the agreed rollback process.
Scope and precautions
Prefer read-only access and limit queries to the relevant period or dataset. This example has not been tested on your systems. Test changes in an approved environment. Keep credentials and sensitive data out of shared evidence.