Data Monitoring / Pipeline Freshness & Data Quality
Data pipelines: Business key uniqueness
An incremental load adds duplicates without changing the reported row count trend dramatically. Compare total rows with distinct required keys and inspect duplicate groups by batch.
Environment and inputs: Data pipelines and analytical datasets; Apache Airflow is an orchestration example. Quality acceptance rules must match the dataset contract.
Failure scenario
An incremental load adds duplicates without changing the reported row count trend dramatically. Confirm the cause before choosing a fix.
Evidence to collect
Compare total rows with distinct required keys and inspect duplicate groups by batch. Record the time, affected scope and a healthy comparison interval.
Worked diagnostic example
Illustrative SQL duplicate-key check; replace table and key with the actual contract. Scope large tables to a suitable batch.
SELECT business_key, COUNT(*) AS copies
FROM monitored_batch
GROUP BY business_key
HAVING COUNT(*) > 1;Fix the confirmed cause
Enforce the agreed key rule or document legitimate multiplicity at a different grain. Write the expected result first. Change one relevant setting at a time.
Check the result and hand over
Repeat the check for business key uniqueness using the same scope and a representative interval. Check that the symptom is resolved, expected records are present and downstream results are correct. If coverage, correctness or latency worsens, restore the recorded configuration through the agreed rollback process.
Scope and precautions
Prefer read-only access and limit queries to the relevant period or dataset. This example has not been tested on your systems. Test changes in an approved environment. Keep credentials and sensitive data out of shared evidence.