Index
An incremental load can update changed records while leaving deleted records visible in reports. Detecting deletion and deciding its analytical effect are separate tasks that belong in scope.
The design to commission
Determine whether the source reports deletion, inactivity or simply stops returning the record. Absence from a paginated query does not prove removal; filters, permissions or incomplete loads can cause it. Define handling by entity. An inactive master record may remain necessary for interpreting historical events, while an incorrect record may require indicator correction. Retention policy comes from business owners and applicable obligations, not unilateral pipeline decisions. Compare event capture with periodic complete reconciliation and estimate source-reading cost. A hybrid approach may fit when not every entity announces deletion. Require a trace explaining when changes were detected and how views were affected, including reprocessing. Reporting owners should approve whether a past result is corrected or remains a documented snapshot rather than leaving that choice implicit in code.
Supplier criteria
- Distinguish deletion from access failures and partial responses.
- Define historical and current-indicator treatment explicitly.
- Have accountable owners approve retention and disposal policies.
Acceptance with an exception
In a hypothetical test, a record temporarily disappears from one page and returns later. The pipeline must not delete it as confirmed removal; an actual deletion must follow the entity's agreed treatment.
Reference for assessing scope
PostgreSQL constraint documentation
Prepare your project with Quantum9
Bring Quantum9 examples of genuine deletions and incomplete queries. Discovery defines how to detect each and what effect it should have on published analytical results.