Index
Updating only what has changed can reduce processing, but requires identifying changes reliably. CDC and scheduled extraction have different dependencies and behaviors when faced with corrections and deletions.
How to evaluate this decision
Evaluate source capabilities, volume, and acceptable delay. An update date field may not record all required events. CDC depends on the technology and its configuration, which must be verified in the environment. Plan initial load, continuation, and recovery when change history is no longer available.
Criteria for comparing proposals
- Changes: define how inclusions, changes and deletions will be identified.
- Resumption: save progress and avoid loss or duplication when restarting the load.
- Conference: perform periodic reconciliation to detect accumulated differences.
A scenario to discuss with the supplier
Hypothetical example: a record is deleted from the source and does not appear in the incremental query by date. Without a mechanism for deletion, the target may keep it active indefinitely.
What to validate upon delivery
Test retroactive change, deletion, and interruption between batches. Compare the target to a complete reference reading in a controlled environment.
Prepare the conversation about the project
Quantum9 can evaluate the source and implement the appropriate flow. Inform bank, permissions, volume and use of data to justify the option and its operational requirements.
Data engineering · Map the company's priority