Index
A pipeline can terminate without a technical error and still produce incomplete data. Monitoring needs to monitor execution, updating and quality of the result delivered to consumers.
How to evaluate this decision
Define expectations per load: plausible volume, covered period and availability period. Differentiate legitimate absence of movement from an empty extraction due to failure. Record progress and dependencies to resume without duplicating data. Alerts should indicate impact and responsibility, avoiding notifications that no one knows how to handle.
Criteria for comparing proposals
- Detection: check execution and results signals, including delay and unexpected drop in volume.
- Recovery: allow resumption by stage or period with reprocessing rules.
- Communication: inform reports when the update is incomplete.
A scenario to discuss with the supplier
Hypothetical example: an API responds success with an empty list because the permission has changed. The job may terminate normally, but volume validation should open an investigation.
What to validate upon delivery
Simulate outage and partial load, then perform recovery. Compare the result with a reference and check whether the consumer distinguishes current data from delayed data.
Prepare the conversation about the project
Quantum9 can instrument and stabilize the pipeline. Bring sources, times, consumers and failure history to define alerts and procedures proportional to the impact.
Data engineering · Map the company's priority