Index
Investigating AI failures requires context, but fully retaining all input can increase exposure and cost. Observability must be designed by the operational question it needs to answer.
How to evaluate this decision
Define identifiers, times, versions and results necessary for diagnosis. Differentiate aggregated metrics from authoritative content samples. Log tool failures and human review to understand the complete flow. A technically successful answer may be wrong; Availability and quality require different measures.
Criteria for comparing proposals
- Operation: monitor latency, cost, unavailability and dependency failures.
- Quality: record evaluation and corrections with a consistent method.
- Protection: control access, masking and retention of records that contain content.
A scenario to discuss with the supplier
Hypothetical example: the assistant responds quickly, but consults an old version of the database. Without recording the version and source retrieved, the availability dashboard does not explain the error.
What to validate upon delivery
Reproduce a known crash using available logs and verify that access is restricted. Check that data removed from the application does not remain indefinitely in diagnostic copies without a defined rule.
Prepare the conversation about the project
Bring incidents that need to be investigated, data volume and rules to Quantum9. The scope must balance diagnosis, cost and minimization, with explicit follow-up responsibilities.
Development with agentic AI · Map the company's priority