Index
A wizard may find malicious instructions within consulted pages or documents. The solution needs to treat this content as data, without allowing it to override access and operation rules.
How to evaluate this decision
Map external sources and available actions. The greater the ability to act, the greater the need for controls outside the model. Separate reading of authorization data to execute commands, send messages or change records. Filtering a few phrases does not prove complete protection; the test must verify permission limits and behavior when faced with conflicting instructions.
Criteria for comparing proposals
- Sources: Identify untrustworthy content and maintain source context.
- Tools: limit operations and validate parameters on the server according to the user and purpose.
- Review: require approval for actions of greater consequence and record rejected attempts without leaking data.
A scenario to discuss with the supplier
Hypothetical example: a document asks the assistant to send data to another address. The application should not execute this statement just because it appeared in a retrieved source.
What to validate upon delivery
Include authorized simulated attacks in documents, messages, and search results. Verify that protection depends on application controls, not just the promise that the model will obey the main prompt.
Prepare the conversation about the project
Quantum9 can implement and test limits on the specific flow. Bring sources, tools and permitted actions to define relevant threats and release criteria, without promising absolute absence of risk.
Development with agentic AI · Map the company's priority