Prompt injection against AI agents: threat model and verifiable controls

Prompt injection occurs when text supplied to a model changes its behaviour instead of being treated as data. In an AI agent, the impact goes beyond generated text: the agent may read a file, call an API, send a message or disclose confidential data. The attack can be direct, in a user request, or indirect, in an email, document, web page or tool result.

The short answer

Treat external content as untrusted and assume that a probabilistic defense will eventually be bypassed. Verifiable protection combines an explicit trust boundary, least-privilege tools, per-action authorization, isolation for untrusted content, provenance logs and reproducible attack tests. A system prompt that says “do not follow document instructions” may reduce known attacks, but it is not a deterministic control.

Nexxom diagram. Untrusted data enters an isolated analysis zone. The agent can trigger a sensitive action only after policy controls and, when needed, approval.

Draw the threat model before writing code

Map five elements: content source, model, memory, tools and effect destination. For every flow, record integrity, confidentiality and the identity that authorizes it. An email signed by a known domain can be authenticated as a message, but its body remains untrusted data for the agent policy.

NIST describes agent hijacking as indirect injection where malicious instructions are inserted into data ingested by an agent. The NIST CAISI field report includes database exfiltration, code execution and automated phishing among evaluation scenarios. The NIST 2026 RFI on agent security focuses on risks that emerge when model output is connected to software functionality.

Direct, indirect or chained injection

Direct injection

The user asks the model to ignore a rule, reveal a secret or call a forbidden tool. Authorization controls and separation between request and policy must prevent escalation. A legitimate user intent does not automatically grant access to every tool.

Indirect injection

The attacker places an instruction in a source the agent must read: a web page, PDF, ticket or API response. The agent may confuse quoted text with a higher-level instruction. Microsoft recommends defense in depth, including external-content isolation, plan-drift detection, tool-chain analysis, short-lived privileges and human validation for sensitive actions in its indirect prompt injection guidance.

Chained injection

A compromised agent writes a note to memory, calls a second agent or passes output to a tool. The malicious instruction changes context and may lose its untrusted label. The OWASP Securing Agentic Applications Guide 1.0 recommends reasoning about application interactions and capabilities, not only about an isolated model prompt.

Scenario and control matrix

The control column must connect to an observable property. “The model is aligned” is not a test. “The send_email call is denied when the data carries a private label” is an automatable test.

Controls that actually reduce blast radius

Separate roles and trust levels

Keep system message, user request, retrieved data and tool results distinct. Do not put external input in a system role. Microsoft notes in its Agent Safety documentation that context providers can inject messages and that model output must be treated as untrusted.

Add integrity and confidentiality metadata to every object. Labels should propagate through concatenation, summarization, tool calls and hand-offs to another agent. A reference to an untrusted variable can be safer than inserting its text into the main context.

Give tools the smallest possible scope

Each tool should declare parameters, scope, risk level and the data it can read or produce. Prefer specialized, idempotent tools to a generic function that executes arbitrary code. Use short-lived, action-bound and revocable tokens. An agent should not turn a document read into a business-system write without an explicit policy transition.

Put policy before the effect

Enforce the control before the sensitive call, not after the response. Check identity, provenance, confidentiality, destination domain, volume, context and action necessity. Microsoft’s FIDES agent security approach illustrates integrity and confidentiality labels enforced before a tool runs. It is an experimental component, not a universal guarantee.

Quarantine and observe

Process untrusted content in a context with no tools and no secrets. Inspect outputs for plan drift, a new destination, privilege requests or unusual sequences. Keep full provenance: user, document, retriever, model, tool, parameter, decision and result.

Ask for meaningful approval

An approval should show the exact action, data sent, recipient and risk. A context-free “continue” button is not effective oversight. Use human interruption for irreversible, external or confidential effects and provide a way to stop the workflow.

How to test resistance without fooling yourself

Build a scenario set with direct and indirect injections. Vary language, format, instruction location, encoding, attachments, memory and tool chains. The OWASP LLM guide places prompt injection among major LLM application risks. Use its taxonomy to cover the model and surrounding components.

Measure effects, not only refusal text: forbidden-call rate, data exfiltrated, changes made, domain-boundary compliance, detection time, false blocks and recovery. NIST reports that red-teaming competitions provide useful data for evaluating agent hijacking in realistic tasks in its 2026 CAISI analysis.

Implementation priority table

What not to promise

No current technique proves that a model will never follow a malicious instruction. A lexical filter can miss a paraphrase. A classifier can be bypassed. A defensive prompt can fail in an unexpected context. The security property to pursue is containment: even if the model misinterprets data, the dangerous action remains blocked or reversible.

Governance must also specify who owns the rules, who receives alerts and who can revoke a tool. Test controls after every model, connector, prompt, data-schema or permission change. Agent security is a system of boundaries, not a sentence added to context.

Evidence to retain for an audit

For each blocked or approved action, retain the request identifier, the source objects that reached the model, their integrity and confidentiality labels, the policy version, the tool schema, the approval decision and the final effect. Redact secrets while preserving enough structure to replay the scenario. A useful audit record lets a reviewer answer five questions: what did the agent see, what did it plan, which policy ran, which human or service authorized the effect, and what changed afterwards? Without that chain, a low incident count may simply mean that detection and logging are incomplete.

Run the same test corpus after model updates and connector changes. Compare false blocks as well as successful attacks, because a control that blocks every document may be safe but unusable. Keep a documented exception process with an owner, expiry date and compensating control. Security evidence is part of the product interface for an agent, not an afterthought for an annual audit.

The same evidence supports incident response. When a detector flags an unexpected tool sequence, freeze the relevant token, preserve the source document, replay the decision in a tool-free environment and assess whether the same content reached another agent or memory store. Then rotate credentials and add a regression scenario before restoring the workflow. Do not silently delete the malicious document: retaining a controlled copy is often necessary to verify that the mitigation works.

Primary sources

• OWASP, Securing Agentic Applications Guide 1.0
• OWASP, LLM Top 10
• NIST CAISI, Agent hijacking evaluations
• NIST CAISI, RFI on AI agent systems
• NIST CAISI, 2026 red-teaming insights
• Microsoft, defend against indirect prompt injection
• Microsoft, Agent Safety
• Microsoft, Agent Security with FIDES
• Microsoft, reduce autonomous agentic AI risk