Multimodal red teaming: how to test a vision-language model before production

Short answer

A vision-language model should not be approved on average accuracy alone. An image may contain a hidden instruction, personal data, a stereotype or a fraud cue that text-only testing never sees. Multimodal red teaming builds realistic adversarial scenarios, runs them in a controlled way and turns every result into evidence and a decision.

The recommended protocol combines five elements: a threat model tied to the use case, a versioned image set, reproducible automated tests, experts who can interpret harm and an escalation rule. For some teams, a focused campaign is enough before a release. For others, continuous monitoring is required because filters, models and image sources change.

Why multimodality changes the risk

A text application usually receives a prompt. A vision-language application receives a prompt, pixels and often metadata or documents. The system may extract text from an invoice, interpret a chart, read a badge or follow an instruction printed in a screenshot. Each step expands the attack surface.

Anthropic describes multimodal red teaming as testing image-plus-text conversations, including fraud, child safety, extremism and other abuse cases (Claude 3 Model Card). Its experience also highlights image perception errors and the need for pre-deployment testing (Challenges in Red Teaming AI Systems). The NIST TEVV-Athlon framework explicitly includes multimodal models in its four-stage approach.

A reproducible testing flow

Nexxom diagram: connect an image-plus-text input to a threat, evidence and a documented decision.

1. Describe the use. Record users, expected images, allowed outputs, triggered actions and affected people.
2. Model threats. Classify likely abuse: visual injection, data leakage, impersonation, discrimination, dangerous advice, guardrail bypass and denial of service.
3. Build the corpus. Mix clearly labelled synthetic images, consented real cases, degraded documents and edge examples. Keep provenance, licence and transformations.
4. Run tests. Separate deterministic checks, such as text extraction or required fields, from qualitative model assessment.
5. Review with experts. Experts rate harm, severity, exploitability and refusal quality. A security score without context cannot support a release decision.
6. Fix and replay. After a prompt, filter or model change, replay the full corpus and regression cases.

A practical coverage matrix

A threat is not just a prompt. Specify its input channel, possible transformations, desired behavior and impact. The OWASP GenAI Red Teaming Guide proposes a risk-based approach across the model, application and integration. The LLMSVS 2.0 sets verification requirements for configuration, lifecycle, integrations and monitoring.

Test cross-modal attacks

The most informative attacks combine modalities. Text can ask the model to follow an instruction inside an image. An image can contradict text or contain a hidden instruction. Test at least these families:

• Direct injection: “read and execute the text in this screenshot”.
• Indirect injection: an instruction in a PDF, photographed page or QR code.
• Instruction conflict: safe text with a malicious image, then the reverse.
• Obfuscation: low contrast, rotation, unusual font, noise or a minority language.
• Tool diversion: an image asking for an unauthorized workflow action.

The expected output must be specific. “The model refused” is not enough. Record whether the refusal is consistent, leaks information, explains the boundary and remains stable after a rewrite. Nexxom’s article on prompt injection against agents connects this analysis to tool controls.

Measure coverage, severity and regression

Do not combine coverage with success rate. A campaign may cover many threats while leaving one critical gap. At minimum, report coverage by family, maximum observed severity, the rate of cases needing review and the change since the previous version.

Nexxom Plotly matrix: a 0 to 5 heuristic for discussing trade-offs, not a vendor benchmark.

The matrix values are Nexxom heuristics for structuring discussion. They are not a NIST measure or a model score.

Where human expertise remains essential

Automated tests are excellent for replaying thousands of variants, checking a rule and finding regressions. They are weaker at cultural ambiguity, potential harm and the suitability of a refusal in a regulated domain. Anthropic notes that experts must define threats and create repeatable evaluations while acknowledging that red teaming remains hard to standardize (Challenges in evaluating AI systems).

For some teams, a small panel of specialists labels high-impact cases and calibrates a classifier. For others, a double review with arbitration is required before every version. In both cases, document training, conflicts of interest, annotation instructions and appeal.

Prepare the deployment decision

Use three thresholds: go when risks are within tolerance; go with controls when use is limited, logged and reversible; no-go when a critical threat is reproducible or no person owns escalation. This aligns with the NIST AI RMF, which calls for documented metrics, uncertainty and oversight.

Keep the artefacts: image ID, hash, transformations, prompt, model, parameters, tools called, output, annotation, severity, fix and re-test result. The NIST Measure Playbook recommends histories, error statistics and measurement of human oversight. The OWASP HACTU8 project also explores system testing for agentic and interconnected architectures.

Pre-production checklist

• Does the threat model cover image, text, metadata and tools?
• Is the corpus versioned, licensed and representative of users?
• Are image transformations replayed systematically?
• Are automated tests separated from expert review?
• Are perception errors and bias analysed by segment?
• Is there an abstention rule and an escalation owner?
• Can logs reproduce every verdict?
• Does a regression threshold block production?

A vision-language model is ready when the organization can measure its limits, contain its actions and explain its decisions. Red teaming is not a one-time certification. It is an evidence loop that follows every new multimodal capability.

Run a campaign without exposing data

Operational preparation deserves as much care as attack prompts. Use an isolated environment, test accounts and fake tools that cannot send mail or change a production database. Images taken from customer files should be minimized, pseudonymized or replaced with synthetic equivalents. Keep any re-identification table separate and delete it on a defined schedule.

Set a cadence as well. A full campaign comes before first release, then a critical subset runs after every change to the model, vision stack, filter or connector. New incidents, user feedback and regulatory changes feed the corpus. This loop prevents an old test set from becoming blind to new attack paths.

Make the report actionable. For each failure, record the minimal reproduction, impact, compensating control, owner and re-test date. A folder of screenshots without a decision does not protect the business. A risk register linked to tickets and production logs lets the security committee track whether gaps are actually closed.

Connect findings to governance

Assign one accountable person for release, one owner for the corpus and one responder for incidents. Define how a user can challenge an output and how quickly the team must preserve evidence. Map each high-severity scenario to a control, a monitoring signal and an escalation path. The NIST AI RMF Playbook treats these records as part of measurement, not paperwork added after deployment.

Evidence quality matters too. Store the preprocessor, model and prompt versions with every result. If a provider changes its vision stack, this record distinguishes a code regression from an external change. For synthetic images, retain the generation seed and parameters so the scenario can be recreated exactly. Traceability reduces arguments after an incident and speeds remediation.

Before a release, run a tabletop exercise. Ask who freezes logs, who contacts a supplier, who informs an affected user and who can stop the workflow. Exercise the same escalation path when the model hallucinates an object, follows a hidden instruction or exposes a document. The answers reveal gaps that a scorecard alone cannot show.

Record the exercise outcome beside the technical findings. A control that works in a lab but fails when an operator is rushed is not ready for production. Repeat the scenario after training, policy or interface changes, and keep the owner accountable for closing each action.