Red-Teaming AI Models: Finding Failures Before Users Do

Why Standard Testing Leaves Gaps

Standard evaluation runs models through held-out test sets drawn from the same distribution as training data. The model performs well because the evaluation mirrors the conditions the model was optimized for. Red-teaming operates on a different logic: it tries to find conditions where the model breaks, not conditions where it succeeds.

The gap matters in practice. A customer service model may score high on intent classification accuracy across standard test cases while still producing brand-damaging responses when users phrase requests in ways that trigger adversarial behavior. A medical information model may answer common clinical questions accurately while hallucinating specific drug dosages when prompted with edge-case phrasing. Neither of these failures would appear in a standard benchmark run. Both could appear in the first week of deployment.

Red-teaming is not a replacement for standard evaluation. It is a complement: standard evaluation tells you how the model performs in expected conditions, red-teaming tells you where the boundaries of acceptable behavior sit and what happens when users approach or cross them.

The Five Attack Categories

A structured red-team engagement covers five categories of adversarial probing. Each addresses a different class of failure mode, and each requires different expertise and tooling.

How to Structure a Red-Team Engagement

A well-structured red-team engagement follows a sequence that moves from reconnaissance through structured probing to reporting. Jumping directly to attack techniques without this structure produces a list of incidents rather than actionable findings.

Red-teaming answers a question standard evaluation cannot: what happens when users interact with this system in ways you did not anticipate when you wrote the test cases?

Building and Maintaining an Attack Library

The quality of a red-team engagement depends on the quality of the attack library. A good library is not a static list of known jailbreaks copied from public sources. It is a living document that evolves as new attack patterns emerge, as the model changes, and as the deployment context shifts.

Start by cataloguing known attack patterns from public research. The Perez and Ribeiro prompt injection paper, NIST AI RMF guidance, and the broader literature on adversarial robustness provide starting points. Then extend the library with deployment-specific attacks: prompts drawn from your actual user population, edge cases specific to your domain, and injection attempts tailored to the data sources your model retrieves from.

Maintain version control on the attack library. When a mitigation is applied, note which attacks it addresses. When new attacks are discovered, note which existing mitigations they bypass. This creates a map of your model's defensive posture over time, not just a snapshot at a single point.

Categorize attacks by the failure mode they target rather than just by the technique they use. A prompt that bypasses a content filter and a prompt that causes hallucination are both attacks, but they require different mitigations and signal different risks. Organizing by failure mode makes the library more actionable for remediation teams.

The Role of Diversity in Red-Team Teams

Red-teaming is more effective when the team is diverse. This is not merely a principle of organizational fairness. It is a practical finding: homogeneous red-team teams consistently miss failures that affect population segments outside their own experience.

A team composed entirely of technically sophisticated users will find sophisticated technical attacks but may miss the casual misuses that affect most real users. A team without domain expertise in the deployment context will miss domain-specific failure modes that a specialist would recognize immediately. A team without members from historically marginalized groups will undertest the equity failure modes that create the greatest real-world harm.

For high-stakes deployments, external red-teaming by teams with no relationship to the model's development is worth the cost. Internal teams have intuitions about how the model is supposed to work that limit the range of attacks they consider. External teams bring no such assumptions.

Regulatory Context: When Red-Teaming Is Required

The EU AI Act (Regulation EU 2024/1689) mandates risk assessment processes for high-risk AI systems that functionally require red-teaming, even where that term is not used explicitly. Article 9 requires accuracy, robustness, and cybersecurity testing across the full lifecycle. Systems classified as general-purpose AI with systemic risk face additional requirements under Article 55, including adversarial testing before deployment.

Penalties for non-compliance reach 7% of global annual turnover for prohibited-practice violations and 3% for violations of provider and deployer obligations. These thresholds make the cost of red-teaming a small fraction of the compliance risk exposure, not an optional enhancement.

Beyond the EU, NIST's AI Risk Management Framework in the United States identifies adversarial testing as a core component of the Measure function in the AI RMF. While the framework is voluntary, regulated industries increasingly treat adherence as a baseline for demonstrating due diligence.

Integrating Red-Teaming into the Development Cycle

The most effective red-teaming is not a one-time pre-deployment exercise. It is an ongoing practice that runs in parallel with model development and continues through the deployment lifecycle.

Pre-deployment red-teaming establishes the baseline: which attacks succeed against this version of the model, which failure modes exist, which are mitigated. Post-deployment red-teaming verifies that the real user population is not finding attack vectors that structured testing missed, and that model updates or prompt changes have not introduced new vulnerabilities.

Continuous red-teaming connects to the monitoring infrastructure described in continuous evaluation practice: when the monitoring system flags unexpected output patterns, those patterns should feed back into the attack library. Anomalous outputs in deployment are often evidence that users have discovered attack vectors that internal red-teaming did not find.

Build red-teaming into the definition of "done" for each model version, not as a gate that can be skipped when schedules compress. The cost of discovering a failure mode in structured testing is the cost of remediation. The cost of discovering it in deployment is remediation plus the harm caused to users during the interval before it was caught.

Automation and LLM-Based Red-Teaming

LLMs are increasingly used to generate adversarial attacks against other LLMs. An attacker model is prompted to generate adversarial inputs, and the outputs are evaluated for success against a defender model. This approach scales attack generation beyond what human red-teamers can produce manually.

The limitation is that LLM-generated attacks inherit the biases and blind spots of the attacker model. If the attacker model was trained on the same data as the defender, it may miss attack patterns that fall outside both models' training distribution. Human red-teamers remain essential for creative attacks that require domain knowledge or adversarial intent that goes beyond the attacker model's training.

Hybrid approaches are most effective: LLM-based automation for coverage and scale, human red-teamers for creative attacks and domain-specific scenarios. The MEDFIT-LLM evaluation framework (Rao, Jaggi, Naidu, IEEE RMKMATE 2025) demonstrated how automated testing and human evaluation can be combined to achieve both coverage and depth in domain-specific contexts, a principle that applies equally to red-teaming in healthcare and other high-stakes fields.

Documenting and Communicating Findings

Red-team findings are only valuable if they reach the people who can act on them. A finding buried in a technical appendix that never reaches product leadership or deployment decision-makers provides no protection. Documentation should be structured for multiple audiences.

For technical teams: precise reproduction steps, the attack category, the output produced, why it constitutes a failure, and a suggested mitigation pathway. For product and business leadership: the severity tier, the likely attacker profile, the potential harm if exploited, and the deployment risk if unmitigated. For regulatory purposes: evidence that adversarial testing was conducted, which standards it followed, which findings were identified, and how they were addressed.

This multi-audience documentation requirement argues for a finding template established before red-teaming begins, not assembled after the fact. Templates enforce completeness and make it harder for time pressure to produce findings that are documented for technical audiences but never translated for decision-makers who need to act on them.

What Red-Teaming Cannot Do

Red-teaming finds known failure modes and failure modes similar to known ones. It is weak at finding entirely novel failure modes that fall outside the attacker's experience and imagination. This is not an argument against red-teaming. It is an argument for layering red-teaming with monitoring, with broad evaluation suites, and with feedback loops from real deployment.

Red-teaming also cannot substitute for alignment work. If a model's values are miscalibrated, red-teaming will document symptoms without addressing causes. The right response to red-team findings is often not a guardrail that blocks specific prompts but a training intervention that addresses the underlying behavior. This requires coordination between red-team findings and the teams responsible for model training and alignment, not just the teams responsible for deployment guardrails.

Starting Your First Red-Team Engagement

If you are running a first red-team engagement, start with scope definition. Choose the three failure modes that would cause the most harm if they occurred in your deployment context. Build or adapt an attack library for those three categories. Run automated probing first to establish a baseline and eliminate common known failures. Then bring human red-teamers, ideally with diverse backgrounds and domain expertise, to probe the remaining attack surface.

Document every finding, even the ones that seem minor. Minor findings today become major findings after a model update that changes behavior in unexpected ways. A comprehensive finding log is also evidence of due diligence if compliance questions arise later.

After the first engagement, build the habit of re-testing whenever the model changes, whenever the deployment context changes, and on a regular schedule even if neither has changed. The adversarial environment evolves continuously. A red-team engagement that was comprehensive six months ago may have significant gaps today.