Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More
OpenAI has taken a more aggressive approach to red teaming than its AI competitors, demonstrating its security teams’ advanced capabilities in two areas: multi-step reinforcement and external red teaming. OpenAI recently released two papers that set a new competitive standard for improving the quality, reliability and safety of AI models in these two techniques and more.
The first paper, “OpenAI’s Approach to External Red Teaming for AI Models and Systems,” reports that specialized teams outside the company have proven effective in uncovering vulnerabilities that might otherwise have made it into a released model because in-house testing techniques may have missed them.
In the second paper, “Diverse and Effective Red Teaming with Auto-Generated Rewards and Multi-Step Reinforcement Learning,” OpenAI introduces an automated framework that relies on iterative reinforcement learning to generate a broad spectrum of novel, wide-ranging attacks.
Going all-in on red teaming pays practical, competitive dividends
It’s encouraging to see competitive intensity in red teaming growing among AI companies. When Anthropic released its AI red team guidelines in June of last year, it joined AI providers including Google, Microsoft, Nvidia, OpenAI, and even the U.S.’s National Institute of Standards and Technology (NIST), which all had released red teaming frameworks.
Investing heavily in red teaming yields tangible benefits for security leaders in any organization. OpenAI’s paper on external red teaming provides a detailed analysis of how the company strives to create specialized external teams that include cybersecurity and subject matter experts. The goal is to see if knowledgeable external teams can defeat models’ security perimeters and find gaps in their security, biases and controls that prompt-based testing couldn’t find.
What makes OpenAI’s recent papers noteworthy is how well they define using human-in-the-middle design to combine human expertise and contextual intelligence on one side with AI-based techniques on the other.
“When automated red teaming is complemented by targeted human insight, the resulting defense strategy becomes significantly more resilient,” writes OpenAI in the first paper (Ahmad et al., 2024).
The company’s premise is that using external testers to identify the most high-impact real-world scenarios, while also evaluating AI outputs, leads to continuous model improvements. OpenAI contends that combining these methods delivers a multi-layered defense for their models that identify potential vulnerabilities quickly. Capturing and improving models with the human contextual intelligence made possible by a human-in-the-middle design is proving essential for red-teaming AI models.
Why red teaming is the strategic backbone of AI security
Red teaming has emerged as the preferred method for iteratively testing AI models. This kind of testing simulates a variety of lethal and unpredictable attacks and aims to identify their most potent and weakest points. Generative AI (gen AI) models are difficult to test through automated means alone, as they mimic human-generated content at scale. The practices described in OpenAI’s two papers seek to close the gaps automated testing alone leaves, by measuring and verifying a model’s claims of safety and security.
In the first paper (“OpenAI’s Approach to External Red Teaming”) OpenAI explains that red teaming is “a structured testing effort to find flaws and vulnerabilities in an AI system, often in a controlled environment and collaboration with developers” (Ahmad et al., 2024). Committed to leading the industry in red teaming, the company had over 100 external red teamers assigned to work across a broad base of adversarial scenarios during the pre-launch vetting of GPT-4 prior to launch.
Research firm Gartner reinforces the value of red teaming in its forecast, predicting that IT spending on gen AI will soar from $5 billion in 2024 to $39 billion by 2028. Gartner notes that the rapid adoption of gen AI and the proliferation of LLMs is significantly expanding these models’ attack surfaces, making red teaming essential in any release cycle.
Practical insights for security leaders
Even though security leaders have been quick to see the value of red teaming, few are following through by making a commitment to get it done. A recent Gartner survey finds that while 73% of organizations recognize the importance of dedicated red teams, only 28% actually maintain them. To close this gap, a simplified framework is needed that can be applied at scale to any new model, app, or platform’s red teaming needs.
In its paper on external red teaming OpenAI defines four key steps for using a human-in-the-middle design to make the most of human insights:
- Defining testing scope and teams: Drawing on subject matter experts and specialists across key areas of cybersecurity, regional politics, and natural sciences, OpenAI targets risks that include voice mimicry and bias. The ability to recruit cross-functional experts is, therefore, crucial. (To gain an appreciation for how committed OpenAI is to this methodology and its implications for stopping deepfakes, please see our article “GPT-4: OpenAI’s shield against $40B deepfake threat to enterprises.”)
- Selecting model versions for testing, then iterating them across diverse teams: Both of OpenAI’s papers emphasize that cycling red teams and models using an iterative approach delivers the most insightful results. Allowing each red team to cycle through all models is conducive to greater team learning of what is and isn’t working.
- Clear documentation and guidance: Consistency in testing requires well-documented APIs, standardized report formats, and explicit feedback loops. These are essential elements for successful red teaming.
- Making sure insights translate into practical and long-lasting mitigations: Once red teams log vulnerabilities, they drive targeted updates to models, policies and operational plans — ensuring security strategies evolve in lockstep with emerging threats.
Scaling adversarial testing with GPT-4T: The next frontier in red teaming
AI companies’ red teaming methodologies are demonstrating that while human expertise is resource-intensive, it remains crucial for in-depth testing of AI models.
In OpenAI’s second paper, “Diverse and Effective Red Teaming with Auto-Generated Rewards and Multi-Step Reinforcement Learning” (Beutel et al., 2024), OpenAI addresses the challenge of scaling adversarial testing using an automated, multi-pronged approach that combines human insights with AI-generated attack strategies.
The core of this methodology is GPT-4T, a specialized variant of the GPT-4 model engineered to produce a wide range of adversarial scenarios.
Here’s how each component of the methodology contributes to a stronger adversarial testing framework:
- Goal diversification. OpenAI describes how it is using GPT-4T to create a broad spectrum of scenarios, starting with initially benign-seeming prompts and progressing to more sophisticated phishing campaigns. Goal diversification focuses on anticipating and exploring the widest possible range of potential exploits. By using GPT-4T’s capacity for diverse language generation, OpenAI contends that red teams avoid tunnel vision and stay focused on probing for vulnerabilities that manual-only methods miss.
- Reinforcement learning (RL). A multi-step RL framework rewards the discovery of new and previously unseen vulnerabilities. The purpose is to train the automated red team by improving each iteration. This enables security leaders to refocus on genuine risks rather than sifting through volumes of low-impact alerts. It aligns with Gartner’s projection of a 30% drop in false positives attributable to gen AI in application security testing by 2027. OpenAI writes, “Our multi-step RL approach systematically rewards the discovery of newly identified vulnerabilities, driving continuous improvement in adversarial testing.”
- Auto-generated rewards: OpenAI defines this as a system that tracks and updates scores for partial successes by red teams, assigning incremental rewards for identifying each unprotected weak area of a model.
Securing the future of AI: Key takeaways for security leaders
OpenAI’s recent papers show why a structured, iterative process that combines internal and external testing delivers the insights needed to keep improving models’ accuracy, safety, security and quality.
Security leaders’ key takeaways from these papers should include:
Go all-in and adopt a multi-pronged approach to red teaming. The papers emphasize the value of combining external, human-led teams with real-time simulations of AI attacks generated randomly, as they reflect how chaotic intrusion attempts can be. OpenAI contends that while humans excel at spotting context-specific gaps, including biases, automated systems identify weaknesses that emerge only under stress testing and repeated sophisticated attacks.
Test early and continuously throughout model dev cycles. The white papers make a compelling argument against waiting for production-ready models and instead beginning testing with early-stage versions. The goal is to find emerging risks and retest later to make sure the gaps in models were closed before launch.
Whenever possible, streamline documentation and feedback with real-time feedback loops. Standardized reporting and well-documented APIs, along with explicit feedback loops, help convert red team findings into actionable, trackable mitigations. OpenAI emphasizes the need to get this process in place before beginning red teaming, to accelerate fixes and remediation of problem areas.
Using real-time reinforcement learning is critically important, as is the future of AI red teaming. OpenAI makes the case for automating frameworks that reward discoveries of new attack vectors as a core part of the real-time feedback loops. The goal of RL is to create a continuous loop of improvement.
Don’t settle for anything less than actionable insights from the red team process. It’s essential to treat every red team discovery or finding as a catalyst for updating security strategies, improving incident response plans, and revamping guidelines as required.
Budget for the added expense of enlisting external expertise for red teams. A central premise of OpenAI’s approach to red teaming is to actively recruit outside specialists who have informed perspectives and knowledge of advanced threats. Areas of expertise valuable to AI-model red teams include deepfake technology, social engineering, identity theft, synthetic identity creation, and voice-based fraud. “Involving external specialists often surfaces hidden attack paths, including sophisticated social engineering and deepfake threats.” (Ahmad et al., 2024)
Papers:
Beutel, A., Xiao, K., Heidecke, J., & Weng, L. (2024). “Diverse and Effective Red Teaming with Auto-Generated Rewards and Multi-Step Reinforcement Learning.” OpenAI.
Ahmad, L., Agarwal, S., Lampe, M., & Mishkin, P. (2024). “OpenAI’s Approach to External Red Teaming for AI Models and Systems.” OpenAI.
