Red teaming LLMs exposes a harsh truth about the AI security arms race

The world of AI is constantly evolving, with new models and technologies emerging at a rapid pace. However, with this rapid advancement comes new challenges, particularly in the realm of security. Red teaming, a practice of testing systems by simulating attacks from an adversary, has become a critical tool for identifying vulnerabilities in AI models.

One of the key findings from red teaming exercises is that it’s not always sophisticated, complex attacks that can bring down a model. In fact, it’s often the persistent, continuous, random attempts that can ultimately lead to a model’s failure. This harsh truth has significant implications for AI developers and platform builders, who must now plan for these types of attacks as they build and release new products.

The arms race in cybersecurity has already begun, with cybercrime costs reaching staggering levels and forecasted to continue rising. Vulnerabilities in Language Model Models (LLMs) have contributed to this trend, with several high-profile incidents resulting in significant financial losses and regulatory scrutiny for the companies involved. The UK AISI/Gray Swan challenge, for example, demonstrated that no current frontier system can resist determined, well-resourced attacks.

As the gap between offensive capability and defensive readiness widens, AI builders must prioritize security testing and integration into their development processes. Tools like PyRIT, DeepTeam, Garak, and OWASP frameworks can help identify and address vulnerabilities before they can be exploited by malicious actors.

Attack surfaces are constantly evolving and shifting, making it even more challenging for red teams to keep up. OWASP’s 2025 Top 10 for LLM Applications highlights the changing landscape of threats facing AI systems, with new vulnerability categories emerging that pose unique risks to generative AI systems. AI-driven models are non-deterministic, introducing unprecedented risks that builders must be prepared to address.

Model providers validate security in different ways, with each company employing unique red teaming processes to test the robustness of their systems. Anthropic and OpenAI, for example, take different approaches to security validation, versioning compatibility, and persistence testing, as reflected in their system cards.

Defensive tools struggle to keep up with adaptive attackers, who can rapidly refine their approaches to bypass traditional defenses. Open-source frameworks like DeepTeam and Garak are emerging to address these challenges, but builder adoption lags behind attacker sophistication.

In conclusion, AI builders must prioritize security in their development processes, implementing strict input and output validation, separating instructions from data, and conducting regular red teaming exercises to identify vulnerabilities. By taking a proactive approach to security, AI builders can stay ahead of evolving threats and protect their systems from malicious attacks.