Red Team vs. AI: How Offensive Security Testing Must Evolve

For years, red teaming followed a fairly predictable rhythm. A skilled team of humans would spend weeks probing an environment, chaining together vulnerabilities, phishing a few employees, and eventually writing up a report that landed on a CISO’s desk with a list of findings. It worked. It still works, to a point. But the threat landscape that red teaming was built to simulate is changing underneath our feet, and the change is driven by one thing: attackers are increasingly using AI, and some of that AI is starting to act on its own.

This isn’t science fiction anymore. The early milestone that signaled where things were heading came from Google, whose Project Zero and DeepMind teams reported that their “Big Sleep” agent found a previously unknown, exploitable memory-safety vulnerability  in SQLite, a widely used open-source database engine—a flaw caught before the affected code reached a stable release. Google described it as likely the first time an AI agent had found this kind of exploitable issue in widely used real-world software.

Of course that result is over a year old now; it was an early signal, and the rate of news and the power of the tooling have continued to accelerate since. If the defenders are building agents that find novel vulnerabilities, you can be certain the attackers are exploring the same territory.

So the question for anyone responsible for offensive security is uncomfortable but simple: if the adversary is becoming autonomous, can your red team still credibly simulate the adversary?

What Actually Changes When Attackers Use AI

It’s tempting to think of AI-assisted attacks as just “the same attacks, but faster.” Speed is part of it, but that framing misses what’s genuinely different. A human attacker and an autonomous one behave in fundamentally distinct ways, and those differences matter for how you test.

• Scale and tirelessness. A human red teamer gets tired, takes weekends off, and can only pursue so many parallel threads. An autonomous agent can enumerate thousands of endpoints, test variations of a prompt injection against dozens of model endpoints, and pivot across a network continuously, around the clock. The reconnaissance phase that used to take days can collapse into hours.



• Rapid adaptation. Traditional attacks tend to follow known patterns that defensive tools are tuned to catch. AI-driven attacks can rewrite themselves on the fly—regenerating payloads, rephrasing phishing lures, and adjusting technique when something gets blocked. This defeats a lot of signature-based detection that assumes attackers reuse recognizable tooling.
• New attack surface entirely. Here’s the part that catches many organizations flat-footed. Your own AI deployments—the LLM-backed chatbots, the retrieval pipelines, the agentic workflows connected to internal data—are themselves targets. Prompt injection, model fingerprinting, and data exfiltration through model interactions are attack vectors that simply didn’t exist in most red team playbooks three years ago.

That last point deserves emphasis. We’re now in a situation where AI is both the weapon and the target. An attacker might use an autonomous agent to probe your infrastructure, and part of what they’re probing is your AI infrastructure. Offensive security has to account for both sides of that equation at once.

Why Yesterday’s Red Team Falls Short

None of this means human red teams are obsolete. Far from it—the judgment, creativity, and business context that experienced practitioners bring remains irreplaceable. But a red team that operates only at human speed, using only human techniques, and that ignores AI-specific attack surface, is now testing against a version of the adversary that is already out of date.

Consider what a conventional penetration test typically misses in an AI-heavy environment. It checks whether your firewall blocks the ports it should. It phishes your staff. It looks for unpatched software. All valuable. But does it test whether an attacker can manipulate your customer-service LLM into revealing data it shouldn’t? Does it check whether your retrieval-augmented pipeline can be poisoned? Does it fingerprint which model endpoints respond to unauthenticated requests? For most organizations, the honest answer is no—and that gap is exactly where autonomous attackers are headed.

There’s also a detection problem. Many organizations tune their defenses to the tempo and signatures of human-driven attacks. When the simulation itself runs at human tempo, the test confirms that defenses work against human-tempo attacks—and tells you nothing about how they’d hold up against something moving far faster and rewriting itself as it goes.

How Offensive Security Has to Evolve

So what does a modern offensive security program actually look like? The evolution isn’t about replacing people with machines. It’s about pairing seasoned human expertise with the same kinds of capabilities attackers are now wielding, and expanding the scope of what gets tested in the first place.

• Test the AI, not just the infrastructure around it. Red teaming now has to include prompt injection testing, jailbreak attempts, model fingerprinting, and probing of agentic workflows that can take actions on behalf of users. If your organization has deployed LLMs anywhere near sensitive data or privileged actions, those deployments need adversarial testing of their own.
• Simulate at machine speed where it matters. Introduce automation and AI-assisted tooling into your own red team so that the simulation reflects the tempo and adaptability of real autonomous threats. If you only ever test at human speed, you’ll have a blind spot the size of the gap between the two.



• Keep humans firmly in the loop. Autonomous tooling is powerful at breadth and speed, but it’s humans who understand what a finding means for the business, which attack paths an intelligent adversary would actually prioritize, and how to chain weaknesses into something that matters. The goal is augmentation, not abdication.
• Test continuously, not annually. The old model of a point-in-time assessment once or twice a year assumed a relatively static threat landscape. When attacker capabilities are evolving quickly—and researchers note rapid iteration in this space—a yearly snapshot ages badly. More frequent, ongoing adversary simulation keeps your picture current.

The honest truth is that most organizations are not there yet. The tooling, the expertise, and the methodologies for AI-aware offensive security are still maturing, and the talent pool of people who genuinely understand both adversarial AI techniques and traditional offensive security is small. That’s a real constraint, and it’s worth naming rather than pretending otherwise.

Where Plurilock Fits

This is precisely the kind of problem Plurilock’s Cyber Adversary Simulation and Response practice was built to handle. Our offensive security work has always combined elite human practitioners—including former intelligence professionals and people who’ve spent careers finding what others miss—with a deep grounding in the conjunction of AI and cybersecurity, which is where our heritage and our IP actually began.

That matters here more than it might in other contexts. AI security testing, prompt injection assessment, and adversarial testing of model deployments aren’t bolt-on services for us; they sit alongside our red and purple team penetration testing, social engineering and deepfake assessments, and incident response work as part of a single, integrated offering. We don’t just test whether your perimeter holds against yesterday’s attacker. We test whether your AI deployments, your proxy layers, and your detection stack hold up against the autonomous, adaptive, machine-speed adversary that’s already out there probing.

The red team isn’t going away. But the version of it that simulates a purely human attacker, on a human schedule, ignoring the AI surface entirely, is quietly becoming a comfortable fiction. The adversary has evolved. The question worth sitting with is whether your offensive security program has evolved to match it—and if you’re not sure, that uncertainty is itself worth acting on. ■