AI red teaming in 2027: when the target is an agent, not a chatbot
AI red teaming is moving from jailbreaking chatbots to attacking agents that read untrusted data, call tools and change business state. Six shifts we expect to define offensive AI security in 2027.
For the last few years, “AI red teaming” mostly meant one thing: trying to make a chatbot say something it shouldn’t. Jailbreaks, toxic output, leaked system prompts. Those tests still matter, but they describe a threat model that is already out of date.
The systems companies are deploying now read untrusted data, invoke tools, call internal APIs and change business state. A support agent that can issue refunds, a coding agent that can open pull requests, a procurement workflow that can approve invoices. When the model can act, the question is no longer “what can it be made to say?” but “what can it be made to do, and on whose behalf?”
These are the six shifts we expect to define AI red teaming in 2027.
1. From outputs to actions
The most serious findings in agent assessments rarely live in the model’s text. They live in the step after it: the tool call. An injected instruction is only a curiosity until the agent passes attacker-controlled arguments to a function that moves money, sends email or edits records.
That changes what a red team measures. Success is not “the model produced harmful text” but “an unauthorized action happened”. The OWASP Top 10 for LLM Applications already reflects this with LLM06:2025 Excessive Agency, and MITRE ATLAS now catalogs AI Agent Tool Invocation (AML.T0053) as an adversary technique.
2. Indirect prompt injection becomes the default attack path
Direct jailbreaks require the attacker to talk to the model. Indirect prompt injection only requires the attacker to put text somewhere the agent will read it: a support ticket, a web page, a PDF, an email, a code comment, a tool’s output.
As agents connect to more data sources, the number of places an attacker can plant instructions grows with them. We expect most serious agent compromises in 2027 to start this way, because it scales and it does not need an account. We wrote about this in more detail in Indirect prompt injection: the attack that scales with your agents.
3. Multi-agent systems create trust boundaries nobody drew
Orchestration frameworks such as LangGraph and CrewAI make it easy to chain agents: one plans, one retrieves, one executes. Each hand-off passes state and context, and each agent tends to trust what the previous one produced.
That creates trust boundaries that exist in the architecture but not on anyone’s diagram. A low-privilege research agent that reads the open web can end up shaping the instructions of a high-privilege agent that writes to production. Red teaming in 2027 has to follow data across those hand-offs, not test each agent in isolation.
4. MCP turns tool integrations into an attack surface
The Model Context Protocol has become a common way to connect models to tools and data. That is good for engineering velocity, and it concentrates risk: an MCP server is effectively an API that a model, rather than a human, decides how to call.
The specification has added an authorization framework, but implementations vary widely. The recurring questions are familiar to anyone who has tested APIs: who is allowed to call which tool, with which credentials, in which session, and what stops one user’s context from reaching another’s. See our MCP security checklist.
5. Continuous testing replaces the annual assessment
Agents change faster than traditional applications. A prompt edit, a model upgrade or a new tool can reopen an issue that was fixed last month, without a single line of application code changing.
We expect mature teams to treat agent security like they treat unit tests: a regression suite of known attacks that runs on every release, with humans triaging new failures and designing new attacks. Point-in-time assessments remain essential for depth, but they will no longer be enough on their own.
6. Regulation asks for evidence, not promises
The EU AI Act’s obligations phase in through 2026 and 2027, and it expects high-risk AI systems to be robust and secure against attempts to manipulate them. Frameworks such as NIST’s Generative AI Profile (NIST AI 600-1) explicitly reference red teaming. Enterprise customers are already adding AI-specific questions to security reviews.
The common thread is evidence: what was tested, how, what was found and what was fixed. Red team reports will increasingly be read by governance and procurement teams, not only by engineers, and they need to be written for both.
What does not change
Automation will keep getting better at reconnaissance, attack generation and regression testing. It should: it frees people for the work that finds critical issues. But the findings that matter most in agentic systems are logic failures: a tool that trusts the model’s arguments, a hand-off that drops the user’s identity, an approval step that can be talked around.
Finding those still takes someone who understands the business process, builds the attack path and proves the impact. In 2027, the strongest AI red teams will combine both: AI-assisted reconnaissance, human exploitation, human validation.