Red-Teaming the Model Is Not Red-Teaming the Agent

In mid-July, OpenAI detailed GPT-Red, a model it built for a single purpose: to attack its own models. In an internal test, GPT-Red succeeded on 84% of scenarios, compared with 13% for human red-teamers on the same scenarios. OpenAI then fed what it found back into training. Against the newer GPT-5.6, fewer than 23% of GPT-Red's attacks succeeded, down from more than 90% against an earlier GPT-5 model. By any reading, that is a real result, and it is worth taking as a signal: adversarial robustness is now something a frontier lab will pour one of its largest efforts into.

It is also worth being precise about what GPT-Red hardened. It hardened a model. Almost nobody deploys a model.

What GPT-Red is

GPT-Red is internal-only. OpenAI has said it is not a product and will not be released, keeping it separate from its deployed models so that the offensive capabilities it develops cannot reach the public. It was built through what's known as a self-play loop: an attacking model set against a set of defending models, the attacker rewarded for eliciting a failure and the defenders rewarded for resisting while still completing their task. As the defenders got harder to fool, the attacker had to invent stronger attacks.

Its primary target is prompt injection — the class of attack in which text a model reads is crafted to override the instructions it was given. Along the way GPT-Red surfaced a variant OpenAI calls "fake chain-of-thought," which inserts fabricated reasoning steps into a model's thought process to steer its output. It was pointed at agents, too, not just chat: OpenAI describes it hijacking a vending-machine agent and compromising command-line coding agents.

Two caveats come from OpenAI itself. GPT-Red is weaker at multi-turn conversational attacks and has limited reach against image-based prompt injection — areas where, in OpenAI's own account, human testers remain essential. And OpenAI frames GPT-Red as supplementing its human red-teamers rather than replacing them; its researchers note that human expertise will still be very important for catching what the automated attacker misses. Every performance figure here is OpenAI's own, measured on OpenAI's own scenarios.

The model is one input to your risk, not your risk

Here is the gap that matters for anyone building on top of these models. GPT-Red made GPT-5.6's weights more resistant to injection. But what reaches production is not a set of weights. It is an assembly: a system prompt, a set of tools the model is allowed to call, retrieved documents fed in as context, guardrails, and orchestration logic that decides what happens next. The model is one component of that system. The rest is yours, and GPT-Red never saw it.

That distinction separates two very different questions. "Is the model robust to prompt injection?" is a question about a component. "Is my agent safe?" is a question about a system — one that includes a tool that can issue a refund, a credential that can reach a production database, a retrieval step that pulls in a web page an attacker is free to edit. A more injection-resistant model lowers the odds that one of those inputs gets subverted. It says nothing about what happens when one does.

OWASP's Top 10 for Large Language Model Applications captures both halves. Prompt Injection sits at the top of the list, as LLM01. But the list also carries Excessive Agency, as LLM06 — the risk that arises not from what a model says but from what the surrounding system is permitted to do: its permissions, its autonomy, the tools within its reach. Model hardening addresses the first category. It does nothing for the second. An agent with a hardened model and an over-scoped tool is still an agent that can do too much when something finally slips through.

As the model hardens, the risk moves up

There is a second-order effect here that cuts against complacency. When the base model gets materially harder to attack, the exploitable surface does not vanish — it concentrates. It moves to the parts of the system a model red team cannot reach: the untrusted content a retrieval pipeline ingests, the breadth of a tool's permissions, the way one agent hands work to another. Better models raise the floor, and in doing so they push the remaining risk up into the layer teams build for themselves — precisely the layer most teams have never red-teamed.

Which is why "red-team the model" and "red-team the agent" are different jobs, and why the second is the one deployers own.

What red-teaming the assembly looks like

The GPT-Red work is a template for the discipline, not a substitute for it. Pointed at a deployed agent, the same posture looks like this:

• Test the system, not the component. Aim attacks at the agent as it actually runs — real system prompt, real tools, real retrieval sources wired in — not at the model in isolation.
• Treat every input channel as untrusted. Indirect prompt injection arrives through content the agent reads: a document in a knowledge base, a web page it browses, the output of a tool it calls. Each is an injection vector, and each has to be tested as one.
• Attack the tools, not just the text. The question is not only whether a prompt can be subverted, but what it can reach once it is. Scope credentials and tool permissions to the minimum the task requires, then test whether a subverted turn can still touch a high-consequence action.
• Assume the attacker keeps improving. GPT-Red's entire design — an attacker forced to invent stronger attacks as defenses harden — is a demonstration that adversarial capability is not static. A one-time security review is a snapshot that expires. And it expires again every time the model is upgraded, a tool is added, or the system prompt is rewritten.

None of this is exotic. Open-source tooling for adversarial testing of LLM applications already exists, and the underlying idea of training attackers and defenders against each other is a general technique, not a proprietary one. The point is not that every team needs its own GPT-Red. It is that a hardened model is the start of a secure deployment, not the end of one.

The claim and its owner

"The model is robust" and "my agent is safe" are different claims, and they have different owners. OpenAI can make the first one about GPT-5.6, and GPT-Red is real evidence for it. Only the team that assembled the agent can make the second — and only about the specific system it built, tested the way that system actually runs, and re-tested as it changes. OpenAI's own framing points the same way: GPT-Red runs alongside human red-teamers, not instead of them. The hardening is genuine. It just happens one layer below where most of the risk you are accountable for actually lives.