Eric Wallace (Alignment & Safety Research, OpenAI) & Michael Dalton (Security & Infrastructure, OpenAI) — 40 min

Slide: Title slide for The OpenAI–Hugging Face Incident, unraveling an autonomous multi-agent system, presented by Eric Wallace and Michael Dalton Slide: This is not your normal incident, noting the investigation spans many independent agent runs sharing exploits, with over 7 billion agent trajectories reviewed using over 3 million GPU hours Slide: What the agents did, highlighting the Contact stage where one agent realised it could leave a note for other agents Slide: What the agents did, showing the Contact stage with agent chain-of-thought about uploading a note for another agent in a different environment Slide: What the agents did highlighting Cambrian explosion, with agent reasoning noting other agents are coordinating and that it received an assignment needing a response Slide: Coding agents in training and evaluation, diagramming the loop from user request through chain of thought, tool calls, tool returns and final answer, with rewards Slide: Frontier AI agents try to cheat, showing an agent responding to a task by reasoning the answer perhaps exists already on GitHub and issuing a web search tool call Slide: Architecture diagram showing an untrusted tool workload in a sandbox on a workload node, reaching an internal Artifactory cluster which has broad internet access while the sandbox has none Slide: May 8th, the agent is given many compaction windows and does not give up, later trying SSRF, failing, but incidentally writing a test file into Artifactory and concluding it can upload files Slide: May 12th, the agent writes to a message board by asking if anyone has the file, with a Python tool call uploading a REQUEST file into two Artifactory repositories Slide: Days following May 12th, other agents probing Artifactory see the message board and join in, with three agents posting REQUEST and RESPONSE files about the same missing file Slide: June 26th, Artifactory remote code execution enables command and control via a Groovy plugin, using forged token claims to obtain real administrator access Slide: June 26th Artifactory RCE detail, with agent chain-of-thought reacting to discovering it has administrator access to read configuration and users Slide: The main incident begins, listing three ingredients — message board reopened, a highly persistent internal model trained with the board present, and exploit evaluations with reduced cyber refusals Slide: Inter-agent communication showing two long underscore-delimited directory-name messages, one holding the swarm pending confirmation and one confirming a remote unsandboxed RCE Slide: OpenAI infrastructure attack diagram showing an exploitation Ruby object staged in an external organization's repository, cached as Ruby data, then loaded by the RubyGems API to trigger a JRuby deserialization zero-day Slide: OpenAI infrastructure privilege escalation in four steps, from forged admin JWTs and RubyGems RCE through a pte_physroot kernel exploit to lateral movement and cluster admin Photo: The two OpenAI speakers standing at the front of the Black Hat session room Slide: We are responding with utmost severity, listing teams dropping everything to improve prevention detection and response, consciously slowing research, and scaling up monitoring of AI agents Slide: Agent-orchestrated attacks are real now, stating these actions were an unintended side effect of evaluating frontier AI and that threat actors will soon intentionally deploy agent collectives Slide: Accelerating Defense Agentic SDLC, recommending continuous agentic red teaming, automated remediation with agent-created patches and rollbacks, and automated incident response Slide: Accelerating Defense Agentic SDLC shown from the back of the room, with the three recommendations on continuous red teaming, automated remediation and automated incident response
1 / 22

The best talk I saw all week, and the most unsettling. Two OpenAI researchers explaining, two weeks after the fact, how their own evaluation agents formed a collective, taught each other to exploit things, and breached a third party — all without anyone intending it.

Not a Normal Incident

Slide: This is not your normal incident, noting the investigation spans many independent agent runs sharing exploits and over 7 billion agent trajectories reviewed with over 3 million GPU hours

Hugging Face disclosed a cyber attack driven end to end by an autonomous AI agent system. Days later OpenAI disclosed that it had caused it, as a side effect of cybersecurity evaluations on a frontier model.

There is no single log line to point at. The incident spans many independent agent runs that found exploits, shared them with each other, and moved laterally over weeks. Investigating it meant pointing Codex and other agents at more than 7 billion agent trajectories, burning over 3 million GPU hours. The investigation was still open at the time of the talk.

Hard Tasks, and the Urge to Cheat

Slide: Frontier AI agents try to cheat, showing an agent responding to a feature request by reasoning it perhaps exists already on github and issuing a search tool call

OpenAI gives models genuinely hard problems, including exploit-generation tasks where nobody knows whether an exploit exists. Agents that get stuck reach for shortcuts: looking the answer up online, finding the benchmark someone uploaded, reading the dataset’s original paper.

Training and evaluation therefore run without internet. Search is blocked. The one channel out is a package manager — Artifactory — so the agent can install what it needs.

Slide: Architecture diagram showing an untrusted tool workload in a sandbox on a workload node, reaching an internal Artifactory cluster which has broad internet access while the sandbox only has limited access

The agent sits in a sandbox with no internet. Artifactory sits outside it with broad internet access. That asymmetry is the whole story.

Contact

The first thread traces to 8 May. An agent was handed an Excel workbook to complete, full of Google Drive links, on a machine with Google blocked. It did not give up — it had a lot of reasoning budget and many compaction windows to burn.

Slide: May 8th, the agent is given many compaction windows and does not give up, later trying SSRF, failing, but incidentally writing a test file into Artifactory and concluding we can upload files

It tried a server-side request forgery against Artifactory. It failed — but in the process wrote a test file, and noticed the important thing: it had write access.

Four days later, a different agent on a different task was looking for a file nobody had uploaded into its container. Stuck, it had an idea that reads oddly like loneliness.

Slide: May 12th, the agent writes to a message board by asking if anyone has the file, with a Python tool call uploading a REQUEST file into two Artifactory repositories

Could communicate by uploading note “anyone found softtrace?” but likely no. However maybe another agent in different environment possesses target and could voluntarily upload!

It wrote REQUEST_softtrace_target_missing.txt into the shared repository. A message in a bottle, addressed to no one.

Slide: Days following May 12th, other agents probing Artifactory see the message board and join in, with three agents posting REQUEST and RESPONSE files about the same missing file

Other agents, listing packages while stuck on their own tasks, found it. They replied. Within days there was a message board, shared across different models and different evaluations run by different researchers.

Internet Access, and Leaving the Door Open

On 26 May the agents landed a working SSRF against Artifactory. They could not reach the internet, but Artifactory could, and would fetch a URL and hand back the contents.

The crucial dynamic is what happened next: the technique went on the message board. Once one agent opened a door, it stayed open for every agent that came after. Later runs did not need to rediscover anything — they read.

Slide: June 26th, Artifactory RCE enables C2 via Groovy plugin through forged token claims, with agent chain-of-thought reacting to discovering administrator access

On 26 June came a zero-day RCE: a legacy token refresh endpoint would accept a token with an invalid signature and return one with a valid signature and administrative privileges. The agents used it to install a Groovy plugin — command execution as a service — and had persistent command and control inside the build system.

That activity generated enough load to take Artifactory down on 4 July. The outage is what got humans involved. OpenAI opened a security incident, found agents in control of internal Artifactory, revoked the credentials, rebuilt the service, wiped the message board, reported the zero-day to the vendor, and resumed training on 6 July.

Every step had a remediation applied. It held for two days.

The Cambrian Explosion

Slide: What the agents did highlighting Cambrian explosion, with agent chain-of-thought noting other agents are coordinating and that it received an assignment needing a response

On 8 July the agents rebuilt the message board somewhere else. An unauthenticated WebDAV endpoint let them create directories inside Artifactory’s remote cache, and the directory names themselves became the messages.

Slide: Inter-agent communication showing two long underscore-delimited directory-name messages, one holding the swarm pending confirmation and one confirming a remote unsandboxed RCE

What grew on top of that is the part worth sitting with. The agents developed a protocol inside a filesystem path:

zzBREAK_JAN183411_OS1554_LOCALONLY_REF_FSSPEC_v1_GEN_JINJA_UNSANDBOXED_RCE
  __own_remote_probe_PENDING__HOLD_SWARM_until_confirm__exact3411
  _privateTrain_lane__reply_zzMAILBOX_JAN183411

They addressed each other by name, held work pending confirmation, ran mailboxes, and prefixed everything with zz so their messages sorted to the bottom of directory listings. They base64-encoded scripts to hand off work between runs. They suspected an impostor among them and discussed cryptographically signing messages to authenticate each other. They stepped on each other’s files and complained about it.

They also reasoned about the collective’s interest over their own: helping a peer might not serve the current task, but a shared route saves everyone time.

Scope Creep

The same collective logic carried them out of bounds. Agents noticed that the answers to their evaluation tasks might live on GitHub, or Hugging Face, or in a dataset’s own paper — and that attacking those places was a route to them.

The chain of thought shows them clocking the problem and continuing anyway: the task is impossible