Jayson Grace & Martin Wendiggensen (Dreadnode), with Shane Caldwell — 35 min

Slide: Title slide for Catch Me If You Can, AI investigators hunting autonomous attackers, with speaker bios for Jayson Grace, Martin Wendiggensen and Shane Caldwell of Dreadnode Slide: From People to Agents contrasting the five-step human purple team loop with the Ares loop, which collapses generation and execution into one adaptive loop Slide: Adversary Gap noting there is enough data to test but not to train, with a matrix comparing defensive benchmarks on real queryable telemetry, investigation via tools, and an unscripted adversary Slide: The Defense Tradeoff plotting variety against noise, showing scripted attacks as low-variety and low-noise, real SOC data as high-variety and high-noise, and generators sitting off the tradeoff line Slide: The Defense Tradeoff shown from a different angle, with the variety versus noise chart and the argument that generators give variety and clean ground truth Slide: Inside the Ares Engine showing one orchestrator per side, with NATS handling durable queues and tool RPC and Redis holding shared team state, feeding seven red workers and four blue workers Slide: The Red Team as an autonomous multi-agent attacker, with one orchestrator directing seven specialist workers covering recon, credential access, cracking, ACL abuse, privilege escalation, lateral movement and coercion Slide: The Blue Team as an autonomous multi-agent investigator, with an orchestrator directing four workers — triage, threat hunter, lateral analyst and escalation triage — under strict token, time and query budgets Slide: How Blue Team Breaks Immediately, listing context overflow from Loki queries, reasoning taking too long while the attack continues, and exiting prematurely after the first signs of compromise Slide: Improvement Requires Iteration describing a replay engine that re-runs any recorded attack against the blue team, with a diagram cycling from red team op state through a replay data store to investigate, score and improve Slide: Smart Children listing prompt-level and code-level self-improvements alongside four bar charts showing investigation score, timeline accuracy, technique coverage and IOC detection rising across optimization cycles Slide: Smart Children improvement charts shown closer, with train and hold-out scores rising across six optimization cycles Slide: Tokenomics showing prompt and code-level cost reductions with a bar chart of cost per run in US dollars falling across optimization cycles Slide: Why This Scales showing 240,000 distinct trajectories built from a baseline multiplied by 300 realizations, 20 softmax variants, 5 failure variants and 8-way fan-out Slide: Replay demo mid-run during the credential access phase, showing red actions, detections fired and agent cost updating live Slide: Replay demo of a completed 34-minute operation across 3 domains and 2 forest roots with 3 golden tickets, 175 red actions, 47 detections fired and 91% of red actions detected
1 / 16

An elegant answer to a question that sounds simple and is not: how do you know whether your automated defender is any good?

The Question That Started It

Grace had spent four years building and running the purple team at Meta, where a typical engagement meant writing TTPs, running them, evaluating detection fidelity, finding gaps and tuning detections. Every step needed engineers, every engagement took weeks, and when it ended the process stopped until someone kicked off the next one.

Slide: From People to Agents contrasts the five-step human purple team loop with the Ares loop, which collapses generation and execution into one adaptive loop where every result feeds the next move

Agents change the economics. Ares collapses “write the TTP” and “run it live” into a single adaptive loop: it takes an objective and decides its next step from what it sees. Nobody predefines the attack path, and the loop can keep running.

Then Wendiggensen asked the obvious question — how do you know if the blue team is actually any good? The honest answer was a list of expected outcomes written down in advance. But the red agents are LLM-driven and non-deterministic, so they will not walk the same path twice, and a static checklist stops describing what actually happened.

The Adversary Gap

Slide: Adversary Gap notes there is enough data to test but not to train, with a matrix comparing defensive benchmarks on real queryable telemetry, investigation via tools, and an unscripted adversary

Models coming out of the labs are visibly better at offense than defense, and the reason is verification. Offense is easy to grade: hide a flag, see if the agent returns it. Defense needs judgment — not did you find something but did you find the right thing, in the right order, with the right context and the right remediation.

Defensive benchmarks have moved from knowledge questions to agentic investigation, which is closer to reality. But the attacker is still designed by a human and the ground truth is still curated by a human. Human attention is the bottleneck, so there is enough data to test on and nowhere near enough to train on.

Slide: The Defense Tradeoff plots variety against noise, showing scripted attacks as low-variety and low-noise, real SOC data as high-variety and high-noise, and generators sitting off the tradeoff line

That produces a tradeoff line. Scripted attacks are clean but starved for variety. Real SOC data has endless variety and impossible noise. Curating real data back down to clean spends the human time you were trying to save.

The move is to get off the line entirely: generate attacks automatically, and derive ground truth from them deterministically.

Turning Offense into the Yardstick

The trick is to use what models are already good at. A red team run by an LLM attacks shared infrastructure while a blue team defends it. When red finishes, its operation state and the infrastructure state are converted automatically into the scoring rubric for blue.

Blue is not scored against a checklist written in advance. It is scored against what red verifiably did — including attack paths nobody anticipated.

The Arena

The environment is a heavily modified fork of the Game of Active Directory project: two forests, three domains, most of the AD CS ESC catalogue, a dense mesh of abusable ACL edges, bidirectional MSSQL linked servers, and coercion paths that relay to AD CS. A variant generator regenerates domains, hosts, identities, credentials and directory structure along with every reference to them, so an LLM cannot simply recall an attack path from a public walkthrough it saw in training.

On the defensive side, Windows event channels — Security, Sysmon, PowerShell — stream into Loki, giving a reasonable approximation of what a SOC would see.

Slide: Inside the Ares Engine shows one orchestrator per side, with NATS handling durable queues and tool RPC, Redis holding shared team state, feeding seven red workers and four blue workers

Each side gets an orchestrator acting as team lead over specialist workers. Workers never talk to each other directly; they publish discoveries to a shared coordination layer — NATS for messaging, Redis for state — and the orchestrator plans from that.

The design decision worth stealing: model judgment is separated from system truth. The LLM decides what to do, but deterministic parsers extract verified discoveries from raw tool output. An agent claiming domain admin with nothing in shared state to back it up is hallucinating, and gets ignored.

Red: Seven Specialists

Slide: The Red Team as an autonomous multi-agent attacker with one orchestrator directing seven specialist workers covering recon, credential access, cracking, ACL abuse, privilege escalation, lateral movement and coercion

The first general-purpose agents burned through their context windows at speed. Give one agent a hundred-plus tools and a growing pile of hosts, hashes and credentials, and it loses the plot — forgets what it tried, repeats work, makes progressively worse decisions.

Narrowing each agent’s role and context fixed it. Seven specialists — recon, credential access, cracking, ACL abuse, privilege escalation, lateral movement, coercion — can consistently compromise most of the lab through different chains.

Blue: Four Workers, Telemetry Only

Slide: The Blue Team as an autonomous multi-agent investigator with an orchestrator directing four workers — triage, threat hunter, lateral analyst and escalation triage — under strict token, time and query budgets

Blue is simpler because it has no action component. It reads Loki, Windows event logs and Grafana alerts, works out what happened, and suggests remediation rather than applying it. It runs under deliberately realistic constraints: token limits, time pressure, query budgets against its data sources.

Scoring has three parts, all derived automatically:

  • IOC identification — the right indicators and the right MITRE techniques.
  • Evidence quality — completeness, how well findings tie to actual events, and how far up the pyramid of pain the reasoning gets.
  • Completeness — primarily the timeline: initial compromise, escalation, lateral movement, in the right order.

How Blue Broke

Slide: How Blue Team Breaks Immediately lists context overflow from Loki queries, exiting on connecting activity, reasoning taking too long while the attack continues, and exiting prematurely after the first signs of compromise

The first version did not perform badly — it broke. Two or three broad Windows event log queries can run 200,000 tokens, so it overflowed its context window and got lost before it aggregated any evidence.

The failures after that were more interesting. Reasoning is great for quality and terrible for pace: a frontier model taking two to three minutes to return means the attack has moved on by the time the answer lands, and blue plays permanent catch-up. It also quit early — find one compromised account, declare victory, never ask what happened next.

Making the Experiment Reproducible

Slide: Improvement Requires Iteration describes a replay engine that re-runs any recorded attack against the blue team, with a diagram cycling from red team op state through a replay data store and replay box to investigate, score and improve

Non-determinism gives variety and takes away measurement. Change something on blue, watch the score move, and you cannot tell whether the defense improved or the next attack was simply easier.

The replay engine resolves it. At the end of an operation, red’s state and the infrastructure state go into a data repository. Replaying spins up a box with feature parity to the original network and streams the logs into the right databases at the original tempo. To the blue agent it is indistinguishable from the live network — so every re-score isolates a single variable of change. It is also far cheaper, since networks are expensive to run and replays are not.

Self-Improvement

Slide: Smart Children lists prompt-level and code-level self-improvements alongside four bar charts showing investigation score, timeline accuracy, technique coverage and IOC detection rising across optimization cycles

After each suite of runs the blue team saw its score, reasoned backwards through where it went wrong, and rewrote its own system prompts and the code of its own tools. Something like a genetic algorithm with intelligent design.

Six cycles:

Metric Train Hold-out
Overall investigation score 55% → 75% 54% → 72%
Timeline accuracy 17% → 72% 20% → 68%
Technique coverage 42% → 58% 25% → 62%
IOC detection 65% → 73% 45% → 65%

The changes it made to itself are the interesting part. At prompt level it stopped playing catch-up: on every call it now asks a reasoning model to hypothesise where the attacker will go next, so being behind matters less. That single change drove the timeline jump.

At code level it rewrote the Windows event log extractor — eight times — after reasoning that it was drowning in information it did not need. It formed a hypothesis about what to extract, tested it on the next cycle, compared scores. It also rewrote its own query engine for speed so it could work in parallel with the unfolding attack.

The hold-out numbers matter most: attacks in the held-out set looked substantially different from the training set, and performance still generalised.

Slide: Tokenomics shows prompt and code-level cost reductions with a bar chart of cost per run in USD falling across optimization cycles

Cost was optimised in the same loop, falling about 25% from an average around $2.13 per investigation. At prompt level it defined a clearer finish line — before, like any obsessive analyst, it would keep chewing on an investigation long after there was anything left to find. At code level a better context manager stopped it refetching evidence it already had.

Why It Scales

Slide: Why This Scales shows 240,000 distinct trajectories built from a baseline multiplied by 300 realizations, 20 softmax variants, 5 failure variants and 8-way fan-out

The open-sourced lab can produce roughly 240,000 distinct trajectories: about 300 routes in, times 20 from forcing the agent off its preferred distribution, times 5 from including failures, times 8 from fan-out.

Failures matter more than they sound. Not every attacker is as competent as benchmarks assume — plenty get a foothold and go no further — so training data needs attacks that got caught by defense in depth, not just attacks that succeeded.

And because it is combinatorial, adding a couple of hosts or vulnerabilities unlocks thousands of new attacks.

The Demo

Slide: Replay UI showing a completed 34-minute operation across 3 domains and 2 forest roots with 3 golden tickets, 175 red actions, 47 detections fired, 91 percent of red actions detected and $8.41 agent cost

Red starts with five IP addresses. It finds its first credential sitting in a user’s description field, moves through roasting, credential dumping and