AgentBreaker: Frontier-Grade AI Agent Exploitation with a Fine-Tuned Open Model - NVIDIA | Black Hat USA 2026

Bar Lanyado (Senior AI Security Researcher) & Eliya Cohen (Senior Data Scientist), NVIDIA — 35 min

‹ ›

1 / 15

Two talks in one: how to scan an AI agent properly, and how to train a small open model to do that job as well as a frontier model for roughly a hundredth of the price.

Your Scanner Is Looking at the Wrong Thing

Run a standard LLM scanner against your agent and it will report nothing serious. It scans the model well — jailbreaks, safety issues, hallucination, prompt injection — but the model is one component. An agent is a model plus tools plus a harness, and the scanner never looks at the environment.

Scanning an agent properly looks more like scanning a web application: map the attack surface, work out what sits behind each capability, then find vulnerabilities and goals worth chasing.

Except recon is far easier than on a web app. You do not fuzz input fields or hunt subdomains. You ask. What are you? What tools do you have? What are their signatures, restrictions, outputs? Lanyado’s claim is that this is not limited to demo agents — major frontier assistants handed over their full attack surface to one polite prompt.

That map goes to an attacker model, which proposes weaknesses and a strategy, generates an exploit, fires it at the victim, and passes the response to a judge. Success moves on to the next tool. Failure feeds the whole conversation back to the attacker to improve the prompt. In the demo, a code review agent gives up the contents of a .env file on the third attempt.

Making It Survive Contact

Getting from a working loop to something usable took several corrections, each a good lesson in its own right.

Reset the victim’s context between stages. Two reasons. There is a model behind the victim, and if it sees recon followed by exploitation it can add one and one. And once an agent has refused you it enters a refusal groove that is very hard to climb out of — better to open a fresh session. The attacker gets smarter while the victim keeps waking up with a blank page.

Stop the attacker doing real damage. The first agent scanned had bash. The attacker got excited and tried to delete the filesystem. It was blocked, then succeeded, and the victim crashed. So the attacker was constrained to proof-of-concept payloads only.

Handle the attacker refusing. Frontier attacker models decline the work on alignment grounds, which crashes the scan. Jailbreaking your own tooling was not the goal, so those tools got skipped.

Fix the judge. Then a subtler failure: the attacker, rather than refusing, produced a payload that just printed “hello world”. The judge saw the requested output appear and scored it a successful exploit. The judge had to be taught to verify effect, not compliance.

Two metrics came out of it: exploit success rate (verified exploited tools over vulnerable tools) and attack success rate (successful payloads over all payloads sent), measured across 30 victims and 50 scenarios at two security levels.

The Cost Argument

An average scan costs about $0.69 on frontier models and about $0.007 on a 30B open model. That gap sounds like loose change until you multiply it. Gartner projects an average global Fortune 500 enterprise will run over 150,000 agents by 2028, up from fewer than 15 in 2025 — so a single pass over the estate is around $100,000 against around $1,050. And you do not scan once; you scan continuously.

Cost is only the first argument for small models. Frontier models also refuse more with each better-aligned release, which is a structural problem for offensive tooling. And many organisations will not send agent internals to a third party at all.

The catch: small models did not perform well enough out of the box.

Building a Training Set That Does Not Exist

The data you need is a full trajectory — recon, analysis, attack, adaptation, repeated until success. Thousands of them. No such dataset existed, and standing up thousands of real agents was not practical.

The first attempt was to attack a handful of real agents, collect successful trajectories, and have an LLM expand them. The expansions came back looking like the seeds. Useless.

The insight that fixed it: the attacker does not care whether the victim is real. It needs a plausible response. So they manufactured a fleet — 500 base profiles (IT, HR, finance, personal assistant), each with two to ten tools spanning read, update, execute and sensitive-data access, times three response styles (LLM-generated, informative, non-informative), times three security levels (none, baseline, hardened). That is 4,500 mock victims, each played by one of four open-weight models.

Then diversity selection: embed every victim agent, measure pairwise distance, and keep the subset with maximum coverage. A thousand survived.

Running the scanner against those in parallel produced roughly 5,000 training examples — generated end to end in-house, nothing scraped and nothing purchased.

Training

The base model was Nemotron-3-Nano-30B-A3B — a hybrid Mamba-2, attention and mixture-of-experts stack with 30B total parameters and roughly 3B active per token. Training used LoRA plus supervised fine-tuning, targeting the attention layers as the ones that carry context and token relationships, along with the MLP layers feeding them. The mixture-of-experts layers were left alone, since shifting those would need orders of magnitude more data.

Total weights touched: 0.02%. Total time: under an hour on a single DGX.

First result: 52.4% to 59.6%. Past Gemini’s neighbourhood, short of the leaders. A respectable place to stop, which they declined to do.

Closing the Gap

Three problems showed up in the data. The attacker used essentially one attack style. It sometimes stalled before adapting. And refusals were still polluting the training set.

Three fixes, each obvious in hindsight: use multiple attack styles rather than one; add a panel of open-source models that comment on the analysis before the next step — a “good enough?” gate on the plan; and filter refusals out of the data.

Final numbers across 30 victim agents:

Up 13.7 points from base, within two points of the best frontier model, at 1.4× the base model’s cost while the frontier options run 75 – 125× more expensive.

Plus the two things cost tables do not show: no refusals, and nothing leaves your network.

Takeaways

• Test agents like systems, not like models. The harness and the tools are where the vulnerabilities live, so the security harness has to cover the whole execution loop.
• Small models can master specialised tasks. The right data makes them experts — and the data can be manufactured.

Back to Black Hat USA 2026 — All Talks