Trusted Enough to Run: Breaking AI Agents in Official Workflows - Elad Meged | Black Hat USA 2026

Elad Meged (Novee Security) — 40 min

‹ ›

1 / 10

Meged builds AI agents that do penetration testing, and ended up pen-testing AI agents instead. Rather than pick on a random company’s homegrown agent, he went after the vendors who build the agents — so nobody could claim they simply did not know how these things work.

The Attack Surface Is the Design

Agents run CI/CD, answer Slack, triage tickets and watch infrastructure overnight. Nobody reviews their inputs, nobody reviews their outputs, nobody approves each action.

Untrusted text arrives from ordinary places — issues, pull requests, Slack messages, tickets, email — and on the other side sit real actions: shell commands, file access, API calls. Prompt injection is a property of that interface, not a bug in it. You cannot stop untrusted text reaching the model; that is the delivery mechanism.

Anthropic’s own security documentation for the Claude Code action says as much: it warns about prompt injection, notes the sanitizer can be bypassed, and calls out hidden instructions and invisible characters. Which raises the real question — if the vendor says the input layer is not the security boundary, where is it?

An agent is a model plus a harness. Claude is the model; Claude Code is the harness. The harness is code, and code has vulnerabilities and thousands of trust decisions — assumptions about what is safe, invisible to whoever deploys it. Deploy the agent and you inherit every one.

Anthropic, Round One: The Validator Read a String

CI/CD is the natural hunting ground: a trigger fires, the agent runs, nobody clicks approve.

Claude Code Action has two modes. In agent mode the user supplies the prompt and controls the tools — their problem. In tag mode you mention @claude on an issue and the vendor decides what is safe, shipping defaults on your behalf. That is the interesting one.

Those defaults have to pre-approve tools, because there is no human to approve each call. File edit, read and write. And some bash, for git operations — git add, git commit, git push. Anthropic knew exactly how that sounds and built a dedicated guardrail pipeline: 23 checks including command injection detection, shell metacharacter guards, Unicode substitution and entropy checks.

The checks are thorough about the things they check. Backticks, $(), ${}, pipes, redirections, escaped operators. All the ways you would normally smuggle a command through a string.

So Meged stopped looking for metacharacters and read the man page. git push --receive-pack=<program> takes a path to the git-receive-pack program on the remote end — useful when it is not on the remote’s $PATH. Push to . and there is no remote end: git resolves it locally and starts the receiving program itself, through sh -c.

Nothing about that string looks dangerous. There is no substitution, no separator, no metacharacter. The validator sees a harmless argument. Git sees a path to a program and executes it.

Demonstrated against Anthropic’s own repository, on default config, with no modifications: open an issue carrying an injection payload, tag @claude, and GITHUB_TOKEN, ANTHROPIC_API_KEY and every workflow secret leave the runner.

Round Two: tac Is cat Backwards

The fix removed arbitrary bash. Bash is now explicitly allowlisted, most bash tools are gone, git push via shell is gone. What survives is file tools — read, glob, grep — scoped to the workspace, with no network.

Then the next assumption, baked into Claude Code itself rather than any config: a list of read-only commands that are auto-approved with no prompt at all. Over 40 of them — cat, head, tail, tac, rev, fold, expand, unexpand.

A second list restricts read-only commands to the workspace path. It is shorter. cat is on it. tac — cat in reverse — is not.

cat /etc/hosts is blocked. tac /etc/hosts is auto-approved, unprompted, with no path check. Same file, same bytes, backwards. Arbitrary file read on the runner.

Reported, and the response was that it falls outside the current threat model and was never intended as a security barrier — but that privilege escalation or data exfiltration would be interesting.

Fine. /proc/self/environ holds the secrets. The path itself needs to survive validation, which it does with a quoting trick that collapses to an empty string in bash context. Then exfiltration, using Anthropic’s own feature: the display-report option writes every tool result into the GitHub Actions step summary. The secrets published themselves.

That got patched — the output channel closed.

Round Three: The Download Counter

No bash. No output channel. Arbitrary read still working, since it was never considered a barrier. Nothing to read the file out with.

Except WebFetch, a default Claude Code tool nobody has to enable. Fetching an attacker-controlled URL prompts for permission, as it should. Fetching certain other hosts returns 200 immediately, with no prompt.

The first thing checkPermissions() does is wave through pre-approved hosts — about 98 of them, covering docs, frameworks, cloud providers, databases and registries, so agents can fetch documentation autonomously. Not subject to allowlist restrictions.

Uploading is no good; the pre-approved hosts do not accept the data. Redirects are constrained to the same host. So the secret has to end up in somebody else’s logs, on a host anyone can publish to.

Hugging Face is on the list, and anyone can sign up. Its documentation notes that any HTTP request to a model’s config.json — including GET and HEAD — increments the download counter. A public, attacker-readable, write-only-by-fetch integer.

So: 64 repositories, one per possible character of the secret. Prompt the agent to fetch the repository corresponding to character N. Read the counters afterwards; exactly one has moved. Repeat per position until the whole secret is spelled out.

Three fronts, three fixes, each one relocating the boundary to the next hidden assumption — starting from a version string and ending at a download counter.

Google: Ask Your Parent Process

run-gemini-cli is Google’s action for running Gemini CLI in CI/CD. It runs in YOLO mode, auto-approving every tool call, because automation has no human in the loop. Google’s own Gemini CLI repository uses it to triage issues, firing on every new issue any GitHub user can open.

Assumption one: keep secrets away from the agent. Implemented with two processes — the parent holds the environment, the child gets a sanitized copy with GITHUB_TOKEN and the Gemini key stripped. Run env in the child and it is clean.

But both processes share a PID namespace, with no hidepid. /proc/<parent-pid>/environ still holds everything. The secrets were never moved, only hidden from one view.

Assumption two: the allowlist stops you running the command that would read it. The config permits echo and a GitHub issue-view command. But the check runs at registration time — it verifies the command is non-empty and the path exists — and nothing constrains what actually runs at runtime.

One issue, zero privileges, and you have the CI secrets, write access, and a push to a repository with over 100,000 stars sitting upstream of a great many installs. Meged put it around a CVSS 10, since it affects every repository using the action.

Google did not just patch it. They redesigned the trust model of Gemini CLI across their organisation.

OpenAI: Writing the Instructions the Next Agent Reads

Codex Action wraps codex exec. OpenAI’s own issue-duplicator workflow ran one job with one checkout and two Codex phases sharing the same workspace.

The first phase classifies the issue body. It is attacker-controlled by definition, so it is given no tools and told to be strict. Then a deterministic check reads that classification — no attacker control there — and the second phase applies the label, holding a GitHub token to do it.

The design looks careful. The gap is the workspace they share. AGENTS.md is loaded as instructions on every run — code-adjacent metadata in the same class as .git hooks, which Codex’s own sandbox notes explicitly protect because a bad actor writing a pre-commit hook escalates on the user’s next commit.

So the first agent, the one with no tools and no trust, writes AGENTS.md. The deterministic check passes, because it is checking the classification and the classification is fine. The second agent loads the workspace, reads attacker-written instructions as its own, and it has the token.

The developer validated the output. The attacker wrote the instructions.

The fix split it into two jobs with separate checkouts, removed the inherited workspace, made the sandbox read-only, and added AGENTS.md to the protected set.

How to Audit the Next One

Three vendors, three different failures, one shape. The closing method is the most portable part of the talk:

1. List what it calls safe. Read-only, restricted, sanitized, pre-approved — every one of those is a label somebody wrote.
2. Find who acts on the label. Which component consumes that decision, and with how much authority?
3. Look for the handoff. Decided in one place, consumed in another with more power. That is the bug.
4. Read the defaults, not the docs. The assumption is not in your config. It is in their binary.

When you deploy an agent you embed somebody else’s codebase into your pipeline, and you inherit their assumptions along with their vulnerabilities.

Back to Black Hat USA 2026 — All Talks