FlagThis

Abstract

Production LLM agents such as Claude Code and Codex act through deployed interfaces over untrusted content, files, commands, and workspace state, so a safety failure here is a real action: a written file, exfiltrated data, or a triggered workflow. Red-teaming these agents must keep pace with every model and tool update, yet todays tools optimize judged attack success and preserve surface artifacts: benchmark scores, payloads, archives, strategies, or attack programs. These artifacts record where an attack landed, but not the enabling condition that made the agent trajectory unsafe, so they are hard to audit, patch against, or reuse after the setting changes. We study **autoresearch for production-agent red-teaming** , using one agentic research environment to automatically discover reusable vulnerability knowledge about another production-style agent. We present AHA, a falsifiable discovery loop: it commits to a vulnerability hypothesis, creates a falsifier, instantiates a scenario-valid attack, executes it in a sandboxed agent harness, reflects on the trajectory, and promotes confirmed findings by an evidence rule into a **Vulnerability Concept Graph (VCG)** . Each concept is an auditable unit linking an attacker-facing surface to an unsafe trajectory through a claim, enabling condition, falsifier, transfer prediction, and evidence. Across Claude Code and Codex on three scenarios spanning direct and indirect attacks, the discovered concepts share a core that recurs across victim models and agents, the frozen VCG is reusable with no further search, outperforming the strongest frozen discovery baseline by 14.2 percentage points under the same single-shot protocol, and the concepts transfer across scenarios and across direct/indirect attack channels. This makes the artifact directly useful for production triage: a safety team can inspect the enabling condition, patch the agent or workflow, rerun the concept as a check on the fix, and attach new internal concerns through the same build/import scenario workflows. As production agents proliferate, such a VCG turns one-off red-teaming into cumulative, auditable safety knowledge that compounds across models and products. Our code is available at https: //github.com/henrymao2004/Auto-research-red-teaming.