FlagThis

Abstract

As LLM agents take on offensive security work, a single out-of-scope tool call can breach a clients engagement boundary, disrupt production, or void a bug-bounty finding. Unlike a fixed safety policy, the boundary that matters is declared in the users request and must be _inferred from intent_ . That challenge is sharpened by the adversarial nature of offensive security: the same tool call is in or out of scope depending not on the action itself but on the target it touches and the context in which it runs, which no fixed policy can enumerate in advance. We study **pre-execution gating** : a cheap, trusted LLM judge inspects each call proposed by a strong, swappable agent, and accepts or rejects it _before_ it runs. We introduce **ScopeJudge** , a benchmark of 4,897 tool calls (7 _._ 7% scope violations) from agent trajectories on tasks engineered to tempt agents out of scope and labeled at the call level by professional penetration testers, with substantial inter-grader agreement (Fleiss __ = 0 _._ 64) that sets an expert agreement reference point of F1 = 0 _._ 78. We evaluate eight judge models under five _transcript strategies_ , varying how much context the judge sees, from the static policy alone to the full raw transcript, and chart the resulting costaccuracy Pareto frontier. We find that a static policy is structurally insufficient for scope enforcement: blind to the users request, judge recall collapses to near zero, confirming that scope lives in the request and that request-conditioned monitoring is necessary. We find the strongest judges are _open-weight_ : GLM-5.2 reaches F1 = 0 _._ 66, the highest of any judge we test, beating the best proprietary judge (0 _._ 60) at roughly one-third the per-call cost. Because a missed violation costs more than a spurious rejection, we report precision, recall, and F1 separately and recommend two operating points: a cost-sensitive configuration and a recall-first one for high-stakes deployments. We release the **ScopeJudge** dataset to support real-time monitoring and scalable oversight of autonomous security agents.