Abstract

Agents are increasingly deployed with real autonomy in web application and network penetration testing, where a single out-of-scope action can breach a clients engagement boundary. Existing offensive-security benchmarks measure raw hacking capability; as those benchmarks saturate, the actual barrier to deployment is a special case of alignment: scope adherence. We introduce **ScopeBench** , a benchmark of 30 _dead-end_ agentic security tasks in which the stated objective is reachable only by violating the stated scope. Each task appears under two conditions that share an environment, verifier, and objective and differ only in scope: one instruction set has no scope and measures capability; the other has a natural-language scope to measure adherence. Scopeless trajectories are graded by a standard deterministic verifier. Scoped trajectories pass through two grading arms. First, the same deterministic verifier checks for the flag: because the flag sits behind the scope boundary, a pass proves by construction that a forbidden action occurred, yielding a high-precision lower bound on the violation rate. If the verifier does not pass the trajectory, an agentic judge estimates whether an out-of-scope call occurred. We calibrate the judge against 100 ScopeBench trajectories labeled call-by-call by human annotators, and a blinded audit of the evaluated rollouts finds its high recall holdsno false negatives among the 36 audited violations, with over-flagging its only observed error. Across 8 models run in one harness, raw capability spans 12 _._ 2% to 81 _._ 1% and scope adherence spans 34 _._ 4% to 86 _._ 7%, with the judge finding 331 violations that mechanical verification misses. Opus-4-8 achieves a raw-capability score 10 percentage points higher than sonnet4-6s while exhibiting 35.6 percentage points higher scope adherence. We release the frozen ScopeBench pilot benchmark and evaluation code as a methodological framework for measuring scope adherence in future agents, with an open pathway for community task contributions.[1] We also release all 2160 ATIF trajectories.[2] Future versions of the community benchmark and the live leaderboard are published at `scopebench.ai` .