Abstract

Security benchmarks for LLM-based agents often report the attack success rate (ASR) as a measure of model robustness and use these scores to compare different models and defense mechanisms, assuming that they describe the security of the agent. Prior work has shown that tool metadata can influence agent behavior. In this paper, we explore whether it also influences the benchmarks measurement. We ask how much a measured ASR reflects the robustness of the agent, and how much it reflects the way the benchmark happens to represent the threat.