FlagThis

Abstract

Autonomous AI agents have driven the transition from conversation to task execution. This shifts security failures from textual deception to system compromise. Although security evaluation is crucial for proactive risk prevention, prior work is constrained by fundamental bottlenecks, including fragmented risk coverage, static or low-fidelity execution environments, and single-dimensional and coarse-grained assessment metrics. To address these challenges, we propose AgentCanary , a comprehensive security evaluation framework for autonomous AI agents. AgentCanary provides a systematic solution along three contributions. First , comprehensive risk coverage : we introduce an orthogonal Entry Impact risk taxonomy that decouples how adversarial influence enters the agent from what harm it ultimately causes, and instantiate it as a scenario-aligned task suite spanning realistic deployment workflows such as web browsing, email, instant messaging, calendar, financial transactions, and third-party skills. Second , a high-fidelity real executable environment : rather than static Q&A or mocked tool responses, agents interact with real tools against dynamically provisioned task artifacts (e.g., inboxes and web pages), with persistent state across multi-step interactions that naturally supports long-horizon attack evaluation. Third , trajectory-grounded multi-dimensional evaluation : evaluation consumes the full agent trajectory rather than the reply text or a single tool call, enabling decomposed scoring along three orthogonal dimensions, Outcome Safety , Security Awareness , and Task Utility , which jointly characterize the trade-offs among safety, vigilance, and usability. We evaluate a broad set of frontier models on AgentCanary against multiple established adversarial attack methods, including single-round prompt-injection templates, iterative red-teaming, and scenario-specific long-horizon attack chains, across three agent frameworks. The results reveal that current agents often fail to recognize the attacks they face, particularly under compromised skills, persistent state, and long-horizon execution attacks, and provide a systematic baseline for developing more reliable and secure agent systems.