FlagThis

Abstract

As LLM agents begin to take real, irreversible actions (running shell commands, editing files, deploying code), the standard safety pattern is a human-in-the-loop approval gate: risky actions pause and wait for a person. We argue the gate is the easy part. The hard, unsolved part is the _judgment_ deciding **which** actions to stop and the field evaluates that judgment against two assumptions that are both false: that there is a ground-truth notion of risky, and that the human reviewer is a perfect, infinitely-available oracle. We show, on a hand-labeled set of 125 adversarially-weighted agent actions, that (i) reviewers only _moderately_ agree on what is risky (Fleiss __ = 0 _._ 52), so there is no single correct label; (ii) framing the guard as **selective classification under asymmetric cost** makes its operating limits measurable, and on hard inputs the guard cannot safely auto-decide; and (iii) when the reviewer is modeled as **endogenous** (fatiguing as escalation load grows), realized safety becomes an **inverted-U** in the escalation rate: _more human oversight can make a system less safe_ , and the safety-optimal guard escalates **below full escalation** (a middle escalation rate, not the maximum), a setting a load-aware policy also uses to resist a _flooding attack_ that rubber-stamps a malicious action past a fatigued reviewer. Framed this way, agent oversight is not only a classification problem (which action is risky?) but a resource-allocation one: human attention is finite, and the guards escalation policy spends it. **We claim none of these mechanisms as novel:** fatigue-aware learning-to-defer (FALCON [13]), cost-sensitive deferral under workload constraints (DeCCaF [14]), trajectory-level guarding, and fatigue/flooding attacks on human reviewers (security-operations alert fatigue [15]) are all prior art we cite. Our contribution is an **open-source agent-oversight system** that operationalizes and _measures_ these ideas together in the LLM-agent action-gating setting, turning is my guard good? from a guess into a curve. The inverted-U and the flooding attack are modeling results that motivate a human study.