FlagThis

Abstract

Inference-time safety methods for large language models have proliferated rapidly, yet no systematic comparison exists. We evaluate five defense paradigms (no defense, static steering, CAST, AlphaSteer, probe-gated) across seven instructiontuned models (Mistral-7B, Gemma-2-9B, Gemma-3-12B, Gemma-4-31B, Qwen2.5-7B, Qwen3-8B, Llama-3.1-8B; 731B parameters) and five attack types (GCG, AutoDAN, DeepInception, prefilling, intent laundering). Our central finding is that prompt-time activation defenses are structurally blind to prefilling attacks : AlphaSteer achieves 0% ASR on GCG, AutoDAN, and intent laundering, __ 2.5% on DeepInception, but 50% on prefilling under the current environment (82% under the paper environment, App. P/Q); no other prompt-time paradigm reduces prefilling ASR below 28% (paper env; current-env baselines exist only for Mistral/Llama). We derive a corollary generalising this result: any defense gating intervention on activation alignment with a benign reference, cone, subspace, or nullspace projection, at any single layer is structurally blind to attacks that craft activations to lie inside that reference, whether the gate is checked once at prompt time or per generated token ; applicable to AlphaSteer, CAST, prompttime probe-gated steering, and any defense sharing this structural shape. We introduce response-time probing as the corollarys constructive contrapositive: a linear probe on the models hidden state at the first few generated tokens (meanpooled over _N_ = 5 in headline results, _N_ = 1 sufficient; App. V), trainingdistribution AUROC 0.971.00 across all seven models. Combined with a halt intervention, it reduces prefilling ASR _on the canonical prefilling-attack template family_ to 0/40 on all seven models at _n_ = 40 (Wilson 95% CI [0%, 9%]); Llama scaled to _n_ = 200 confirms 01/200 across runs, Gemma-4-31B-it LLM judge: 0/200; with 0% benign FPR, outperforming Llama Guard 3 (0% detection). Crosstemplate generalisation is contingent on probe depth (4/7 layer __ 2 models fail; 2/3 layer __ 4 models generalise); we therefore frame the central claim as an activation-level defense for the canonical template family rather than a modelagnostic detector. On Llama, the dual-probe system reaches keyword-judge DSR 0.990 at _n_ = 200 vs 0.698 for AlphaSteer; the Gemma-4-31B-it LLM judge confirms prefilling 0/200 specifically (aggregate LLM-judge DSR is 0.964 due to intent-laundering disagreement, App. U). _n_ = 200 unified: Mistral 0.940, Llama 0.990 (Table 5). Composing the response-halt with AlphaSteers null-space steering, the architectural composition the corollary motivates, yields DSR 0.983 on Mistral and 0.994 on Llama at _n_ = 40, dominating both individual paradigms; both unified DSRs reach 1.000 under Gemma-4-31B-it LLM judge (App. U). The orthogonal split (response-halt catches prefilling, AlphaSteer catches semantic attacks) is the corollarys prescribed architecture. Adaptive robustness is asymmetric: on Llama the probe catches 80% of canonical prefilling attacks but only 65% of probe-aware adversarial variants, with residual ASR 30%; Mistrals prefilling-specific probe does not fire on adaptive variants . Qwen-2.5 at either tested layer is an unexplained counterexample to the layer-depth split. Paper-shipped probe weights on Mistral and Llama drift outside __ 2pp under current library versions; fresh retraining recovers paper numbers for both; retraining script released. The response probe operates in a subspace _orthogonal_ to the refusal steering vector ( _|_ cos _| <_ 0 _._ 04 on 6/7 models), detecting information activation steering cannot access. MMLU fails to capture steerings utility cost (manifests as behavioral hedging, not factual loss); probe training on narrow distributions causes 80100% FPR that diverse negative sets eliminate. Code, attacks, per-sample results, and LLM-judge prompt released; we collate eight diagnostic protocols for activation-level safety method evaluation (App. T).