FlagThis

Abstract

Safety evaluation of large language models (LLMs) is commonly performed by querying models with unsafe or jailbreak prompts and judging whether their outputs violate a safety policy. Although useful, output-level evaluation is expensive, sensitive to judge choice, and tied to fixed question banks. We propose SafeVec, a white-box evaluation procedure that measures safety from internal representations rather than generated outputs. Given a safety-aligned reference model, SafeVec extracts refusal directions and scores a target model according to how strongly its hidden states align with these directions under unsafe and jailbreak prompts. The resulting metric, RAS ( Refusal Alignment Score), maps representation-level refusal alignment to a calibrated 0-100 safety score. Across Llama, Gemma, and Qwen families, RAS separates aligned models from uncensored and abliterated variants, correlates with output-level attack success rate, and is substantially faster than judge-based evaluation. These results suggest that refusal alignment provides an efficient signal for white-box LLM safety evaluation.