FlagThis

Abstract

Fine-tuning is the dominant way to specialize large language models (LLMs), yet a malicious data provider can quietly embed harmful behavior into a downstream fine-tuning corpus so that the resulting model retains its professional skill (e.g., code generation) while violating human values on demand. Existing safety-realignment defenses recover alignment only under restrictive assumptions, and we identify three vulnerabilities that undermine them in practice: they frequently damage the models specialized skill during repair; their effectiveness collapses when the defender cannot observe the attackers prompt template; and even a successfully realigned model can be re-jailbroken by simply switching the system prompt. We propose Routing-based On-Policy Distillation (ROPD), a realignment framework that models the difference between the aligned and the compromised output probability distributions rather than aligning to a specific prompt template. ROPD routes every realignment token to one of two frozen teachers—an original-model safety teacher that supplies a largely template-independent refusal prior, and a fine-tuned task teacher that preserves downstream ability—and matches the student to the routed teacher with a top-K KL objective.