FlagThis

Abstract

Existing defense methods have shown robust performance against harmful fine-tuning when downstream tasks are explicitly mixed with harmful content. However, intentionally crafted training data may instead entangle harmful instructions with benign downstream samples. To study this setting, we propose Embedded Attack, a fine-tuning threat formulation in which harmful question-answer pairs are embedded inside benign training samples. A case study with representative guardrail models shows that this construction can substantially reduce example-level detection, motivating the need for fine-tuning-stage defenses that operate beyond coarse data filtering. To mitigate this challenge, we propose Dual-Reference SFT (DR-SFT), which extends the objective of Direct Preference Optimization (DPO) to supervised fine-tuning (SFT), thereby enabling token-level contrastive regularization and gradient updates to defend against harmful finetuning attacks.