Abstract

Safety fine-tuning of language models typically requires a curated adversarial dataset. We take a different approach: score each candidate prompts difficulty by how often the target models own rollouts are judged harmful, then fine-tune on the hardest prompts paired with the models own non-jailbroken rollouts. On Llama-3-8B-Instruct and Llama-3.2-3B-Instruct, this approach cuts the WildJailbreak attack success rate from 11 _._ 5% and 20 _._ 1% down to 13%, but pushes refusal on jailbreak-shaped benign prompts from 1422% to 7494%. Interleaving the same hard prompts 1:1 with adversarially-framed benign prompts (prompts that look like jailbreaks but have benign intent) cuts that refusal back down to 3051% on 8B and 5272% on 3B, at a cost of 26 percentage points of attack success rate. Within the mixed regime, training on the hardest half of the eligible pool rather than a random half cuts the remaining ASR by 3550% (about 3 percentage points) on both models.