Programmable Backdoors in Vision-Language Models VLMs
This research identifies a critical vulnerability in Vision-Language Models (VLMs) enabling "programmable" backdoors through an any-to-any caption-control framework. Unlike static backdoors, attackers utilize a heuristic poisoning strategy to instill a general "trigger-as-instruction" rule, decoupling poisoning from target selection. By employing feature-space trigger steganography via norm-controlled perturbations, adversaries can synthesize stealthy visual triggers at inference time to force the model to generate arbitrary, previously unseen target captions. This mechanism bypasses classical defenses—including pruning and fine-tuning—while maintaining the model's original utility on clean datasets, allowing for covert and arbitrary semantic control of multimodal outputs.