Programmable Backdoors in Vision-Language Models VLMs

This research identifies a critical vulnerability in Vision-Language Models (VLMs) enabling "programmable" backdoors through an any-to-any caption-control framework. Unlike static backdoors, attackers utilize a heuristic poisoning strategy to instill a general "trigger-as-instruction" rule, decoupling poisoning from target selection. By employing feature-space trigger steganography via norm-controlled perturbations, adversaries can synthesize stealthy visual triggers at inference time to force the model to generate arbitrary, previously unseen target captions. This mechanism bypasses classical defenses—including pruning and fine-tuning—while maintaining the model's original utility on clean datasets, allowing for covert and arbitrary semantic control of multimodal outputs.

• Threat Model & Vulnerability Overview

  • Shift from static trigger-target pairs to a dynamic, programmable output engine.
  • Decoupling of the training-time poisoning phase from the inference-time target selection phase.
  • Ability to synthesize triggers for previously unseen captions without requiring model retraining.

• Attack Mechanics & Exploitation Vector

  • Heuristic Poisoning Strategy: Instills a generalized "trigger-as-instruction" rule rather than simple pattern memorization.
  • Feature-Space Steganography: Maps arbitrary target captions to stealthy visual perturbations or non-semantic patches.
  • Any-to-Any Framework: Enables an end-to-end pipeline for dynamic target selection via visual steganography.

• Systemic & Security Impact

  • Arbitrary Semantic Control: Moves the threat from simple misclassification to the ability to dictate any model output.
  • High Attack Success Rate (ASR): Demonstrates high effectiveness in generating semantically aligned outputs for target captions unseen during poisoning.
  • Utility Preservation: Maintains baseline model performance on clean tasks, making the backdoor nearly impossible to detect through standard utility testing.

• Defense Evasion & Countermeasures

  • Classical Defense Failure: Proven effective against traditional backdoor mitigations, including pruning, fine-tuning, and trigger detection.
  • Visual Stealth: Use of norm-controlled perturbations ensures triggers are visually invisible or produce minimal impact on image quality.
  • Alignment Gaps: Exposes the inadequacy of current AI alignment and sanitization techniques against feature-space poisoning.

• Conclusion

  • Elevates the VLM risk profile from static vulnerabilities to a programmable exploitation primitive.
  • Necessitates a shift in defense toward detecting feature-space anomalies rather than specific visual patterns.
  • Increases the danger of deploying open-weight LMMs without verified provenance and rigorous adversarial auditing.