FlagThis

Abstract

Adversarial examples reveal vulnerabilities in Vision-Language Pre-training (VLP) models and provide insights for improving robustness. A key property of adversarial examples is cross-model transferability, which enables transferbased black-box attacks. However, existing transfer-based attacks often rely heavily on the surrogate model, causing cross-model performance drops. One key reason is that adversarial optimization may follow surrogate model responses more than input semantics, making the update direction effective on the surrogate but less transferable to unseen targets. We call this dependency surrogate-specific bias. Motivated by this observation, DeBiasAttack improves cross-model transferability by correcting this bias in adversarial optimization directions. It maintains two perturbation branches. The main branch optimizes a perturbation on the original image and obtains the main adversarial gradient used to disrupt image-text alignment. The reference branch optimizes a perturbation on a weak-semantic image constructed from the dataset mean image by adding small Gaussian noise that is resampled at each iteration. Because this weak-semantic image contains little clear visual content, its optimization reflects surrogate responses more than image semantics, and its reference adversarial gradient estimates surrogate-specific bias. DeBiasAttack removes the aligned projection of the main gradient on the reference gradient before updating the adversarial image. It then performs context-aware text substitution using the updated adversarial image, yielding adversarial image-text pairs that transfer better across models. DeBias-Attack is the first transfer-based VLP attack that corrects surrogate-specific bias through gradient correction. Experiments show strong performance across VLP models, downstream tasks, and both open-source and closedsource multimodal large language models.