FlagThis

Abstract

both cases, audio quality is treated as a secondary concern rather than an explicit optimization objective. While black-box attacks on ASR are well studied, methods that generate audio preserving both similarity to the original utterance and naturalness while inducing semantic divergence in the transcription remain limited. This gap motivates testing approaches that operate within a structured representation space rather than applying unconstrained perturbations to the raw audio signal. Automatic speech recognition (ASR) systems have achieved high accuracy with transformer-based models, enabling deployment in critical applications. However, they remain vulnerable to adversarial manipulation, particularly in black-box settings where attacks must preserve perceptual naturalness. This work introduces GATAS, a black-box testing approach that generates failure inducing inputs by operating in the phoneme-level latent space of a textto-speech model. Instead of perturbing waveforms directly, the approach interpolates latent representations to induce transcription errors while remaining within the manifold of natural speech. The attack is formulated as a multi-objective optimization problem balancing semantic divergence and perceptual quality. Our empirical evaluation against both white-box and black-box baselines shows that GATAS achieves a 98% success rate while producing lower distortion and higher perceptual quality, as confirmed by human studies. Despite operating without gradient access, GATAS remains competitive against white-box methods, highlighting that representation and perceptual alignment are more critical than access to model internals. Overall, our results demonstrate that untargeted latent-space optimization enables the efficient generation of realistic and effective test cases for ASR systems. To address this gap, this work proposes GATAS ( **G** ener **a** tive **T** esting of **A** utomatic Speech Recognition **S** ystems), a novel blackbox testing approach for ASR systems that generates natural-sounding audio while altering transcription semantics. GATAS leverages a text-to-speech (TTS) model, specifically StyleTTS2 [31], which represents speech as phoneme-level embedding vectors encoding acoustic properties. Instead of modifying the waveform directly, the approach interpolates between the ground-truth phoneme embeddings and a noise reference, with a per-phoneme scalar weight controlling the degree of perturbation. This shifts individual phoneme realizations while the TTS decoder ensures that the generated audio remains natural-sounding. GATAS formulates test generation as a multi-objective optimization problem that balances semantic divergence against perturbation imperceptibility, enabling the discovery of test cases that induce meaningful transcription changes while preserving audio quality.