Adversarial Attack Against Voice Assistants

Project Objectives

Driven by advanced speech recognition technologies, voice assistant systems have been widely integrated into smart and IoT devices (e.g., Google Home, Amazon Echo). Although these systems provide great convenience, they also bring growing concerns about their security and robustness. The underlying machine learning models (e.g., for speech and speaker recognition) adopted by the state-of-the-art voice assistant systems, are inherently vulnerable to well-crafted audio adversarial perturbations, which causes misclassification while being imperceptible to human listeners. In this project, we aim to study such security vulnerabilities of voice assistant systems by developing effective and practical adversarial attacks.

Illustration of the static-speech attaack scenario and the proposed synchronization-free adversarial attack.

Technology Rationale

  1. Synchronization-free attack. Existing audio adversarial attacks assume prior knowledge of the entire speech input (e.g., voice commands) to generate the audio adversarial perturbations, which is hard to achieve since the adversary cannot anticipate what the victim will say. These attacks also require a precise alignment between the speech input and the generated adversarial perturbation. To address these challenges, we develop a systematic approach to generate audio adversarial perturbations that can alter the speech/speaker recognition results on streaming audio (e.g., live human speech) in a synchronization-free manner.

  2. Input-agnostic universal attack. Most audio adversarial attacks require to generate the perturbations based on each individual audio input, which would cost considerable time training perturbations for each individual voice input and thus make real-time attacks impossible. To develop such a practical attack, we propose to craft audio-agnostic universal perturbations which can be added into any audio input to spoof the embedded deep learning model, making it output the adversary-desired results.

  3. Over-the-air attack: Existing audio adversarial attacks mainly focus on digital scenarios, in which the generated adversarial example is directly fed into the speaker recognition system without being played through a loudspeaker. In this project, we explore the possibility of conducting over-the-air adversarial attacks in practical scenarios, in which the adversarial examples are played through a loudspeaker to compromise voice assistant systems. 

Technical Approach

  1. Synchronization-free attack. We design a series of techniques to release the requirement of synchronizing the audio input and the perturbation. We propose to add a sub-second audio adversarial perturbation and adopt a gradient-based adversarial machine learning algorithm to maximize the expected output probability of the target class over different delay conditions. This process enables the adversarial perturbation to be added at any timestamp of the audio input while maintaining effective. 

  2. Input-agnostic universal attack. We propose to build a universal perturbation that can spoof a state-of-the-art speaker recognition system, X-vector. The universal perturbation can be directly applied to arbitrary speakers’ any utterance, making the X-vector model output the adversary desired speaker label. To overcome the issue of varying utterance length, we dynamically construct the universal perturbation based on the length of the input utterance, by repeating a short-length adversarial perturbation. 

  3. Over-the-air attack: The digital adversarial perturbation would be most likely to lose its effectiveness during the over-the-air process. To address this challenge, we model the sound distortions during audio propagation using room impulse response (RIR), which characterizes the preposition of acoustic signals propagating through different paths (i.e., direct path and other reflected paths) with various attenuations and delays. We develop a gradient-based optimization method by integrating the RIR in the adversarial perturbation generation process.  

Project Status

This project has led to papers in AAAI 2021, ACM CCS 2020, ACM HotMobile 2020, ICASSP 2020. We have reported the synchronization-free, input-agnostic universal, and over-the-air attacks against both speech and speaker recognition systems. Figure 3 illustrates adding a sub-second perturbation to the streaming audio to attack the voice assistant system. Figure 4 shows the attack success rates against both speaker and speech recognition systems with different perturbation duration. 

Figures illustrating adversarial perturbation


Yi Xie, Zhuohang Li, Cong Shi, Jian Liu, Yingying Chen, Bo Yuan. Enabling Fast and Universal Audio Adversarial Attack Using Generative Model. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2021.

Zhuohang Li, Yi Wu, Jian Liu, Yingying Chen, Bo Yuan. AdvPulse: Universal, Synchronization-free, and Targeted Audio Adversarial Attacks via Sub-second Perturbations. In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security (ACM CCS), pp. 1121-1134, 2020.

Yi Xie, Cong Shi, Zhuohang Li, Jian Liu, Yingying Chen, Bo Yuan. Real-time, Universal, and Robust Adversarial Attacks Against Speaker Recognition Systems. In Proceedings of the International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 1738-1742, 2020.

Zhuohang Li, Cong Shi, Yi Xie, Jian Liu, Bo Yuan, Yingying Chen. Practical Adversarial Attacks Against Speaker Recognition Systems. In Proceedings of the 21st International Workshop on Mobile Computing Systems and Applications (ACM HotMobile), pp. 9-14, 2020.