AdVerb: Visually Guided Audio Dereverberation

ICCV 2023


Sanjoy Chowdhury1, Sreyan Ghosh1, Subhrajyoti Dasgupta2, Anton Ratnarajah1, Utkarsh Tyagi1, Dinesh Manocha1
1University of Maryland, College Park   2Mila and Université de Montréal
Paper arXiv Video Code

AdVerb leverages visual cues of the environment to estimate clean audio from reverberant audio.
E.g - Given a reverberant sound produced in a large hall, our model attempts to remove the reverb effect to predict the anechoic or clean audio.

Abstract

We present AdVerb, a novel audio-visual dereverberation framework that uses visual cues in addition to the reverberant sound to estimate clean audio. Although audio-only dereverberation is a well-studied problem, our approach incorporates the complementary visual modality to perform audio dereverberation. Given an image of the environment where the reverberated sound signal has been recorded, AdVerb employs a novel geometry-aware cross-modal transformer architecture that captures scene geometry and audio-visual cross-modal relationship to generate a complex ideal ratio mask, which, when applied to the reverberant audio predicts the clean sound. The effectiveness of our method is demonstrated through extensive quantitative and qualitative evaluations. Our approach significantly outperforms traditional audio-only and audio-visual baselines on three downstream tasks: speech enhancement, speech recognition, and speaker verification, with relative improvements in the range of 18% - 82% on the LibriSpeech test-clean set. We also achieve highly satisfactory RT60 error scores on the AVSpeech dataset.

Video

Architecture

Overview of AdVerb: It estimates clean source audio from a reverberant speech signal leveraging two primary components:
1) The visual stream processing path comprises a HorizonNet-based backbone to obtain 1D feature sequences, which are subsequently passed to the cross-modal geometry-aware attention subnetwork. 2) The audio processing module applies STFT to get 2D spectrograms which are fed to the cross-modal encoder. The cross-attention subnetwork powered by geometry-aware (Shifted) Window Blocks, Panoptic Blocks, and Relative Position Embedding generates a complex ideal ratio mask.

Results


Reverberant Audio

AdVerb Output

Reverberant Audio

AdVerb Output

Reverberant Audio

AdVerb Output

Reverberant Audio

AdVerb Output

GradCAM Visualizations

Grad-CAM visualization of activated regions. Our model attends to regions that cause heavy reverberation effects.
Some failure cases. The ✓ denotes the regions with correct activation while ✗ spurious detections.

BibTeX

@article{chowdhury2023adverb,
    title={AdVerb: Visually Guided Audio Dereverberation},
    author={Chowdhury, Sanjoy and Ghosh, Sreyan and Subhrajyoti, Dasgupta and Ratnarajah, Anton and Tyagi, Utkarsh and Manocha, Dinesh},
    journal={ICCV},
    year={2023}
}