Acoustic Visualization: The Audio Camera

Spherical microphone arrays provide an ability to compute the acoustical intensity corresponding to different spatial directions in a given frame of audio-data. These intensities may be exhibited as an image and these images updated at a high frame rate to achieve a video stream if the data capture and intensity computations can be performed sufficiently quickly, there by creating a frame-rate audio camera. We describe how such a camera can be built in [1,2] and show that the processing can be done sufficiently quickly using graphics processors. The joint processing of captured frame-rate audio and video images enables applications such as visual identification of noise sources, beamforming and noise-suppression in video conferencing and others, provided it is possible to account for the spatial differences in the location of the audio and the video cameras. Based on the recognition that the spherical array can be viewed as a central projection camera it is possible to perform such joint analysis. In this web page we provide sample applications and videos demonstrating the usefulness of the proposed approach.

1) Joint Audio-Video Calibration

2) Vision Guided Beamforming

3) Acoustic image overlay

4) Visualizing the Reverberant Structure of Concert Halls

5) Speech Separation

 

For more information see the following papers

[1] Adam O' Donovan, Ramani Duraiswami and Jan Neumann. Microphone Arrays as Generalized Cameras for Integrated Audio Visual Processing, accepted as an oral presentation, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Minneapolis, 2007 pdf

[2] Adam O'Donovan, Ramani Duraiswami, Nail A. Gumerov, Real Time Capture of Audio Images and Their Use with Video
Proceedings 2007 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics pdf

[3]Adam O’Donovan, Ramani Duraiswami and Dmitry Zotkin. Imaging Concert Hall Acoustics Using Visual and Audio Cameras. Submitted. Submitted 2007 pdf

*We would like to thank Dr. David Luebke of NVidia for the donation of the GPU card used in the real time audio camera. We would also like to thank David Ross of the VA for funding that partially supported these projects.