Virtual 3D Audio

Humans hear 3D sounds after they are filtered by their individual ears and head structures. The virtual audio rendering pipeline involves performing a large number of long convolutions with filters that vary as the listener moves in the environment or changes his/her pose.

Directional Audio Response of a moving car recorded by a Microphone Array

We are interested in developing new fast Fourier transform algorithms based on fast multipole methods. These algorithms may have certain advantages over the conventional FFT when implemented on parallel machines, and when low-latency versions are needed. We plan to explore finite-multipole based fast Fourier transforms for low-latency convolutions on the CPU-GPU cluster for enabling real-time virtual audio environments. Current projects are described below:

Real-Time Sound Camera using GPUs and CPUs

Spherical microphone arrays provide an ability to compute the acoustical intensity corresponding to different spatial directions in a given frame of audio-data. These intensities may be exhibited as an image and if the data capture and intensity computations can be performed sufficiently quickly, a video, thereby creating a frame-rate audio camera.

Intensity Image from a Sound Camera

We describe how such a camera can be built and the processing done sufficiently quickly using graphics processors. The joint processing of and captured frame-rate audio and video images enables applications such as visual identification of noise sources, beamforming and noise-suppression in video conferenceing and others, provided it is possible to account for the spatial differences in the location of the audio and the video cameras. Based on the recognition that the spherical array can be viewed as a central projection camera it is possible to perform such joint analysis. We provide several examples of real-time applications. We have also investigated large-scale ranking problems and machine learning algorithms on the cluster [Raykar et al. 07].