Set a transparent glass vase next to an acrylic plastic replica, and from across the room, the eye can easily be deceived. Both objects share the same contours, reflecting light and casting shadows in nearly identical ways. Yet one light tap with a spoon reveals the difference: the glass rings out with a clear, sharp resonance, while the plastic produces a dull, muted thud.
While modern computer graphics can render stunningly photorealistic 3D models, digital environments remain largely silent or rely on generic audio effects. Today’s digital objects typically reproduce appearance but omit the rich physical information conveyed by sound.
To bridge this sensory gap, University of Maryland researchers have developed Audio-Visual Modal Sound Field (AV-MSF). The system pairs visual geometry with acoustic physics to give 3D digital objects a realistic, responsive voice — a development that could help robots handle fragile items, enhance virtual reality and streamline game audio production.
Traditionally, modeling how an object sounds required choosing between expensive computer simulations and data-hungry AI models that needed thousands of recorded taps. AV-MSF addresses these bottlenecks with a simple insight: regions with similar local geometry and visual features tend to vibrate in similar ways. By combining camera images of an object with roughly six recorded taps, the system builds a digital sound profile.
The research was co-led by Zisen Shao, a second-year master’s student in computer science, and Zihao Wei, a second-year Ph.D. student in computer science, working alongside Derong Jin, a third-year Ph.D. student in computer science, and Ruohan Gao, an assistant professor of computer science with an affiliate appointment in the University of Maryland Institute for Advanced Computer Studies (UMIACS).
Together, the team found that a surprisingly small amount of real-world audio data was needed to map an entire object’s acoustic response.
Shao said that distributing these reference strikes across an object provides enough acoustic coverage for most everyday items, while simpler, highly symmetric shapes like a ball may require only a single strike. To capture those initial reference strikes in the lab, the researchers use a calibrated impact hammer — a specialized tool that allows the system to isolate an object’s acoustic profile while accounting for the hammer’s own contribution to the sound.
The system also distinguishes an object's decaying resonances from non-modal effects, such as background noise, and models these components separately to improve sound reconstruction.
From these minimal inputs, AV-MSF predicts and generates the acoustic response of an object struck at any location on its surface, including regions that were never physically touched.
Compared with previous methods, AV-MSF achieves a twofold improvement in sound synthesis accuracy and a 10% improvement in contact localization or identifying where an impact occurred. In listening comparisons, the synthesized hits closely resemble recordings of actual physical contact. Go here for examples.
“As our model can learn from real-world data to render sounds at novel locations, the resulting sounds closely resemble real recordings, rather than sounding AI-generated,” Shao said. “Our focus is on ensuring the rendered sounds are not only realistic, but also physically consistent and grounded in the corresponding impact locations.”
Beyond generating convincing audio, acoustic feedback could help autonomous machines better understand the physical world. As Shao pointed out, relying on vision alone can mislead a machine — shadows, poor lighting or bad camera angles can distort an object’s appearance. Sound provides an additional physical signal. Because it stems directly from an object’s internal vibrations, tapping can reveal information about its material properties.
By tapping an unfamiliar object, a robot could infer whether it is fragile or sturdy and adjust its manipulation force accordingly. Precise contact localization could also help robots respond to collisions: when an unexpected impact occurs, the machine could identify where contact happened and use that information to correct its trajectory or adjust its grip.
The technology also offers possibilities for gaming, digital heritage and virtual reality, which Shao viewed as one of the project’s most immediate practical applications.
“Many museum objects are too fragile to be physically handled or placed on long-term display,” he said. “It would be a cool application to build virtual museum artifacts that people can not only see, but also interact with to hear their sounds, enabling more immersive and accessible engagement with art and cultural heritage.”
For game developers and sound designers, the system could reduce the labor needed to create sound effects for complex digital assets. Creators can also edit an object’s acoustic properties to make it sound as though it were made of a different material, for example, transforming a ceramic vase into one that sounds like solid oak or hollow metal.
While AV-MSF works well with rigid or elastic objects that return to their original shape after impact, some physical behaviors remain outside its current scope. Highly deformable materials like rubber or silicone, as well as events like structural fractures, violate the system’s assumptions about linear vibration and remain difficult to model. Highly reflective or semi-transparent surfaces can also complicate the initial 3D visual reconstruction, though the system’s modular design allows it to incorporate improvements in computer vision.
Next, the team aims to move the technology beyond controlled lab settings. By pairing AI vision models with tools that separate sounds from background noise, future systems could map an object’s acoustic signature directly from everyday video clips.
—Story by Melissa Brachfeld, UMIACS communications group