As scientists work toward a future where robots can handle everyday household chores like washing dishes, folding laundry and ironing shirts, one of the field’s biggest hurdles is ensuring these machines can operate reliably in the unpredictable environments of real homes.
At the University of Maryland, doctoral student Seungjae “Jay” Lee is developing new data-driven methods designed to bridge the gap between impressive laboratory demonstrations and dependable real-world performance. His work centers on enabling robots to learn not only from their own physical experiences, but also from the vast reservoir of human activity captured online.
“People often focus on designing better model architectures, but for artificial intelligence that integrates AI algorithms into physical systems, the real bottleneck is the dataset itself,” Lee says.
Lee points to what researchers describe as a “scarcity” problem in robotics. Unlike large language models that can learn from massive volumes of readily available internet text, physical robots require specialized data—tactile feedback, sensor readings and action trajectories—collected in real-world settings. Gathering this data is slow, expensive and technically demanding.
“If we can transfer knowledge from web-scale human data into robotics, we can overcome the scarcity problem,” Lee explains.
A second-year Ph.D. student in computer science, Lee envisions household robots that can assist with routine tasks within the next five to 10 years. But while robotic systems often perform well in controlled lab environments, their reliability frequently declines in real homes, where lighting conditions, layouts and object arrangements constantly vary.
Training robots to handle this complexity demands enormous amounts of diverse data—something Lee’s research aims to provide.
One project Lee is involved in, Imagine, Verify, Execute, offers a framework that allows robots to learn through autonomous exploration rather than relying solely on pre-programmed instructions. As robots test and refine their actions in real time, they effectively generate their own training data.
“If you record that journey, it becomes training data,” Lee says. “The robot is generating its own experience.”
In a complementary effort, Lee took the lead in developing TraceGen, a system that mines hundreds of thousands of publicly available human videos to extract meaningful hand and object motion. By analyzing in-the-wild footage from large datasets and platforms such as YouTube, TraceGen estimates camera position, depth and motion trajectories to isolate the movements required to complete specific tasks.
These human-derived behaviors are then used to train robotic systems alongside data generated by robots themselves. Lee describes the robotics data ecosystem as a pyramid: scarce but high-value real robot data at the top; more abundant but imperfect simulation data in the middle; and massive quantities of diverse human video data forming the base.
TraceGen was recently accepted to the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), scheduled for June in Denver—an important milestone in the computer vision and robotics communities.
Lee’s research is advised by Furong Huang and Jia-Bin Huang, associate professors of computer science with appointments in the University of Maryland Institute for Advanced Computer Studies (UMIACS).
“What sets Seungjae apart is his rare combination of vision and execution,” says Furong Huang. “He operates with professionalism, precision and strong ownership of his work. I see him as a rising leader at the intersection of machine learning and robotics, with the potential to shape how intelligent systems learn and interact with the physical world.”
This summer, Lee will further test his approach during an internship with NVIDIA’s Generalist Embodied Agent Research (GEAR) team. There, he will integrate large-scale human video data into advanced robotic platforms to evaluate performance gains in real-world environments.
Lee said the experience will support his long-term goal of advancing household robotics in industry before eventually returning to academia.
If successful, his work could help bring dependable robotic assistants out of controlled research settings and into everyday homes—moving the field one step closer to robots that can truly share in daily life.
—Story by Diya Sharma, UMIACS communications group