The problem is that training multiskilled robots requires lots of data. Pinto’s solution is to find novel ways to collect that data—in particular, getting robots to collect it as they learn, an approach called self-supervised learning (a technique also championed by Meta’s chief AI scientist and Pinto’s NYU colleague Yann LeCun, among others).
“Lerrel’s work is a major milestone in bringing machine learning and robotics together,” says Pieter Abbeel, director of the robot learning lab at the University of California, Berkeley. “His current research will be looked back upon as having laid many of the early building blocks of the future of robot learning.”
The idea of a household robot that can make coffee or wash dishes is decades old. But such machines remain the stuff of science fiction. Recent leaps forward in other areas of AI, especially large language models, made use of enormous data sets scraped from the internet. You can’t do that with robots, says Pinto.
Self-driving-car companies clock millions of hours on the road, collecting data to train the models that power their vehicles. Makers of household robots face a similar challenge, recording many hours of robot’s-eye footage of different tasks being carried out in different settings.
Pinto hit one of his first milestones back in 2016, when he created the world’s largest robotics data set at the time by getting robots to create and label their own training data and running them 24/7 without human supervision.
He and his colleagues have since developed learning algorithms that allow a robot to improve as it fails. A robot arm might fail many times to grasp an object, but the data from those attempts can be used to train a model that succeeds. The team has demonstrated this approach with both a robot arm and a drone, turning each dropped object or collision into a hard-won lesson.