Robots learn human tasks by watching videos

According to a research team at Carnegie Mellon University (CMU) in the United States, a robot can already watch videos to learn how to perform simple daily household chores. In research, scientists found Robots can perform these tasks by watching videos of people doing common activities, such as opening drawers, opening doors, or answering the phone.

To date, other studies have trained artificial intelligences (AI) by showing exactly how a task is done, for example using a simulated environment or showing a human performing the task. But, HE The model proposed by CMU manages to make a robot learn a task in just 25 minutes just by watching a video.

The new model is called Visual-Robotics Bridge (VRB) and is an enhanced version of another model known as WHIRL (In-the-Wild Human Imitating Robot Learning) that was previously used to train robots.

The major difference between the models is that WHIRL requires a human to perform a task literally in front of a robot in a given environment. In VRB, on the other hand, the physical presence of a person or a particular environment is not required.

“In this paper, we plan to fill this gap by using videos of human interactions in an ecocentric way. Using videos of human behavior on the Internet, we develop a visual model that predicts where and how a human in the scene might interact. The structure of these behavioral features directly allows the robot to perform many complex tasks. allows,” the researchers explain in the study.

video watching robots

Researchers trained AIs Performing 12 types of actions that humans perform nearly every day, such as opening a door, lifting a lid, opening a can of soup, opening a drawer, and more.. After the robots passed 200 hours of training, the tasks were successfully accomplished.

As the team explains, they hope in the future the VRB model will be used to train AIs to perform more complex and multi-step tasks. The study notes that the researchers used four robot learning paradigms to build the model, including offline imitation learning, exploration, goal-dependent learning, and action parameterization for reinforcement learning.

The study states, “The Vision-Robotics Bridge (VRB) is a scalable approach to learning useful features from passive human video data and deploying them to many different robot learning paradigms.”