Skild AI and NVIDIA: robots learn tasks through video

In a dynamic environment like factories and warehouses, it is crucial for robots to adapt quickly to changing tasks and processes. Skild AI introduces the S1 robot foundation model, which is capable of learning new and unfamiliar tasks simply by watching a single video demonstration. This innovative approach, known as in-context learning, reduces the need for extensive reprogramming and training.

The development of S1 took place on NVIDIA's AI infrastructure, which is part of a broader collaboration aimed at moving robots with flexible intelligence from the lab environment to practice. The CEO of Skild AI, Deepak Pathak, emphasizes that learning through experience represents a significant shift in robotics, with NVIDIA's Isaac Lab and Cosmos technologies being crucial in creating the diverse experiences necessary for robots.

The S1 can perform unknown tasks that take up to 10 minutes, such as potting plants and making pancakes. A noteworthy test showed that the S1 was able to demonstrate potting a plant and then gain autonomy in just 11 minutes. This is a significant improvement compared to similar AI systems that achieved only a 9% success rate on new, complex tasks.

By using this direct learning approach, operators can demonstrate new tasks without the need for a new dataset or training session for each change. This not only increases efficiency but also accelerates future implementations, as evidenced by the collaboration with Foxconn for the assembly of NVIDIA systems. Skild AI's S1 is equipped with NVIDIA technologies that optimize training and validation, including simulations and reinforcement learning, enabling robust robot intelligence.

Read the full article from Nvidia.