This research introduces a groundbreaking approach that enables robots to perform peg insertion tasks on unseen objects without task-specific training, leveraging vision-language models for generalizable perception.

• Identifies potential mating holes using vision-language models to understand object functionality
• Employs a novel pose estimation technique to accurately align pegs with holes
• Demonstrates real-world effectiveness on a variety of industrial assembly tasks
• Achieves high success rates in zero-shot scenarios where traditional methods fail

This advancement represents a significant step toward more adaptable manufacturing systems that can handle diverse assembly tasks without reprogramming, potentially reducing setup times and increasing flexibility in industrial environments.

Zero-Shot Peg Insertion: Identifying Mating Holes and Estimating SE(2) Poses with Vision-Language Models