OmniDrive introduces a holistic dataset that bridges the gap between 2D vision-language models and 3D driving environments, enabling advanced reasoning capabilities for autonomous vehicles.

Key Innovations:

• Counterfactual reasoning approach that evaluates potential scenarios to improve decision-making
• Alignment of vision-language models with full 3D understanding for real-world driving applications
• Comprehensive dataset designed specifically for autonomous driving challenges
• Integration of both visual perception and language reasoning in dynamic driving contexts

Engineering Impact: This research addresses a critical challenge in autonomous driving by extending AI reasoning capabilities from 2D to 3D environments, potentially improving safety and decision-making in complex traffic scenarios.

OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning