Register now After registration you will be able to apply for this opportunity online.
Learning Multi-Modal Robust Environmental Representation for Off-road Autonomous Driving
This project seeks to improve off-road autonomous driving by developing a system that merges data from multiple sensors—LiDAR and RGBD cameras—into a real-time, robust environmental model. The goal is to enhance environmental representation by integrating geometric data with image-based learning, using deep learning to include semantic information for better navigation and decision-making in a challenge off-road environment.
Navigating off-road terrains is challenging due to the dynamic and unstructured environment, sensor occlusions, state estimation drift, dynamic motions, and many more challenges. Traditional single-sensor modality approaches often fall short of creating a comprehensive and adaptive environmental representation capable of overcoming these challenges. This project aims to advance the field by developing an integrated framework that fuses multi-sensor data from LiDAR and RGBD cameras to learn a real-time, robust environmental representation for off-road autonomous driving, specifically focusing on how we can exploit powerful representations learned in the image space to form better representations also in the perceived geometric data.
In addition to creating a high-fidelity geometric model of the environment, the framework should allow for implicit encoding of visual such as semantic information by combining geometry and image features utilizing deep learning. Preliminary results indicate that this multi-faceted, semantically-aware environmental representation significantly improves the autonomous system's navigation and decision-making capabilities. By comprehensively addressing the unique challenges of off-road autonomous driving, this project aims to contribute significantly to the development of safer and more reliable navigation systems for such scenarios.
Navigating off-road terrains is challenging due to the dynamic and unstructured environment, sensor occlusions, state estimation drift, dynamic motions, and many more challenges. Traditional single-sensor modality approaches often fall short of creating a comprehensive and adaptive environmental representation capable of overcoming these challenges. This project aims to advance the field by developing an integrated framework that fuses multi-sensor data from LiDAR and RGBD cameras to learn a real-time, robust environmental representation for off-road autonomous driving, specifically focusing on how we can exploit powerful representations learned in the image space to form better representations also in the perceived geometric data. In addition to creating a high-fidelity geometric model of the environment, the framework should allow for implicit encoding of visual such as semantic information by combining geometry and image features utilizing deep learning. Preliminary results indicate that this multi-faceted, semantically-aware environmental representation significantly improves the autonomous system's navigation and decision-making capabilities. By comprehensively addressing the unique challenges of off-road autonomous driving, this project aims to contribute significantly to the development of safer and more reliable navigation systems for such scenarios.
- Implementation of a self-supervised learning pipeline to learn powerful geometric features using the supervision from existing image feature extractors
- Verification on autonomous driving dataset (Nuscenes, KITTI)
- Transfer to off-road driving data
- Evaluation of degenerated perception capabilities when sensors are occluded and verification of the enhanced situational awareness.
- Evaluation of robustness to state estimation drifts that arise from high-dynamic motions.
- Evaluation of the implicit encoding of semantic information such as vegetation, road conditions, or potential hazards by training a linear probe to predict the semantics.
- Deployment on the real-robot potentially for navigation (TBD)
- Implementation of a self-supervised learning pipeline to learn powerful geometric features using the supervision from existing image feature extractors - Verification on autonomous driving dataset (Nuscenes, KITTI) - Transfer to off-road driving data - Evaluation of degenerated perception capabilities when sensors are occluded and verification of the enhanced situational awareness. - Evaluation of robustness to state estimation drifts that arise from high-dynamic motions. - Evaluation of the implicit encoding of semantic information such as vegetation, road conditions, or potential hazards by training a linear probe to predict the semantics. - Deployment on the real-robot potentially for navigation (TBD)
- Python, PyTorch
- Applied deep learning-based methods
- Eager to learn and highly motivated
- Ideally worked on a code base or track record of prior coding projects
- Python, PyTorch - Applied deep learning-based methods - Eager to learn and highly motivated - Ideally worked on a code base or track record of prior coding projects
Fan Yang (fanyang1@ethz.ch)
Vaishakh Patil (patilv@ethz.ch)
Pascal Roth (rothpa@ethz.ch)
Fan Yang (fanyang1@ethz.ch) Vaishakh Patil (patilv@ethz.ch) Pascal Roth (rothpa@ethz.ch)
Learning Multi-Modal Robust Environmental Representation for Off-road Autonomous Driving.pdf
