Register now After registration you will be able to apply for this opportunity online.
Diffusion Navigation for ANYmal - Semester Project, Master Thesis
Navigation of ANYmal using Diffusion Policy including real-world experiments and deployment.
References
[1] Reuss, Moritz, et al. "Goal-conditioned imitation learning using score-based diffusion policies." RSS 2023
[2] Shah, Dhruv, et al. "ViNT: A foundation model for visual navigation." CoRL 2023
[3] Sridhar, Ajay, et al. "Nomad: Goal masked diffusion policies for navigation and exploration." 2024 (ICRA). IEEE, 2024.
A core challenge in developing navigation systems for legged robots is ensuring they behave in ways that align with human social norms and expectations. For example, robots should prefer walking on sidewalks and waiting for green lights at intersections. In this project, we aim to address this challenge by integrating socially aligned behavior into the navigation system of a legged robot. Our approach leverages imitation learning from human demonstrations. We plan to train a diffusion policy model that predicts the optimal trajectory for the robot to reach a specified target goal (defined by an image, text, or pose). For training data, we will use an existing dataset of expert demonstrations collected across over 100 diverse environments, in which the ANYmal robot performs a wide range of tasks. Additionally, we will supplement this data with cost-effective human navigation data captured via GoPro footage, using "GLOMAP - Global Structure-from-Motion Revisited" to extract relevant labels.
One of the key challenges will be learning to recover from unexpected situations, e.g. facing a wall, which may not be in the training data domain. To address this, we plan to deploy an initial baseline model on the robot. In cases where the robot encounters unexpected states, we will switch control to a human operator who will demonstrate the correct behavior. We will investigate if this iterative approach will help us achieve robust autonomous navigation, and allow us to fully autonomously navigate throughout the LEE office spaces. The project’s scope can be adjusted based on the requirements of a Master's thesis or Semester project and will be co-supervised between the RSL and Intuitive Robots Lab (KIT).
We offer weekly supervision, cluster access, student workstations, and working with our legged robot ANYmal.
A core challenge in developing navigation systems for legged robots is ensuring they behave in ways that align with human social norms and expectations. For example, robots should prefer walking on sidewalks and waiting for green lights at intersections. In this project, we aim to address this challenge by integrating socially aligned behavior into the navigation system of a legged robot. Our approach leverages imitation learning from human demonstrations. We plan to train a diffusion policy model that predicts the optimal trajectory for the robot to reach a specified target goal (defined by an image, text, or pose). For training data, we will use an existing dataset of expert demonstrations collected across over 100 diverse environments, in which the ANYmal robot performs a wide range of tasks. Additionally, we will supplement this data with cost-effective human navigation data captured via GoPro footage, using "GLOMAP - Global Structure-from-Motion Revisited" to extract relevant labels.
One of the key challenges will be learning to recover from unexpected situations, e.g. facing a wall, which may not be in the training data domain. To address this, we plan to deploy an initial baseline model on the robot. In cases where the robot encounters unexpected states, we will switch control to a human operator who will demonstrate the correct behavior. We will investigate if this iterative approach will help us achieve robust autonomous navigation, and allow us to fully autonomously navigate throughout the LEE office spaces. The project’s scope can be adjusted based on the requirements of a Master's thesis or Semester project and will be co-supervised between the RSL and Intuitive Robots Lab (KIT). We offer weekly supervision, cluster access, student workstations, and working with our legged robot ANYmal.
Work packages
- Literature Review (Diffusion models and Navigation systems)
- Implementation - Offline Training (Network Training and Data Preparation)
- Analysis - Evaluation of Network Performance
- Real-World Controller Implementation
- Deployment on ANYmal
Work packages - Literature Review (Diffusion models and Navigation systems) - Implementation - Offline Training (Network Training and Data Preparation) - Analysis - Evaluation of Network Performance - Real-World Controller Implementation - Deployment on ANYmal
- Good theoretical understanding of ML
- Excellent knowledge of Python
- (Optional) Experience using ROS and cluster compute
- Good theoretical understanding of ML - Excellent knowledge of Python - (Optional) Experience using ROS and cluster compute
Please write a mail to: Jonas Frey(jonfrey@ethz.ch) and Moritz Reuss (moritz.reuss@kit.edu). Please include your CV and transcript.
Please write a mail to: Jonas Frey(jonfrey@ethz.ch) and Moritz Reuss (moritz.reuss@kit.edu). Please include your CV and transcript.
