Register now After registration you will be able to apply for this opportunity online.
This opportunity is not published. No applications will be accepted.
Semantic Robotic Manipulation and Multi-Task Learning
Robotic Manipulation is one of the areas of robotics which has benefited the most from recent advances in large pre-trained machine learning models. At the Soft Robotics Lab, we aim to leverage such models for innovative applications to multi-task manipulation of rigid and soft objects. In this thesis, we plan to 1) set up a manipulation pipeline for control and data collection, and 2) advance the state of imitation learning by leveraging pre-trained semantic models.
Recent works in multi-task robotic manipulation aim to condition policies learnt from imitation learning with language task information (BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning. Jang et al., 2021), leading to surprising zero-shot generalization performance. Other works, leverage pre-trained contrastively learnt multi-modal embedders such as CLIP (Learning transferable visual models from natural language supervision. Radford et al., 2021) to render manipulation imitation learning more sample-efficient (Cliport: What and where pathways for robotic manipulation. Shridhar et al., 2022).
At the Soft Robotics Lab, we are interested in how these advances in multi-task manipulation can be leveraged for applications of robotics in daily life and human-robot collaboration. For this reason, we are in the process of developing a manipulation experimental setup with a Franka Emika Panda and a Flexiv robotic arm. We are looking for a motivated student with robotics experience to join our research effort, help us develop a manipulation pipeline built for machine learning, set up data collection for imitation learning, and assist in the development of novel ideas in the field of pre-trained networks for semantic reasoning.
**Responsibilities**
- Develop a manipulation library and framework for the Soft Robotics Lab based on Franka and Flexiv APIs
- Design and implement a manipulation benchmark and an imitation learning demonstration dataset
- Produce an imitation learning manipulation demo
**What you offer**
A motivated Master’s student with an excellent background in Robotics, knowledge of ROS, and some machine learning background. Knowledge of Python is mandatory, knowledge of Pytorch is a plus.
**What we offer**
The interdisciplinary collaborative environment at the intersection of machine learning, rigid and soft robotics offered by the Soft Robotics Lab and the ETH AI Center.
Recent works in multi-task robotic manipulation aim to condition policies learnt from imitation learning with language task information (BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning. Jang et al., 2021), leading to surprising zero-shot generalization performance. Other works, leverage pre-trained contrastively learnt multi-modal embedders such as CLIP (Learning transferable visual models from natural language supervision. Radford et al., 2021) to render manipulation imitation learning more sample-efficient (Cliport: What and where pathways for robotic manipulation. Shridhar et al., 2022). At the Soft Robotics Lab, we are interested in how these advances in multi-task manipulation can be leveraged for applications of robotics in daily life and human-robot collaboration. For this reason, we are in the process of developing a manipulation experimental setup with a Franka Emika Panda and a Flexiv robotic arm. We are looking for a motivated student with robotics experience to join our research effort, help us develop a manipulation pipeline built for machine learning, set up data collection for imitation learning, and assist in the development of novel ideas in the field of pre-trained networks for semantic reasoning.
**Responsibilities** - Develop a manipulation library and framework for the Soft Robotics Lab based on Franka and Flexiv APIs - Design and implement a manipulation benchmark and an imitation learning demonstration dataset - Produce an imitation learning manipulation demo
**What you offer** A motivated Master’s student with an excellent background in Robotics, knowledge of ROS, and some machine learning background. Knowledge of Python is mandatory, knowledge of Pytorch is a plus.
**What we offer** The interdisciplinary collaborative environment at the intersection of machine learning, rigid and soft robotics offered by the Soft Robotics Lab and the ETH AI Center.
This thesis involves:
- Becoming familiar with a Franka Emika Panda and/or a Flexiv robotic arm
- Setting up a framework to combine arm control with exteroception through 2.5d cameras
- Collect a robotic trajectory dataset for imitation learning
- Reproducing the results of existing multi-task manipulation work
- Advancing the imitation learning state of the art with pre-trained machine learning models
- Potential publication of results in top Machine Learning / Robotics venues
This thesis involves: - Becoming familiar with a Franka Emika Panda and/or a Flexiv robotic arm - Setting up a framework to combine arm control with exteroception through 2.5d cameras - Collect a robotic trajectory dataset for imitation learning - Reproducing the results of existing multi-task manipulation work - Advancing the imitation learning state of the art with pre-trained machine learning models - Potential publication of results in top Machine Learning / Robotics venues
Elvis Nava - elvis.nava@ai.ethz.ch (Main Contact)
Prof. Robert K Katzschmann - rkk@ethz.ch
Elvis Nava - elvis.nava@ai.ethz.ch (Main Contact) Prof. Robert K Katzschmann - rkk@ethz.ch