Register now After registration you will be able to apply for this opportunity online.
This opportunity is not published. No applications will be accepted.
Visual-Inertial Teach and Repeat for Robotic Manipulation
The goal of this project is to develop an intuitive robot programming tool by capturing task executions from an uncalibrated visual-inertial sensor strapped to a human operator.
Robots are commonly used in assembly lines, where teach/repeat using internal sensors such joint encoders is an intuitive way of programming new trajectories. However, this relies on the assumption that the task conditions, e.g. poses of objects, do not change. This assumption breaks as robots move to more unstructured environments such as a human homes which require visual sensing capabilities. Imitation learning [1] studies the acquisition of mappings from world state to robot actions from examples provided by a teacher. In particular, recent works in visual one-shot imitation learning [2,3] aim to learn new tasks from single demonstrations.
In this work, we propose to develop an intuitive robot programming tool by capturing task executions from an uncalibrated visual-inertial sensor strapped to a human operator. This will require learning the relationship between the hand (human or robot) and objects that are interacted with, as well as a mapping from the recorded human demonstrations to a robot setup. The student should familiarize himself/herself with the current state-of-the-art in imitation learning and domain adaptation, and propose the most suitable approach for the before-mentioned setup. The final demonstration should show the robot successfully repeating simple manipulation skills, such as pick-and-place or opening a drawer.
[1] B. D. Argall, S. Chernova, M. Veloso, and B. Browning, “A survey of robot learning from demonstration,” Robotics and Autonomous Systems, vol. 57, no. 5, pp. 469–483, May 2009.
[2] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine, “One-Shot Visual Imitation Learning via Meta-Learning,” in Conference on Robot Learning, 2017, pp. 357–368.
[3] P. Sermanet et al., “Time-Contrastive Networks: Self-Supervised Learning from Video,” in 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018, pp. 1134–1141.
Robots are commonly used in assembly lines, where teach/repeat using internal sensors such joint encoders is an intuitive way of programming new trajectories. However, this relies on the assumption that the task conditions, e.g. poses of objects, do not change. This assumption breaks as robots move to more unstructured environments such as a human homes which require visual sensing capabilities. Imitation learning [1] studies the acquisition of mappings from world state to robot actions from examples provided by a teacher. In particular, recent works in visual one-shot imitation learning [2,3] aim to learn new tasks from single demonstrations.
In this work, we propose to develop an intuitive robot programming tool by capturing task executions from an uncalibrated visual-inertial sensor strapped to a human operator. This will require learning the relationship between the hand (human or robot) and objects that are interacted with, as well as a mapping from the recorded human demonstrations to a robot setup. The student should familiarize himself/herself with the current state-of-the-art in imitation learning and domain adaptation, and propose the most suitable approach for the before-mentioned setup. The final demonstration should show the robot successfully repeating simple manipulation skills, such as pick-and-place or opening a drawer.
[1] B. D. Argall, S. Chernova, M. Veloso, and B. Browning, “A survey of robot learning from demonstration,” Robotics and Autonomous Systems, vol. 57, no. 5, pp. 469–483, May 2009.
[2] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine, “One-Shot Visual Imitation Learning via Meta-Learning,” in Conference on Robot Learning, 2017, pp. 357–368.
[3] P. Sermanet et al., “Time-Contrastive Networks: Self-Supervised Learning from Video,” in 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018, pp. 1134–1141.
- Literature review on imitation learning and domain adaptation
- Theoretical formulation of the problem
- Designing a sensor mount for the human demonstrator
- Data collection
- Implementation and validation of the algorithm
- Documentation and dissemination
- Literature review on imitation learning and domain adaptation - Theoretical formulation of the problem - Designing a sensor mount for the human demonstrator - Data collection - Implementation and validation of the algorithm - Documentation and dissemination
- Highly motivated and independent student
- Background in computer vision and deep learning
- Great programming skills in Python
- Experience with ROS is a plus
- Highly motivated and independent student - Background in computer vision and deep learning - Great programming skills in Python - Experience with ROS is a plus
- Michel Breyer <michel.breyer@mavt.ethz.ch>
- Jen Jen Chung <jenjen.chung@mavt.ethz.ch>
- Michel Breyer <michel.breyer@mavt.ethz.ch> - Jen Jen Chung <jenjen.chung@mavt.ethz.ch>