Register now After registration you will be able to apply for this opportunity online.
AR/VR Remote Traning of Industrial Cobot
This project aims at using the integrated hand tracking functionality of recent AR/VR Head-Mounted Displays to train a cobot remotely on a new task by transferring the user's gestures into robot motions.
Keywords: human-robot interaction, industrial cobot, remote training, hand tracking
Collaborative robots (cobots) are designed to work safely in the immediate human environment and can therefore be very well integrated into industrial processes.
In order to train cobots efficiently on new tasks, it would be beneficial to learn new processes or variations in processes directly from a human worker.
This project aims at utilizing hand tracking capabilities of recent AR Head-Mounted Displays, i.e. Microsoft Hololens 2, to record human hand motions during object manipulation and then transpose these gestures into robot grabbing motions. In a second step, these trained grabbing motions are used to conviniently train a new procedure or a variation of the procedure to the Cobot.
The project is part of a cooperation with Accenture Labs (Sophia Antipolis, France). Human hand motions will be recorded and analyzed at ETH Zurich and tested remotely in a duplicate setup with a physical cobot in France, accessible in real-time remotely.
Collaborative robots (cobots) are designed to work safely in the immediate human environment and can therefore be very well integrated into industrial processes.
In order to train cobots efficiently on new tasks, it would be beneficial to learn new processes or variations in processes directly from a human worker.
This project aims at utilizing hand tracking capabilities of recent AR Head-Mounted Displays, i.e. Microsoft Hololens 2, to record human hand motions during object manipulation and then transpose these gestures into robot grabbing motions. In a second step, these trained grabbing motions are used to conviniently train a new procedure or a variation of the procedure to the Cobot.
The project is part of a cooperation with Accenture Labs (Sophia Antipolis, France). Human hand motions will be recorded and analyzed at ETH Zurich and tested remotely in a duplicate setup with a physical cobot in France, accessible in real-time remotely.
Main tasks include:
1. State-of-the-art analysis
2. Familiarize with the Unity 3D game environment and Microsoft Hololens 2
3. Set up a demonstrator procedure with several physical objects in a lab environment
4. Record human hand motions and train a model to transpose these gestures into robot motions (how to grab an object)
5. Design and implement an intuitive control that allows the user to conveniently train a new procedure or variations of the procedure to the cobot (e.g., changing the order or orientation of manipulated objects) by simple hand gestures and/or drag-and-drop functionalities
6. Test and validate the final workflow in a small case study.
Main tasks include:
1. State-of-the-art analysis
2. Familiarize with the Unity 3D game environment and Microsoft Hololens 2
3. Set up a demonstrator procedure with several physical objects in a lab environment
4. Record human hand motions and train a model to transpose these gestures into robot motions (how to grab an object)
5. Design and implement an intuitive control that allows the user to conveniently train a new procedure or variations of the procedure to the cobot (e.g., changing the order or orientation of manipulated objects) by simple hand gestures and/or drag-and-drop functionalities
6. Test and validate the final workflow in a small case study.
... a very autonomous and methodical way of working. You know how to structure a project, how to derive meaningful work packages and how to systematically develop solutions.
... solid programming skills in a common programming language (e.g., python, c#...).
... You are motivated to familiarize yourself with this fascinating topic of training robots based on human motion and intuitive gestures.
Success in product development depends heavily on the competence and skills of teams and individuals. This is why we dedicate our research to create knowledge that enables the value-adding use of new technologies - and to make this knowledge tangible and teachable. Industrial and clinical needs are the driving forces for our interdisciplinary research. Our work is distinguished by a variety of methods, ranging from simulation to validation of real applications. Our research changes the way we develop products, and our expertise changes the way we create sustainable value.
Success in product development depends heavily on the competence and skills of teams and individuals. This is why we dedicate our research to create knowledge that enables the value-adding use of new technologies - and to make this knowledge tangible and teachable. Industrial and clinical needs are the driving forces for our interdisciplinary research. Our work is distinguished by a variety of methods, ranging from simulation to validation of real applications. Our research changes the way we develop products, and our expertise changes the way we create sustainable value.
- Franka Emika Panda collaborative robot (www.franka.de/technology/)
- remote (teleoperated) cobot training
- human motion to robot motion
- thesis in direct cooperation with Accenture Labs (Digital Experiences team)
