Register now After registration you will be able to apply for this opportunity online.
This opportunity is not published. No applications will be accepted.
Advanced User Interfaces for Shared Task Human-Robot Collaboration
The goal of this thesis is to develop a 3D interactive user interface for our human-robot collaboration system with a robotic arm using AR and modalities such as speech, gesture recognition, and eye tracking.
Keywords: Robotics, User Interface, Multimodal, Data Science, Computer Vision, Human-Robot Collaboration, Deep Learning
We developed a human-robot collaboration system, which allows a robot arm to function as a “third hand” in shared tasks. Currently, the user interface is based on the Hololens 2 AR glasses and uses gestures and eye-tracking to give robot commands. However, our rudimentary UI implementation is clunky and distracts the user from the task at hand. To develop a more seamless user interface, this project aims at exploring a wide range of sensors and input modalities, such as speech, gestures, full-body, hand and eye-tracking using cameras, microphones and IMUs. We are looking for a person that is passionate about robotics and wants to work with multimodal data to improve the collaboration between humans and robots.
We developed a human-robot collaboration system, which allows a robot arm to function as a “third hand” in shared tasks. Currently, the user interface is based on the Hololens 2 AR glasses and uses gestures and eye-tracking to give robot commands. However, our rudimentary UI implementation is clunky and distracts the user from the task at hand. To develop a more seamless user interface, this project aims at exploring a wide range of sensors and input modalities, such as speech, gestures, full-body, hand and eye-tracking using cameras, microphones and IMUs. We are looking for a person that is passionate about robotics and wants to work with multimodal data to improve the collaboration between humans and robots.
You will be introduced to our robot system and the sensors available (cameras, microphone, smartwatch with IMU). Your tasks include:
- Exploring potential interactions to control the robot in different scenarios and collaboration modes
- Digitalizing the required interactions using techniques from computer vision and data science
- Implementing the resulting user interface into a demonstrator, which can be shown in fairs and conferences
You will be introduced to our robot system and the sensors available (cameras, microphone, smartwatch with IMU). Your tasks include:
- Exploring potential interactions to control the robot in different scenarios and collaboration modes
- Digitalizing the required interactions using techniques from computer vision and data science
- Implementing the resulting user interface into a demonstrator, which can be shown in fairs and conferences
- Strong programming skills (Python, C#, C++, …) - An interest in robotics - Experience with machine learning, data science or computer vision - The ability to take initiative and shape the direction of the project - Enthusiasm for tackling practical challenges
As part of our research at the AR Lab within the Human Behavior Group we are working on automatically analyzing a user’s interaction with his environment in scenarios such as surgery or in human-robot collaboration. By collecting real-world datasets during those scenarios and using them for machine learning tasks such as activity recognition, object pose estimation or image segmentation we can gain an understanding of how a user performed during a given task. We can then utilize this information to provide the user with real-time feedback on his task using mixed reality devices, such as the Microsoft HoloLens, that can guide him and prevent him from doing mistakes.
As part of our research at the AR Lab within the Human Behavior Group we are working on automatically analyzing a user’s interaction with his environment in scenarios such as surgery or in human-robot collaboration. By collecting real-world datasets during those scenarios and using them for machine learning tasks such as activity recognition, object pose estimation or image segmentation we can gain an understanding of how a user performed during a given task. We can then utilize this information to provide the user with real-time feedback on his task using mixed reality devices, such as the Microsoft HoloLens, that can guide him and prevent him from doing mistakes.
- Master thesis
Please send your CV and master thesis grades to Sophokles Ktistakis (ktistaks@ethz.ch)
Please send your CV and master thesis grades to Sophokles Ktistakis (ktistaks@ethz.ch)
