Register now After registration you will be able to apply for this opportunity online.
This opportunity is not published. No applications will be accepted.
Sensor Fusion from Edge Devices to Improve Robustness of Machine Learning Systems
This project aims at the development of a system that fuses sensor data from Microsoft HoloLens 2 and the Azure Kinect Camera and uses the fused data to train machine learning models.
By superimposing virtual information on top of a users enviroment, augmented reality (AR) can support
critical manual procedures in areas such as surgery or machine maintenance operations.
To adapt how and what information is displayed to the user in an automated way
machine learning based detections, such as activity recognition or object pose estimation
are used to infer the user's intention and environment.
To offer a real benefit to the user, those detections have to provide high reliability,
while working under diverse conditions.
One approach to increase the robustness of such machine learning algorithms is to fuse information from multimodal sensors
and to integrate multiple detection systems into a larger system.
By superimposing virtual information on top of a users enviroment, augmented reality (AR) can support critical manual procedures in areas such as surgery or machine maintenance operations. To adapt how and what information is displayed to the user in an automated way machine learning based detections, such as activity recognition or object pose estimation are used to infer the user's intention and environment.
To offer a real benefit to the user, those detections have to provide high reliability, while working under diverse conditions. One approach to increase the robustness of such machine learning algorithms is to fuse information from multimodal sensors and to integrate multiple detection systems into a larger system.
The main focus will lie on fusing image and hand/head pose data from the HoloLens 2 with image data from the Azure Kinect.
Once the system works reliably, other sensor modalities and devices can be added as well.
Based on the candidate's preferences, the project focus can also lie on improving the machine learning architecture of the final prediction system as well.
To achieve this goal, this project will consist of three stages:
1. Implementing sensor streaming from multiple devices to a server for further processing.
2. Implementing sensor fusion on the server.
3. Training multiple prediction systems on the fused sensor data and integrating them into a larger system.
The main focus will lie on fusing image and hand/head pose data from the HoloLens 2 with image data from the Azure Kinect. Once the system works reliably, other sensor modalities and devices can be added as well. Based on the candidate's preferences, the project focus can also lie on improving the machine learning architecture of the final prediction system as well.
To achieve this goal, this project will consist of three stages:
1. Implementing sensor streaming from multiple devices to a server for further processing. 2. Implementing sensor fusion on the server. 3. Training multiple prediction systems on the fused sensor data and integrating them into a larger system.
- enthusiasm about working on networking, data (especially image) processing and machine learning - good programming skills in either Python (or Java, C#, C, C++) - methodical way of working
As part of our research at the AR Lab within the Human Behavior Group we are working on automatically analyzing a user’s interaction with his environment in scenarios such as surgery or in industrial machine interactions. By collecting real-world datasets during those scenarios and using them for machine learning tasks such as activity recognition, object pose estimation or image segmentation we can gain an understanding of how a user performed during a given task. We can then utilize this information to provide the user with real-time feedback on his task using mixed reality devices, such as the Microsoft HoloLens, that can guide him and prevent him from doing mistakes.
As part of our research at the AR Lab within the Human Behavior Group we are working on automatically analyzing a user’s interaction with his environment in scenarios such as surgery or in industrial machine interactions. By collecting real-world datasets during those scenarios and using them for machine learning tasks such as activity recognition, object pose estimation or image segmentation we can gain an understanding of how a user performed during a given task. We can then utilize this information to provide the user with real-time feedback on his task using mixed reality devices, such as the Microsoft HoloLens, that can guide him and prevent him from doing mistakes.