Register now After registration you will be able to apply for this opportunity online.
This opportunity is not published. No applications will be accepted.
Exploring GANs for Real-World to Simulation Style Transfer
To enhance the power of our object detection and 3D pose estimation models that were trained on simulated training data, this thesis aims at exploring the use generative adversarial neural networks (GANs) to perform style transfer between real-world and synthetically rendered images.
To provide automated, context-aware user support to mixed reality applications in areas such as surgery, machine maintenance or robotics it is important to understand the user’s interactions with the objects in his environment. Key elements towards this goal are object detection and pose estimation models, which we train mostly on simulated images from the 3D rendering software Blender. Although simulated training data offers the ability to quickly exchange the virtual environment or key objects according to the intended use-case, models trained on simulated data alone suffer from performance degradation once they are deployed to real-world environments. To bridge this simulation-to-reality gap, this thesis aims at exploring generative adversarial neural networks (GANs) to perform real-world to simulation neural style transfer.
Further reading:
Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks (Zhu et al.)
To provide automated, context-aware user support to mixed reality applications in areas such as surgery, machine maintenance or robotics it is important to understand the user’s interactions with the objects in his environment. Key elements towards this goal are object detection and pose estimation models, which we train mostly on simulated images from the 3D rendering software Blender. Although simulated training data offers the ability to quickly exchange the virtual environment or key objects according to the intended use-case, models trained on simulated data alone suffer from performance degradation once they are deployed to real-world environments. To bridge this simulation-to-reality gap, this thesis aims at exploring generative adversarial neural networks (GANs) to perform real-world to simulation neural style transfer.
Further reading:
Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks (Zhu et al.)
You will explore how different GAN architectures can be used to translate between real images and those rendered in simulated environments and integrate those image translations into downstream tasks such as 3D pose estimation.
Your main tasks include:
- Familiarizing yourself with the current state and usage of GANs for our use cases in surgery, machine maintenance or robotics
- Familiarizing yourself with the 3D rendering software Blender to create pipelines that render synthetic images in different environments
- Implementing state of the art GAN networks using existing libraries and frameworks such as Pytorch, Tensorflow and OpenCV
- Creating datasets and benchmarking models under different influence factors to expose strengths and weaknesses of the models.
You will explore how different GAN architectures can be used to translate between real images and those rendered in simulated environments and integrate those image translations into downstream tasks such as 3D pose estimation. Your main tasks include:
- Familiarizing yourself with the current state and usage of GANs for our use cases in surgery, machine maintenance or robotics
- Familiarizing yourself with the 3D rendering software Blender to create pipelines that render synthetic images in different environments
- Implementing state of the art GAN networks using existing libraries and frameworks such as Pytorch, Tensorflow and OpenCV
- Creating datasets and benchmarking models under different influence factors to expose strengths and weaknesses of the models.
- Passion: Implementing state-of-the art neural networks and making them work well on use cases such as surgery or robotics is exciting to you. - Hands on approach: You get things working quickly. - Ability to take ownership: You want to shape the direction of this project. - You need to have advanced programming skills in at least one language such as Python, C#/Java, C++.
As part of our research at the AR Lab within the Human Behavior Group we are working on automatically analyzing a user’s interaction with his environment in scenarios such as surgery or in industrial machine interactions. By collecting real-world datasets during those scenarios and using them for machine learning tasks such as activity recognition, object pose estimation or image segmentation we can gain an understanding of how a user performed during a given task. We can then utilize this information to provide the user with real-time feedback on his task using mixed reality devices, such as the Microsoft HoloLens, that can guide him and prevent him from doing mistakes.
As part of our research at the AR Lab within the Human Behavior Group we are working on automatically analyzing a user’s interaction with his environment in scenarios such as surgery or in industrial machine interactions. By collecting real-world datasets during those scenarios and using them for machine learning tasks such as activity recognition, object pose estimation or image segmentation we can gain an understanding of how a user performed during a given task. We can then utilize this information to provide the user with real-time feedback on his task using mixed reality devices, such as the Microsoft HoloLens, that can guide him and prevent him from doing mistakes.
- Master Thesis - Semester Project - GANs - Neural Style Transfer - 3D Modeling in Blender
Please send your CV and master course grades to Sophokles Ktistakis (ktistaks@ethz.ch)
Please send your CV and master course grades to Sophokles Ktistakis (ktistaks@ethz.ch)