Register now After registration you will be able to apply for this opportunity online.
This opportunity is not published. No applications will be accepted.
Closing Sim-to-Real Visual Domain Gap for Transparent and Reflective Objects
Robots need to manipulate a wide range of unknown objects, from transparent to shiny surfaces. The goal of this project is to investigate learning techniques to bridge the visual domain gap between high-fidelity rendered scenes and real-world images for scene understanding.
While humans are adept at differentiating an object and its texture properties, robots are easily fooled by common household objects such as glass tables, shiny toasters, or plastic bottles. The limitations of depth sensing on current hardware limit real-world data collection for these objects. This affects the generalization of the learned models to different lighting conditions and scene variations [1]. Prior works [1, 2] rely on stereo images based sparse cost volume computation or real-world. Recent advances in physically-based rendering show promise in collecting high-fidelity renderings of diverse synthetic scenes and creating large datasets for learning [2]. The goal of this project is to leverage these advances and investigate how to close the visual domain gap for deploying learned models for the 3D shape estimation of transparent or shiny objects.
References:
- Sajjan, Shreeyak, et al. "Clear grasp: 3d shape estimation of transparent objects for manipulation." ICRA (2020).
- Kollar, Thomas, et al. "Simnet: Enabling robust unknown object manipulation from pure synthetic data via stereo." CoRL (2022).
While humans are adept at differentiating an object and its texture properties, robots are easily fooled by common household objects such as glass tables, shiny toasters, or plastic bottles. The limitations of depth sensing on current hardware limit real-world data collection for these objects. This affects the generalization of the learned models to different lighting conditions and scene variations [1]. Prior works [1, 2] rely on stereo images based sparse cost volume computation or real-world. Recent advances in physically-based rendering show promise in collecting high-fidelity renderings of diverse synthetic scenes and creating large datasets for learning [2]. The goal of this project is to leverage these advances and investigate how to close the visual domain gap for deploying learned models for the 3D shape estimation of transparent or shiny objects.
References: - Sajjan, Shreeyak, et al. "Clear grasp: 3d shape estimation of transparent objects for manipulation." ICRA (2020). - Kollar, Thomas, et al. "Simnet: Enabling robust unknown object manipulation from pure synthetic data via stereo." CoRL (2022).
- Setup a learning environment for synthetic data generation using NVIDIA Omniverse / Isaac Sim
- Literature review on online learning techniques, such as automatic domain randomization, and recent advances in implicit representations, such as neural fields
- Designing a method to perform 3D shape estimation using only simulated data
- Show generalization of the learned architecture on real-world data
- Setup a learning environment for synthetic data generation using NVIDIA Omniverse / Isaac Sim - Literature review on online learning techniques, such as automatic domain randomization, and recent advances in implicit representations, such as neural fields - Designing a method to perform 3D shape estimation using only simulated data - Show generalization of the learned architecture on real-world data
- Highly motivated and autonomous student
- Prior knowledge and experience in 3D Vision
- Programming experience in Python and PyTorch
- Highly motivated and autonomous student - Prior knowledge and experience in 3D Vision - Programming experience in Python and PyTorch
While submitting your application, please include your CV including relevant experience and the latest transcript.
- Mayank Mittal (mittalma@ethz.ch)
- Vaishakh Patil (patilv@ethz.ch)
- René Zurbrügg (zrene@ethz.ch)
While submitting your application, please include your CV including relevant experience and the latest transcript.
