Register now After registration you will be able to apply for this opportunity online.
Soft object reconstruction
This project consists of reconstructing soft object along with their appearance, geometry, and physical properties from image data for inclusion in reinforcement learning frameworks for manipulation tasks.
Keywords: Computer Vision, Structure from Motion, Image-based Reconstruction, Physics-based Reconstruction
As 3D reconstruction [2,3], real-time data-driven rendering [4,5], and learning-based control technologies [6,7] are becoming more mature, recent efforts in reinforcement learning are moving towards end-to-end policies that directly consume images in order to generate control commands [8]. However, many of the simulated environments are limited to a composition of rigid objects. In recent years, the inclusion of differentiable particle-based simulation borrowed from computer graphics has enabled the inclusion of non-rigid or even fluid elements. Ideally, we can generate such representations from real world data in order to extend data-driven world simulators to arbitrary new objects with complex physical behavior.
The present thesis focuses on this problem and aims at reconstructing soft objects in terms of their geometry, appearance, and physical behavior. The goal is to make use of the Material Point Method (MPM) in combination with vision-based cues and physical priors in order to reconstruct accurate 3D models of soft objects. The developed models will finally be included into an RL learning environment such as Isaac-Gym in order to train novel manipulation policies for soft objects.
The proposed thesis will be conducted at the Robotics and AI Institute, a new top-notch partner institute of Boston Dynamics pushing the boundaries of control and perception in robotics. Selection is highly competitive. Potential candidates are invited to submit their CV and grade sheet, after which students will be invited to an on-site interview.
[1] Modeling of Deformable Objects for Robotic Manipulation: A Tutorial and Review, Front. Robot. AI, 7, 2020
[2] Global Structure-from-Motion Revisited, ECCV 2024
[3] MASt3R-SLAM: Real-Time Dense SLAM with 3D Reconstruction Priors, CVPR 2025
[4] NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, CVPR 2020
[5] 3D Gaussian Splatting for Real-Time Radiance Field Rendering, SIGGRAPH 2023
[6] Learning to walk in minutes using massively parallel deep reinforcement learning, CoRL 2022
[7] Champion-Level Drone Racing Using Deep Reinforcement Learning, Nature, 2023
[8] π0: A Vision-Language-Action Flow Model for General Robot Control, Arxiv: https://arxiv.org/abs/2410.24164
As 3D reconstruction [2,3], real-time data-driven rendering [4,5], and learning-based control technologies [6,7] are becoming more mature, recent efforts in reinforcement learning are moving towards end-to-end policies that directly consume images in order to generate control commands [8]. However, many of the simulated environments are limited to a composition of rigid objects. In recent years, the inclusion of differentiable particle-based simulation borrowed from computer graphics has enabled the inclusion of non-rigid or even fluid elements. Ideally, we can generate such representations from real world data in order to extend data-driven world simulators to arbitrary new objects with complex physical behavior.
The present thesis focuses on this problem and aims at reconstructing soft objects in terms of their geometry, appearance, and physical behavior. The goal is to make use of the Material Point Method (MPM) in combination with vision-based cues and physical priors in order to reconstruct accurate 3D models of soft objects. The developed models will finally be included into an RL learning environment such as Isaac-Gym in order to train novel manipulation policies for soft objects.
The proposed thesis will be conducted at the Robotics and AI Institute, a new top-notch partner institute of Boston Dynamics pushing the boundaries of control and perception in robotics. Selection is highly competitive. Potential candidates are invited to submit their CV and grade sheet, after which students will be invited to an on-site interview.
[1] Modeling of Deformable Objects for Robotic Manipulation: A Tutorial and Review, Front. Robot. AI, 7, 2020
[2] Global Structure-from-Motion Revisited, ECCV 2024
[3] MASt3R-SLAM: Real-Time Dense SLAM with 3D Reconstruction Priors, CVPR 2025
[4] NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, CVPR 2020
[5] 3D Gaussian Splatting for Real-Time Radiance Field Rendering, SIGGRAPH 2023
[6] Learning to walk in minutes using massively parallel deep reinforcement learning, CoRL 2022
[7] Champion-Level Drone Racing Using Deep Reinforcement Learning, Nature, 2023
[8] π0: A Vision-Language-Action Flow Model for General Robot Control, Arxiv: https://arxiv.org/abs/2410.24164
● Literature research
● Design of suitable reconstruction method based on visual data and physical priors
● Dataset collection and testing
● Cross-validation against contact-based reconstruction methods
● Embedding into Isaac-Gym for training novel manipulation policies
● Literature research
● Design of suitable reconstruction method based on visual data and physical priors
● Dataset collection and testing
● Cross-validation against contact-based reconstruction methods
● Embedding into Isaac-Gym for training novel manipulation policies
● Excellent knowledge of Python or C++
● Computer vision experience
● Interest in optimization with physics representations
● Excellent knowledge of Python or C++
● Computer vision experience
● Interest in optimization with physics representations
Laurent Kneip (lkneip@theaiinstitute.com)
Sina Mirrazavi (smirrazavi@theaiinstitute.com)
Please include your CV and up-to-date transcript.
Laurent Kneip (lkneip@theaiinstitute.com)
Sina Mirrazavi (smirrazavi@theaiinstitute.com)
Please include your CV and up-to-date transcript.
202502_Soft_object_reconstruction (1).pdf
