Register now After registration you will be able to apply for this opportunity online.
This opportunity is not published. No applications will be accepted.
Affordances Perception for Autonomous Manipulation
Nowadays, there is huge potential for robots to automate tasks in households, factories, health care and agriculture. Tasks are generally accomplished through interaction of a manipulator with the surrounding environment. Affordances are an emerging topic that addresses the problem of perceiving obj
Keywords: Perception for Robotics, Affordances
Affordances for robotics are a dense representation of the object functions: namely they map each pixel/point to how this could be interacted with, as shown for example in the 3D AffordanceNet dataset [2]. In order to reduce the complexity of the task, we want to consider movements along planar or rotary joints:
Pushing/sliding objects on a table
Open drawer/door/lid of box with joint
Traditional approaches learn a mapping to a set of predefined actions like push and pull [1] in a self supervised manner through random interaction with the environment. In order to overcome the limitations of this approach, one could directly learn a vector field which encompasses all action types and deploy “smarter” interaction policies [4]. These predictions could be later fused over time to create a consistent dense affordance map that can be used for closed loop manipulation control [3].
In this project the student will set up a photo-realistic physics simulation [5] where a force can be applied to the objects in the scene. The simulation setup will be used to create a synthetic dataset for training a deep network that takes as an input the visually observed scene and the ground truth semantic segmentation mask and learns how the object can be moved by sampling interactions with the objects.
**References**
- [1] Where2Act, Mo et al., 2021, https://arxiv.org/pdf/2101.02692.pdf
- [2] 3D AffordanceNet: A Benchmark for Visual Object Affordance Understanding. Deng et al., CVPR2021, http://arxiv.org/abs/2103.16397
- [3] Learning Affordance Landscapes for Interaction Exploration in 3D Environments, Nagarajan et al., NeurIPS 2020, https://arxiv.org/pdf/2008.09241.pdf
- [4] Geometric Affordance Perception: Leveraging Deep 3D Saliency With the Interaction Tensor, Ruiz et al., Frontiers in Neurorobotics 2020, https://www.frontiersin.org/articles/10.3389/fnbot.2020.00045/full
- [5] Nvidia Omniverse Platform https://cutt.ly/SnByPKe
Affordances for robotics are a dense representation of the object functions: namely they map each pixel/point to how this could be interacted with, as shown for example in the 3D AffordanceNet dataset [2]. In order to reduce the complexity of the task, we want to consider movements along planar or rotary joints: Pushing/sliding objects on a table Open drawer/door/lid of box with joint Traditional approaches learn a mapping to a set of predefined actions like push and pull [1] in a self supervised manner through random interaction with the environment. In order to overcome the limitations of this approach, one could directly learn a vector field which encompasses all action types and deploy “smarter” interaction policies [4]. These predictions could be later fused over time to create a consistent dense affordance map that can be used for closed loop manipulation control [3]. In this project the student will set up a photo-realistic physics simulation [5] where a force can be applied to the objects in the scene. The simulation setup will be used to create a synthetic dataset for training a deep network that takes as an input the visually observed scene and the ground truth semantic segmentation mask and learns how the object can be moved by sampling interactions with the objects.
**References**
- [1] Where2Act, Mo et al., 2021, https://arxiv.org/pdf/2101.02692.pdf - [2] 3D AffordanceNet: A Benchmark for Visual Object Affordance Understanding. Deng et al., CVPR2021, http://arxiv.org/abs/2103.16397 - [3] Learning Affordance Landscapes for Interaction Exploration in 3D Environments, Nagarajan et al., NeurIPS 2020, https://arxiv.org/pdf/2008.09241.pdf - [4] Geometric Affordance Perception: Leveraging Deep 3D Saliency With the Interaction Tensor, Ruiz et al., Frontiers in Neurorobotics 2020, https://www.frontiersin.org/articles/10.3389/fnbot.2020.00045/full - [5] Nvidia Omniverse Platform https://cutt.ly/SnByPKe
- Literature review about affordances
- Photorealistic simulator setup for data collection
- Training and validation of perception model
- Real world experiments with real data
- Literature review about affordances - Photorealistic simulator setup for data collection - Training and validation of perception model - Real world experiments with real data
- Highly motivated and independent student
- Interest in perception and robotics
- Good programming skills in Python and C++
- Experience with ROS or Git is a plus
- Highly motivated and independent student - Interest in perception and robotics - Good programming skills in Python and C++ - Experience with ROS or Git is a plus