Register now After registration you will be able to apply for this opportunity online.
This opportunity is not published. No applications will be accepted.
Interactive exploration for object-level mapping
With this project, we want to go beyond the classical exploration approaches for mapping of unknown environments by allowing the agent to interact with the objects in the scene. You will implement exploration and interaction strategies for a mobile robot in a simulation environment with the goal to perform faster and more accurate object-level mapping using interactive exploration.
The awareness of individual object instances is an important requirement for successful manipulation of objects through robots. Interaction with the scene can help in effectively identifying object instances in the environment [1] when estimating object shape from pure observation remains ambiguous. This behavior can also be observed in humans, who intuitively enhance their visual perception through manipulation of objects to better infer their properties or simply increase and optimize their field of view (e.g. grasp and rotate an object or push it aside to resolve occlusions). Enabling interactive behavior for exploration of unknown spaces is therefore expected to lead to faster and more accurate object-level mapping of the environment.
Within the project we will investigate how we can design/learn interactive exploration policies for object-level mapping [2] in a simulation environment [3] containing simple objects. We will use affordances [4] as a foundation for robot -dependent action selection. The project can be, but doesn’t have to be based on a pre-existing project using Reinforcement Learning. The final goal of the project is to validate our hypothesis that interactive exploration leads to better object-level mapping of the environment.
References:
[1] J. Bohg et al., "Interactive Perception: Leveraging Action in Perception and Perception in Action," in IEEE Transactions on Robotics, vol. 33, no. 6, pp. 1273-1291, Dec. 2017
[2] Grinvald, Margarita, et al. "TSDF++: A multi-object formulation for dynamic object tracking and reconstruction." 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.
[3] Kiana Ehsani et al., “ManipulaTHOR: A Framework for Visual Object Manipulation”, in Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[4] Horton, Thomas E., Arpan Chakraborty, and Robert St Amant. "Affordances for robots: a brief survey." AVANT. Pismo Awangardy Filozoficzno-Naukowej 2 (2012): 70-84.
[5] Nagarajan, Tushar, and Kristen Grauman. "Learning affordance landscapes for interaction exploration in 3d environments." Advances in Neural Information Processing Systems 33 (2020): 2005-2015.
[6] Jayaraman, Dinesh, and Kristen Grauman. "Learning to look around: Intelligently exploring unseen environments for unknown tasks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
[7] Ramakrishnan, Santhosh K., Dinesh Jayaraman, and Kristen Grauman. "An exploration of embodied visual exploration." International Journal of Computer Vision 129.5 (2021): 1616-1649.
The awareness of individual object instances is an important requirement for successful manipulation of objects through robots. Interaction with the scene can help in effectively identifying object instances in the environment [1] when estimating object shape from pure observation remains ambiguous. This behavior can also be observed in humans, who intuitively enhance their visual perception through manipulation of objects to better infer their properties or simply increase and optimize their field of view (e.g. grasp and rotate an object or push it aside to resolve occlusions). Enabling interactive behavior for exploration of unknown spaces is therefore expected to lead to faster and more accurate object-level mapping of the environment.
Within the project we will investigate how we can design/learn interactive exploration policies for object-level mapping [2] in a simulation environment [3] containing simple objects. We will use affordances [4] as a foundation for robot -dependent action selection. The project can be, but doesn’t have to be based on a pre-existing project using Reinforcement Learning. The final goal of the project is to validate our hypothesis that interactive exploration leads to better object-level mapping of the environment.
References:
[1] J. Bohg et al., "Interactive Perception: Leveraging Action in Perception and Perception in Action," in IEEE Transactions on Robotics, vol. 33, no. 6, pp. 1273-1291, Dec. 2017
[2] Grinvald, Margarita, et al. "TSDF++: A multi-object formulation for dynamic object tracking and reconstruction." 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.
[3] Kiana Ehsani et al., “ManipulaTHOR: A Framework for Visual Object Manipulation”, in Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[4] Horton, Thomas E., Arpan Chakraborty, and Robert St Amant. "Affordances for robots: a brief survey." AVANT. Pismo Awangardy Filozoficzno-Naukowej 2 (2012): 70-84.
[5] Nagarajan, Tushar, and Kristen Grauman. "Learning affordance landscapes for interaction exploration in 3d environments." Advances in Neural Information Processing Systems 33 (2020): 2005-2015.
[6] Jayaraman, Dinesh, and Kristen Grauman. "Learning to look around: Intelligently exploring unseen environments for unknown tasks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
[7] Ramakrishnan, Santhosh K., Dinesh Jayaraman, and Kristen Grauman. "An exploration of embodied visual exploration." International Journal of Computer Vision 129.5 (2021): 1616-1649.
- Literature review
- Familiarize with simulation environment for interactive exploration (ManipulaTHOR [3]).
- Design of a suitable exploration strategy for interactive mapping using ground truth affordances and segmentation.
- Include learning framework for affordance and segmentation.
- Evaluation of the proposed strategy.
- Literature review - Familiarize with simulation environment for interactive exploration (ManipulaTHOR [3]). - Design of a suitable exploration strategy for interactive mapping using ground truth affordances and segmentation. - Include learning framework for affordance and segmentation. - Evaluation of the proposed strategy.
- High motivation and interest in the topic.
- Methodological and goal-oriented approach.
- Good programming skills in Python/C++.
- Experience with a simulation environment or learning based methods is a plus.
- High motivation and interest in the topic. - Methodological and goal-oriented approach. - Good programming skills in Python/C++. - Experience with a simulation environment or learning based methods is a plus.
Paula Wulkop (paula.wulkop@mavt.ethz.ch),
Antonia Hüfner (ahuefner@ethz.ch)
Paula Wulkop (paula.wulkop@mavt.ethz.ch), Antonia Hüfner (ahuefner@ethz.ch)