Register now After registration you will be able to apply for this opportunity online.
This opportunity is not published. No applications will be accepted.
Physics-aware 3D Human Pose Estimation for Mixed Reality
How do we design differentiable methods for full-body pose estimation from sparse sensors to create physically plausible results?
Keywords: Computer vision, Deep Learning, Metaverse, Human Pose Estimation, Robotics
Today's Mixed Reality head-mounted displays track the user's head pose in world space as well as the user's hands for interaction in both Augmented Reality and Virtual Reality scenarios. While this is adequate to support user input, it unfortunately limits users' virtual representations to just their upper bodies. Full-body human pose estimation using only headset and controllers has attracted much attention recently. However, existing methods usually directly predict the human pose without the consideration of physical constraints, leading to physically implausible results. Some recent work tried to solve this problem by combining non-differentiable simulators with reinforcement learning, but they usually do not generalize well to unseen motions.
Today's Mixed Reality head-mounted displays track the user's head pose in world space as well as the user's hands for interaction in both Augmented Reality and Virtual Reality scenarios. While this is adequate to support user input, it unfortunately limits users' virtual representations to just their upper bodies. Full-body human pose estimation using only headset and controllers has attracted much attention recently. However, existing methods usually directly predict the human pose without the consideration of physical constraints, leading to physically implausible results. Some recent work tried to solve this problem by combining non-differentiable simulators with reinforcement learning, but they usually do not generalize well to unseen motions.
The goal of this project is to explore and develop differentiable physics-based methods for full body human pose estimation methods from sparse motion sensing.
**Requirements**
- Solid background in kinematic and dynamic modeling.
- Fluent programming skills in Python.
- Experience with deep learning libraries (e.g., Pytorch).
- Prior experience with physics simulators is a plus.
The goal of this project is to explore and develop differentiable physics-based methods for full body human pose estimation methods from sparse motion sensing.
**Requirements**
- Solid background in kinematic and dynamic modeling. - Fluent programming skills in Python. - Experience with deep learning libraries (e.g., Pytorch). - Prior experience with physics simulators is a plus.