Enhanced Human Pose Estimation for Augmented Reality

Augmented reality applications like virtual fitting rooms can benefit from human estimation as one of the most advanced methods of detecting and recognizing the position of a human body in space. This can be used in e-commerce where shoppers struggle to fit their clothes before buying. Human pose estimation can be applied to track key points on the human body and pass this data to the augmented reality engine that will fit clothes on the user. This can be applied to any body part and type of clothes, or even face masks. The essence of the technology lies in detecting points of interest on the limbs, joints, and even the face of a human. These key points are used to produce a 2D or 3D representation of a human body model. Whether we deal with a fitness app, an app for rehabilitation, face masks, or surveillance, real-time processing is highly required. Of course, the performance of the model will depend on the chosen algorithm and hardware, but the majority of existing open-source models provide quite a long response time. In the opposite scenario, the accuracy suffers. So, is it possible to improve existing 3D human pose estimation models to achieve acceptable accuracy with real-time processing?

Augmented reality applications like virtual fitting rooms can benefit from human estimation as one of the most advanced methods of detecting and recognizing the position of a human body in space. This can be used in e-commerce where shoppers struggle to fit their clothes before buying. Human pose estimation can be applied to track key points on the human body and pass this data to the augmented reality engine that will fit clothes on the user. This can be applied to any body part and type of clothes, or even face masks.
The essence of the technology lies in detecting points of interest on the limbs, joints, and even the face of a human. These key points are used to produce a 2D or 3D representation of a human body model.
Whether we deal with a fitness app, an app for rehabilitation, face masks, or surveillance, real-time processing is highly required. Of course, the performance of the model will depend on the chosen algorithm and hardware, but the majority of existing open-source models provide quite a long response time. In the opposite scenario, the accuracy suffers. So, is it possible to improve existing 3D human pose estimation models to achieve acceptable accuracy with real-time processing?

The purpose of this project is to enhance a standard vision based pose estimation system (installed in the facility of an industrial partner) with novel sensors, such as IMU (inertial module), ultra-wideband, radars, and more. The project includes the necessity of a custom dataset collection, wearable sensor node design, algorithm exploration, and interfacing with the existing industrial system. Therefore, it is the perfect project for students interested in working on the full implementation stack, from hardware, to sensor selection and algorithms.

The purpose of this project is to enhance a standard vision based pose estimation system (installed in the facility of an industrial partner) with novel sensors, such as IMU (inertial module), ultra-wideband, radars, and more. The project includes the necessity of a custom dataset collection, wearable sensor node design, algorithm exploration, and interfacing with the existing industrial system. Therefore, it is the perfect project for students interested in working on the full implementation stack, from hardware, to sensor selection and algorithms.

Dr. Tommaso Polonelli (tommaso.polonelli@pbl.ee.ethz.ch)

Dr. Tommaso Polonelli (tommaso.polonelli@pbl.ee.ethz.ch)