Max Planck ETH Center for Learning SystemsAcronym | MPG ETH CLS | Homepage | http://learning-systems.org/ | Country | [nothing] | ZIP, City | | Address | | Phone | | Type | Alliance | Current organization | Max Planck ETH Center for Learning Systems | Members | |
Open Opportunities3D hand pose forecasting is a new benchmark introduced by HoloAssist [1]. Existing action forecasting work mostly focuses on providing semantic labels of future actions and does not provide explicit 3D guidance on hand poses. Predicting 3D hand poses can be useful for various applications, and it can augment instructions and spatially guide users in different tasks. In this benchmark, we take 3 seconds inputs similar to other 3D body location forecasting literature and forecast the continuous 3D hand poses for the next 0.5, 1.0, and 1.5 seconds. The evaluation metric is the average of mean per joint position error over time in centimeters compared to ground truth. To have a proper evaluation metric that can help 3D action guidance, we remove the mistakes from the action sequences and only forecast 3D hand pose for the correct labels.
[1] Wang, X., Kwon, T., Rad, M., Pan, B., Chakraborty, I., Andrist, S., ... & Pollefeys, M. (2023). Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 20270-20281). - Computer Vision, Virtual Reality and Related Simulation
- ETH Zurich (ETHZ), Master Thesis, Semester Project
| Action recognition is an essential task in computer vision and has numerous applications in various fields, including robotics, surveillance, and healthcare. The recognition of actions involves the analysis of temporal and spatial information within a video sequence. Current state-of-the-art methods use 3D hand and object poses for action recognition, where the object's corners are commonly used for representation. However, this approach has limitations in accurately modeling the hand-object interaction. In [1], we show that leveraging hand-object contact-map representation helps improve action recognition. However, this representation can be learned implicitly for the task of action recognition.
[1] https://arxiv.org/pdf/2309.10001.pdf - Computer Vision, Virtual Reality and Related Simulation
- ETH Zurich (ETHZ), Master Thesis, Semester Project
| The recent development of LLMs (Large Language Models), such as ChatGPT and Llama, opens up new possibilities for understanding procedural actions. In the past, action recognition was restricted to the classification of visual frames. However, with LLMs, the model can observe the whole action sequence in a more effective way and even predict the future actions [1]. In this project, students will explore how LLMs can improve action recognition in procedural tasks. Specifically, given a high-level procedural task (e.g., making coffee, copying a paper), students will use existing pretrained action recognition models to predict the top 5 actions for each clip and feed them into the LLMs to refine and correct the predicted actions. As a comparison, students will also establish a baseline using simple machine learning and statistical methods to correct actions.
[1] Palm: Predicting Actions through Language Models @ Ego4D Long-Term Action Anticipation Challenge 2023, CVPR'23 workshop
- Computer Vision, Text Processing
- ETH Zurich (ETHZ), Master Thesis, Semester Project
| Reading text manuals to set up and manipulate devices takes a lot of time and is not intuitive when it comes to 3D instruction. Despite the advent of Mixed Reality (MR) devices, 3D instruction is still limited and expensive to set up. In this project, we will develop an app, an adaptive 3D hand guidance system that projects instructional 3D hand poses in MR devices with pre-recorded instructional videos using MR devices. - Computer Vision, Virtual Reality and Related Simulation
- ETH Zurich (ETHZ), Master Thesis, Semester Project
| After a neurological injury (such as stroke), many patients suffer from impairment of the hand and finger function. Clinical assessments aim to measure and quantify those impairments for a better understanding and to specifically target those deficits in rehabilitation. One aspect of hand function, that is not truly understood yet is finger individuation: the ability to move one finger independently of the others. In a previously developed assessment device, we use force sensors attached to a hand module to measure this dexterous skill. This individuation device measures finger flexion (pushing) over different force levels, but the individuation ability in extension (pulling) remains unknown. The aim of this project is to implement an extension assessment (by adapting the existing protocol) and compare as well as test it before its implementation into the clinical routine. - Biomedical Engineering, Clinical Sciences, Electrical and Electronic Engineering, Human Movement and Sports Science, Interdisciplinary Engineering, Mechanical and Industrial Engineering, Neurosciences, Other
- Bachelor Thesis, Internship, Master Thesis, Semester Project
| After a neurological injury (such as stroke), many patients suffer from impairment of the hand and finger function. Clinical assessments aim to measure and quantify those impairments for a better understanding and to specifically target those deficits in rehabilitation. One aspect of hand function, that is not truly understood yet is finger individuation: the ability to move one finger independently of the others. In a previously developed assessment device, we use force sensors attached to a hand module to measure this dexterous skill. This individuation device measures finger flexion (pushing) over different force levels, using a simple user interface. But to facilitate the measurement process and increase comprehension for cognitively impaired patients, we need to improve the assessment visualization and execution. - Computer Software, Electrical and Electronic Engineering
- Bachelor Thesis, Internship, Master Thesis, Semester Project
| After a neurological injury (such as stroke), many patients suffer from impairment of the hand and finger function. Clinical assessments aim to measure and quantify those impairments for a better understanding and to specifically target those deficits in rehabilitation. One aspect of hand function, that is not truly understood yet is finger individuation: the ability to move one finger independently of the others. In a previously developed assessment device, we use force sensors attached to a hand module to measure this dexterous skill. This individuation device will be used in a clinical setting to measure neurological patients. But before it can routinely be put into practice, its reliability (in a test-retest setting) and validity must be proven. - Biomedical Engineering, Clinical Sciences, Human Movement and Sports Science, Neurosciences, Other, Public Health and Health Services
- Bachelor Thesis, Internship, Master Thesis, Semester Project
| Ski touring provides a unique and immersive outdoor experience, but the ascent can impose a considerable amount of strain on the body, especially for novices, elderly, or people with disabilities. The objective of this master thesis is to redesign an existing concept and functional model of an electric ski touring device that supports hill ascents, aiming to enhance the ski touring experience for individuals with lower fitness levels by making it less physically demanding and more enjoyable. The current model must be optimized with respect to weight, function, energy consumption, and usability (donning/doffing). After successful fabrication and testing, first steps shall be performed to identify intellectual property and market needs, and finally plan the commercialization of the e-touring ski. - Engineering and Technology
- Master Thesis
| Implementation of a localization and state estimation system for a special ground robot using IMU and 2D LiDAR. The ground robot is part of a Gravity offload system for testing of spacecraft solar arrays (which are designed for microgravity) on earth. - Intelligent Robotics
- Semester Project
| The goal of this project is to use language prompts to help find object parts in 3D. - Computer Vision
- Master Thesis, Semester Project
|
|