Register now After registration you will be able to apply for this opportunity online.
This opportunity is not published. No applications will be accepted.
Exploring GPT-4 for visual assistance
One of the new disciplines at the upcoming CYBATHLON is the vision assistance race. Visual impaired people are severely limited in their autonomy of completing many daily tasks. Available vision aids are limited to specific domains, such as reading text out loud, but fail to generalize. Smart vision assistive technologies could provide more intuitive, comprehensive and reliable support in daily tasks.
The CYBATHLON challenges contain a variety of daily situations, such as shopping, finding a free seat or ringing the correct doorbell. The goal is to develop an assistive device capable of fulfilling all challenges.
In this project, we want to explore the use of large language models, such as GPT-4, for solving various challenges in the vision assistance race. An example for this is the blind use of touch screens, whose user interfaces can differ widely depending on the application. Interpreting menu structures and finding the correct items to select is a significant challenge here.
This project will include a literature review and an exploration of which particular problems could be solved using GPT-4. The student should design prompts to solve these problems and setup experiments to demonstrate their robustness.
In this project, we want to explore the use of large language models, such as GPT-4, for solving various challenges in the vision assistance race. An example for this is the blind use of touch screens, whose user interfaces can differ widely depending on the application. Interpreting menu structures and finding the correct items to select is a significant challenge here. This project will include a literature review and an exploration of which particular problems could be solved using GPT-4. The student should design prompts to solve these problems and setup experiments to demonstrate their robustness.
- Literature research
- Investigation into existing approaches
- Adapting existing or developing new approaches for the CYBATHLON challenge
- Evaluation in real-world experiments
- Literature research - Investigation into existing approaches - Adapting existing or developing new approaches for the CYBATHLON challenge - Evaluation in real-world experiments
- Highly motivated and independent student.
- Programming skills in Python
- Experience with machine learning is a plus
- Enrolled at ETH Zurich
- Highly motivated and independent student. - Programming skills in Python - Experience with machine learning is a plus - Enrolled at ETH Zurich
To apply, please send your CV, transcript of records and a few sentences describing your motivation for this project to: Patrick Pfreundschuh (patrick.pfreundschuh@mavt.ethz.ch), Kiavash Fathi (fath@zhaw.ch) and Cornelius von Einem (cornelius.voneinem@mavt.ethz.ch)
To apply, please send your CV, transcript of records and a few sentences describing your motivation for this project to: Patrick Pfreundschuh (patrick.pfreundschuh@mavt.ethz.ch), Kiavash Fathi (fath@zhaw.ch) and Cornelius von Einem (cornelius.voneinem@mavt.ethz.ch)
Cybathlon Student Project GPT-4.pdf
