Register now After registration you will be able to apply for this opportunity online.
LLMxRobot: Language-Guided Control for Autonomous Driving
Develop a framework to integrate Large Language Models (LLMs) for interacting with
robotic cars leveraging the Jetson Orin AGX and the ForzaETH robotic platform.
Traditional autonomous driving approaches focus on data-driven learning methods, but re-
cent work emphasizes a knowledge-driven approach, particularly with the use of LLMs.
LLMxRobot aims to explore how LLMs, integrated locally on embedded computing plat-
forms, can effectively shape robotic control behaviors through natural language while rea-
soning based on real-time sensor inputs. The challenge is to make LLMs both efficient in
terms of latency and powerful enough to influence driving strategies in real-world environ-
ments.
The proposed system will be applied to autonomous driving platforms, specifically focusing
on real-time decision-making and control interactions.
Experience with Python, PyTorch, ROS1/2, and Huggingface. Knowledge of LLMs and
robotics is highly recommended.
Traditional autonomous driving approaches focus on data-driven learning methods, but re- cent work emphasizes a knowledge-driven approach, particularly with the use of LLMs. LLMxRobot aims to explore how LLMs, integrated locally on embedded computing plat- forms, can effectively shape robotic control behaviors through natural language while rea- soning based on real-time sensor inputs. The challenge is to make LLMs both efficient in terms of latency and powerful enough to influence driving strategies in real-world environ- ments. The proposed system will be applied to autonomous driving platforms, specifically focusing on real-time decision-making and control interactions. Experience with Python, PyTorch, ROS1/2, and Huggingface. Knowledge of LLMs and robotics is highly recommended.
The primary goal is to build and test an LLM-based framework for autonomous driving that
operates locally, shaping behavior and reasoning in real-time. Tasks include:
• Integrating LLMs to shape driving constraints and rewards.
• Developing an efficient LLMxMPC (Model Predictive Control) framework.
• Optimizing LLMs for local processing on the Jetson Orin AGX.
• Conducting fine-tuning and performance evaluation in various driving scenarios.
The primary goal is to build and test an LLM-based framework for autonomous driving that operates locally, shaping behavior and reasoning in real-time. Tasks include: • Integrating LLMs to shape driving constraints and rewards. • Developing an efficient LLMxMPC (Model Predictive Control) framework. • Optimizing LLMs for local processing on the Jetson Orin AGX. • Conducting fine-tuning and performance evaluation in various driving scenarios.