Advancing Autonomous Navigation: Dynamic Space Mapping and Scene Segmentation in Public Environments

**Background** This project aims to create a dynamic map that provides a high-level understanding of public space usage. Specifically, we seek to identify areas such as accessible areas, crowded areas, queues, and pathways within the spaces we navigate. Existing research mainly focuses on (i) crowd segmentation via fully observable camera-based sensor setups, or (ii) basic clustering algorithms that lack the necessary semantic segmentation to comprehend underlying crowd behaviors for navigation purposes. To bridge this gap, we are developing novel scene segmentation techniques that extract space usage information from Lidar and point cloud-based sensor setups through advanced deep learning methods. We utilize both real-world data from crowded public spaces and simulated environments using the NVIDIA Isaac Sim platform to address the scarcity of relevant labeled datasets. The project goals include advancing public space segmentation methods and constructing relevant public scenarios for navigation using NVIDIA Isaac Sim. This is an exploratory field with no standardized metrics, aiming to construct maps that mimic human understanding of public environments to enhance robot navigation. Simulation data allows us to obtain high-quality data with perfect ground truth, closely resembling real-world scenarios of interest. Understanding the dynamic space in public environments is essential for autonomous navigation within human-centric areas. By comprehending the dynamic space around us, we can identify which areas are more accessible or crowded, and more easily integrate into crowd pathways by understanding their flow. This high-level, slowly changing information, combined with immediate, lower-level details about our surroundings and their dynamics, should enable better navigation strategies than what currently exists. Currently, the predominant strategy for various autonomous delivery robots is to stop and wait if any dynamic obstacle is detected in their immediate vicinity. This approach is not viable for personal mobility vehicles with a human onboard. Therefore, a better integration into the public space environment is required for smart personal mobility vehicles, such as wheelchairs. **Your Tasks** - Literature review of related works. - Familiarizing yourself with the current simulation and pipeline setup. - Improving ground truth data and engineering relevant metrics. - Selecting or developing, training, and validating a state-of-the-art deep learning segmentation model. - Enhancing available training and validation datasets with simulated data and potentially improving the simulated environment in NVIDIA Isaac Sim Not all tasks need to be necessarily tackled. The thesis can either focus on specific points in more detail or cover the entire pipeline, improving it with slightly less depth on each task. **Your Benefits** - Gain unique access and first-hand experience in one of the leading institutions on long-term health management - At the Swiss Paraplegic Center at Nottwil. - Work in a rapidly developing and growing field of robotics for social navigation support. - Gain experience with NVIDIA Isaac Sim, a leading simulation software for the near future. - Become familiar with state-of-the-art deep learning models for image segmentation. - Conduct research on a server PC equipped with 2xRTX4090 GPUs for fast processing. - Improve a pipeline that can be tested in real-time, real-world scenarios in public environments. **Your Profile** - Strong coding skills, particularly in Python. - Previous experience with machine learning and deep learning. - Knowledge of virtual environments (conda / docker) - Interest in or previous experience with simulator software such as Gazebo, Mujoco, or ideally NVIDIA Isaac Sim. - Ideally, knowledge and some experience with Robot Operating System (ROS). - Structured and reliable working style - Ability to work independently on a challenging topic - Availability for at least 6 months.

**Background**
This project aims to create a dynamic map that provides a high-level understanding of public space usage. Specifically, we seek to identify areas such as accessible areas, crowded areas, queues, and pathways within the spaces we navigate. Existing research mainly focuses on (i) crowd segmentation via fully observable camera-based sensor setups, or (ii) basic clustering algorithms that lack the necessary semantic segmentation to comprehend underlying crowd behaviors for navigation purposes. To bridge this gap, we are developing novel scene segmentation techniques that extract space usage information from Lidar and point cloud-based sensor setups through advanced deep learning methods.
We utilize both real-world data from crowded public spaces and simulated environments using the NVIDIA Isaac Sim platform to address the scarcity of relevant labeled datasets. The project goals include advancing public space segmentation methods and constructing relevant public scenarios for navigation using NVIDIA Isaac Sim. This is an exploratory field with no standardized metrics, aiming to construct maps that mimic human understanding of public environments to enhance robot navigation. Simulation data allows us to obtain high-quality data with perfect ground truth, closely resembling real-world scenarios of interest.
Understanding the dynamic space in public environments is essential for autonomous navigation within human-centric areas. By comprehending the dynamic space around us, we can identify which areas are more accessible or crowded, and more easily integrate into crowd pathways by understanding their flow. This high-level, slowly changing information, combined with immediate, lower-level details about our surroundings and their dynamics, should enable better navigation strategies than what currently exists.
Currently, the predominant strategy for various autonomous delivery robots is to stop and wait if any dynamic obstacle is detected in their immediate vicinity. This approach is not viable for personal mobility vehicles with a human onboard. Therefore, a better integration into the public space environment is required for smart personal mobility vehicles, such as wheelchairs.

**Your Tasks**
- Literature review of related works.
- Familiarizing yourself with the current simulation and pipeline setup.
- Improving ground truth data and engineering relevant metrics.
- Selecting or developing, training, and validating a state-of-the-art deep learning segmentation model.
- Enhancing available training and validation datasets with simulated data and potentially improving the simulated environment in NVIDIA Isaac Sim

Not all tasks need to be necessarily tackled. The thesis can either focus on specific points in more detail or cover the entire pipeline, improving it with slightly less depth on each task.

**Your Benefits**
- Gain unique access and first-hand experience in one of the leading institutions on long-term health management - At the Swiss Paraplegic Center at Nottwil.
- Work in a rapidly developing and growing field of robotics for social navigation support.
- Gain experience with NVIDIA Isaac Sim, a leading simulation software for the near future.
- Become familiar with state-of-the-art deep learning models for image segmentation.
- Conduct research on a server PC equipped with 2xRTX4090 GPUs for fast processing.
- Improve a pipeline that can be tested in real-time, real-world scenarios in public environments.

**Your Profile**
- Strong coding skills, particularly in Python.
- Previous experience with machine learning and deep learning.
- Knowledge of virtual environments (conda / docker)
- Interest in or previous experience with simulator software such as Gazebo, Mujoco, or ideally NVIDIA Isaac Sim.
- Ideally, knowledge and some experience with Robot Operating System (ROS).
- Structured and reliable working style
- Ability to work independently on a challenging topic
- Availability for at least 6 months.

The end goal of this project is to improve navigation in highly dynamic environments and enhance the understanding of assistive robots in public spaces. To achieve this, the project aims to improve a pipeline that takes pointcloud-based data, people detections, and tracks (including orientation and velocity) as inputs, and outputs a dynamic map of the environment that provides a high-level understanding of the usage of surrounding space. There are multiple steps in this pipeline that can be enhanced: 1. Ground Truth Generation: Create algorithms and metrics for generating ground truth data that closely resemble human understanding of dynamics in public environments. 2. Deep Learning Model Selection: Select or develop a state-of-the-art deep learning model for space (image) segmentation. 3. Scenario Creation and Simulation Improvement in NVIDIA Isaac Sim: Develop and refine relevant scenarios in NVIDIA Isaac Sim to provide high-quality ground truth data for training and validating the selected model. 4. Global Dynamic Map Integration: Merge local space segmentations into a comprehensive global dynamic map of the environment. Improving these aspects will advance our ability to understand and navigate public spaces more effectively.

The end goal of this project is to improve navigation in highly dynamic environments and enhance the understanding of assistive robots in public spaces. To achieve this, the project aims to improve a pipeline that takes pointcloud-based data, people detections, and tracks (including orientation and velocity) as inputs, and outputs a dynamic map of the environment that provides a high-level understanding of the usage of surrounding space. There are multiple steps in this pipeline that can be enhanced:
1. Ground Truth Generation: Create algorithms and metrics for generating ground truth data that closely resemble human understanding of dynamics in public environments.
2. Deep Learning Model Selection: Select or develop a state-of-the-art deep learning model for space (image) segmentation.
3. Scenario Creation and Simulation Improvement in NVIDIA Isaac Sim: Develop and refine relevant scenarios in NVIDIA Isaac Sim to provide high-quality ground truth data for training and validating the selected model.
4. Global Dynamic Map Integration: Merge local space segmentations into a comprehensive global dynamic map of the environment.
Improving these aspects will advance our ability to understand and navigate public spaces more effectively.

Main Supervisor: Dominik Wojcikiewicz (SCAI-Lab, ETHZ | SPZ) Host: Dr. Diego Paez (SCAI-Lab, ETHZ | SPZ) Please send your CV and latest transcript of records from my studies to Dominik Wojcikiewicz (dominik.wojcikiewicz@hest.ethz.ch)

Main Supervisor: Dominik Wojcikiewicz (SCAI-Lab, ETHZ | SPZ)
Host: Dr. Diego Paez (SCAI-Lab, ETHZ | SPZ)

Please send your CV and latest transcript of records from my studies to Dominik Wojcikiewicz (dominik.wojcikiewicz@hest.ethz.ch)