Register now After registration you will be able to apply for this opportunity online.
This opportunity is not published. No applications will be accepted.
Model-Aware Neural Networks for Dynamic Vision Sensor Reconstruction
The primary aim of the project is to develop a modern model-aware neural network to reconstruct imaging data from Dynamic Vision Sensors in collaboration with Sony research Europe.
This opportunity is offered as a master's project.
Keywords: dynamic vision sensors, image reconstruction, variational networks, deep learning
Dynamic Vision Sensors (DVS) are cameras based on the imaging principles of the biological retina [1,2]. Instead of capturing the full frame of a scene, DVS register “ON” and “OFF” events asynchronously whenever a pixel detects logarithmic changes of the temporal contrast. Although DVS sensors are low-power, fast and intrinsically allow to capture High-Dynamic Range (HDR) data, the sensors output a continuous stream of raw event data which only correlates to the image of interest. Therefore, to accurately reconstruct the image, it is necessary to solve the inverse problem posed by the DVS acquisition model.
Model-based neural networks [3,4] allow to enforce such prior knowledge about the DVS acquisition process and train the filtering subnetwork using simulated data. Such class of networks has been demonstrated to work efficiently [3], have limited amount of tunable weights and allow to conduct training using synthetic-only data in the domain of medical imaging [5].
**LITERATURE**
1. P. Lichtsteiner, C. Posch, T. Delbruck, A 128 X 128 120db 30mw asynchronous vision sensor that responds to relative intensity change. 2006 IEEE International Solid State Circuits Conference - Digest of Technical Papers. 2006 Feb.
2. Brandli C, Berner R, Yang M, Liu S, Delbruck T, A 240 × 180 130 dB 3 µs Latency Global Shutter Spatiotemporal Vision Sensor, IEEE Journal of Solid-State Circuits, Volume: 49 , Issue: 10 , Oct. 2014
3. Vishnevskiy V, Walheim J, Kozerke S. Deep variational network for rapid 4D flow MRI reconstruction. Nature Machine Intelligence. 2020 Apr;2(4):228-35.
4. Hammernik K, Klatzer T, Kobler E, Recht MP, Sodickson DK, Pock T, Knoll F. Learning a variational network for reconstruction of accelerated MRI data. Magnetic resonance in medicine. 2018 Jun;79(6):3055-71.
5. Vishnevskiy V, Rau R, Goksel O. Deep variational networks with exponential weighting for learning computed tomography. MICCAI 2019 (pp. 310-318). Springer, Cham.
Dynamic Vision Sensors (DVS) are cameras based on the imaging principles of the biological retina [1,2]. Instead of capturing the full frame of a scene, DVS register “ON” and “OFF” events asynchronously whenever a pixel detects logarithmic changes of the temporal contrast. Although DVS sensors are low-power, fast and intrinsically allow to capture High-Dynamic Range (HDR) data, the sensors output a continuous stream of raw event data which only correlates to the image of interest. Therefore, to accurately reconstruct the image, it is necessary to solve the inverse problem posed by the DVS acquisition model.
Model-based neural networks [3,4] allow to enforce such prior knowledge about the DVS acquisition process and train the filtering subnetwork using simulated data. Such class of networks has been demonstrated to work efficiently [3], have limited amount of tunable weights and allow to conduct training using synthetic-only data in the domain of medical imaging [5].
**LITERATURE**
1. P. Lichtsteiner, C. Posch, T. Delbruck, A 128 X 128 120db 30mw asynchronous vision sensor that responds to relative intensity change. 2006 IEEE International Solid State Circuits Conference - Digest of Technical Papers. 2006 Feb.
2. Brandli C, Berner R, Yang M, Liu S, Delbruck T, A 240 × 180 130 dB 3 µs Latency Global Shutter Spatiotemporal Vision Sensor, IEEE Journal of Solid-State Circuits, Volume: 49 , Issue: 10 , Oct. 2014
3. Vishnevskiy V, Walheim J, Kozerke S. Deep variational network for rapid 4D flow MRI reconstruction. Nature Machine Intelligence. 2020 Apr;2(4):228-35.
4. Hammernik K, Klatzer T, Kobler E, Recht MP, Sodickson DK, Pock T, Knoll F. Learning a variational network for reconstruction of accelerated MRI data. Magnetic resonance in medicine. 2018 Jun;79(6):3055-71.
5. Vishnevskiy V, Rau R, Goksel O. Deep variational networks with exponential weighting for learning computed tomography. MICCAI 2019 (pp. 310-318). Springer, Cham.
The goal of this project is to develop state-of-the-art model-based neural networks for the novel DVS sensor technology in order to learn an efficient reconstruction algorithm and improve the acquisition process.
**TASK LIST**
- Paid full-time Internship at Sony Research Europe (Schlieren Office), for 1 month:
- Become accustomed to Dynamic Vision Sensor technology and familiarize with the software tools and infrastructure;
Master thesis at Sony Research Europe (Schlieren Office) under joint Sony and ETH Zurich supervision, for 6 months:
- Literature review on inverse problems in biomedical imaging.
- Literature review on model-aware neural networks.
- Implement a simple DVS modelling software under the guidance of industry experts.
- Implement and train a model-based neural network.
- Evaluate the network performance in the context of original scene reconstruction.
- Write a report.
**REQUIREMENTS**
- Willingness to learn both a new type of vision sensor and a new machine learning field.
- Knowledge of Matlab and Python with its numerical libraries; familiarity with standard Unix workflows.
- Candidate is assumed to work with applied aspects of numerical linear algebra, computational statistics, signal processing, optimization theory and automatic differentiation tools (Tensorflow, JAX). Experience in any of these areas is a plus.
The goal of this project is to develop state-of-the-art model-based neural networks for the novel DVS sensor technology in order to learn an efficient reconstruction algorithm and improve the acquisition process.
**TASK LIST**
- Paid full-time Internship at Sony Research Europe (Schlieren Office), for 1 month: - Become accustomed to Dynamic Vision Sensor technology and familiarize with the software tools and infrastructure; Master thesis at Sony Research Europe (Schlieren Office) under joint Sony and ETH Zurich supervision, for 6 months: - Literature review on inverse problems in biomedical imaging. - Literature review on model-aware neural networks. - Implement a simple DVS modelling software under the guidance of industry experts. - Implement and train a model-based neural network. - Evaluate the network performance in the context of original scene reconstruction. - Write a report.
**REQUIREMENTS**
- Willingness to learn both a new type of vision sensor and a new machine learning field. - Knowledge of Matlab and Python with its numerical libraries; familiarity with standard Unix workflows. - Candidate is assumed to work with applied aspects of numerical linear algebra, computational statistics, signal processing, optimization theory and automatic differentiation tools (Tensorflow, JAX). Experience in any of these areas is a plus.
- Dr. Valery Vishnevskiy, ETH Zurich, D-ITET, Institute for Biomedical Engineering, vishnevskiy@biomed.ee.ethz.ch
- Dr. Diederik Moeys, Sony Research Zurich
- Prof. Sebastian Kozerke, ETH Zurich, D-ITET, Institute for Biomedical Engineering
- Dr. Valery Vishnevskiy, ETH Zurich, D-ITET, Institute for Biomedical Engineering, vishnevskiy@biomed.ee.ethz.ch - Dr. Diederik Moeys, Sony Research Zurich - Prof. Sebastian Kozerke, ETH Zurich, D-ITET, Institute for Biomedical Engineering