Register now After registration you will be able to apply for this opportunity online.
This opportunity is not published. No applications will be accepted.
Multimodal Generative Learning for Diagnosing Pulmonary Hypertension in Newborns
Pulmonary hypertension (PH) in newborns poses significant diagnostic challenges due to its association with various diseases and its impact on morbidity and mortality. Early and accurate detection is essential for effective management, yet current manual echocardiographic assessment is time-consuming and requires expertise. This project aims to develop an automated machine learning method using multimodal variational autoencoders (VAEs) and diffusion models to predict PH in newborns from ultrasound, ECG data, and clinical variables. Leveraging a cohort of 270 newborns from the University Children’s Hospital Regensburg, the project will enhance interpretability and feature representation by assessing the significance of each data type and utilizing synthetic data augmentation. The hybrid approach of combining VAEs with diffusion models is expected to improve prediction accuracy and generalization, advancing early detection and understanding of PH in newborns.
Background
Pulmonary hypertension (PH) in newborns is a challenging condition linked to various pulmonary, cardiac, and systemic diseases that can lead to significant morbidity and mortality. Thus, accurate and early detection of PH is crucial for appropriate and successful management. While heart ultrasound (echocardiography) serves as the primary diagnostic tool in pediatrics, manual assessment is both time-consuming and expertise-demanding, highlighting the necessity for an automated approach.
In prior work, we used spatio-temporal convolutional architectures for the prediction of PH from each view, and aggregate the predictions of the different views using majority voting. While results were promising, the generalization to a hold-out test cohort could be improved by using more generative AI methods.
Objectives
In this project proposal, the goal is to develop an automated machine learning method for predicting PH in newborns. To achieve this, we will utilize our existing cohort of 270 newborns, along with their ultrasound and electrocardiogram (ECG) data, provided by our collaborators at the Department of Neonatology, University Children’s Hospital Regensburg (KUNO), Regensburg, Germany. This data includes images of the beating heart from various views.
Specifically, the student will employ generative learning techniques, such as multimodal variational autoencoders (VAEs), to predict PH using the ultrasound, ECG data, and clinical variables. The project will also focus on interpretability by assessing the significance of each modality in the prediction task and conducting intra-modality experiments. This approach will help determine the contribution of each data type towards the overall prediction and improve the model’s transparency and reliability.
Additionally, diffusion models could be used for data augmentation, generating realistic synthetic ultrasound and ECG data to bolster the training dataset and improve model robustness. A hybrid approach can also be explored, combining diffusion models with multimodal VAEs. This involves using diffusion models to pre-process or transform the data before feeding it into the VAEs, potentially leading to improved feature representation and prediction accuracy. By integrating these techniques, the project can leverage the strengths of both VAEs and diffusion models, potentially leading to significant advancements in the early detection and understanding of pulmonary hypertension in newborns.
Requirements
. Proficiency in Python programming (and TensorFlow/PyTorch).
. Solid background in machine learning and deep learning: have taken courses such as Introduction to Machine Learning, Advanced Machine Learning, Deep Learning, Computational Intelligence Lab, or equivalent.
. Ability to independently read and understand publications in the fields of machine learning.
. Interest in applied topics, particularly in medicine (background on the topic or related is NOT required).
. Ideally, a Master's student in Computer Science/Data Science (or Engineering or a similar field, with the aforementioned skills).
Background
Pulmonary hypertension (PH) in newborns is a challenging condition linked to various pulmonary, cardiac, and systemic diseases that can lead to significant morbidity and mortality. Thus, accurate and early detection of PH is crucial for appropriate and successful management. While heart ultrasound (echocardiography) serves as the primary diagnostic tool in pediatrics, manual assessment is both time-consuming and expertise-demanding, highlighting the necessity for an automated approach.
In prior work, we used spatio-temporal convolutional architectures for the prediction of PH from each view, and aggregate the predictions of the different views using majority voting. While results were promising, the generalization to a hold-out test cohort could be improved by using more generative AI methods.
Objectives
In this project proposal, the goal is to develop an automated machine learning method for predicting PH in newborns. To achieve this, we will utilize our existing cohort of 270 newborns, along with their ultrasound and electrocardiogram (ECG) data, provided by our collaborators at the Department of Neonatology, University Children’s Hospital Regensburg (KUNO), Regensburg, Germany. This data includes images of the beating heart from various views.
Specifically, the student will employ generative learning techniques, such as multimodal variational autoencoders (VAEs), to predict PH using the ultrasound, ECG data, and clinical variables. The project will also focus on interpretability by assessing the significance of each modality in the prediction task and conducting intra-modality experiments. This approach will help determine the contribution of each data type towards the overall prediction and improve the model’s transparency and reliability.
Additionally, diffusion models could be used for data augmentation, generating realistic synthetic ultrasound and ECG data to bolster the training dataset and improve model robustness. A hybrid approach can also be explored, combining diffusion models with multimodal VAEs. This involves using diffusion models to pre-process or transform the data before feeding it into the VAEs, potentially leading to improved feature representation and prediction accuracy. By integrating these techniques, the project can leverage the strengths of both VAEs and diffusion models, potentially leading to significant advancements in the early detection and understanding of pulmonary hypertension in newborns.
Requirements
. Proficiency in Python programming (and TensorFlow/PyTorch).
. Solid background in machine learning and deep learning: have taken courses such as Introduction to Machine Learning, Advanced Machine Learning, Deep Learning, Computational Intelligence Lab, or equivalent.
. Ability to independently read and understand publications in the fields of machine learning.
. Interest in applied topics, particularly in medicine (background on the topic or related is NOT required).
. Ideally, a Master's student in Computer Science/Data Science (or Engineering or a similar field, with the aforementioned skills).
The main goals to fulfill in this thesis are:
. Review literature on pulmonary hypertension in newborns, focusing on its causes, diagnosis, and existing prediction methods.
. Familiarize with and pre-process the cohort of 270 newborns' ultrasound and ECG data for machine learning models.
. Implement and train multimodal variational autoencoders (VAEs) using the ultrasound, ECG data, and clinical variables to predict pulmonary hypertension.
. Assess the interpretability of VAEs by evaluating the importance of each modality (ultrasound, ECG, and clinical variables) in the prediction task.
. Perform intra-modality experiments to analyze the contribution of different data types (e.g., different ultrasound views) towards the prediction of PH.
. Implement diffusion models for data augmentation to generate realistic synthetic ultrasound and ECG data, enhancing the training dataset.
. Explore a hybrid approach by combining diffusion models with multimodal VAEs for improved feature representation and prediction accuracy.
. Compare the new hybrid models with previously used spatio-temporal convolutional architectures and majority voting approach to assess improvements in generalization.
. Document findings, analyze results, and highlight potential advancements in early detection and understanding of PH in newborns.
. Prepare a final report and presentation summarizing the methodology, experiments, results, and conclusions.
The main goals to fulfill in this thesis are:
. Review literature on pulmonary hypertension in newborns, focusing on its causes, diagnosis, and existing prediction methods.
. Familiarize with and pre-process the cohort of 270 newborns' ultrasound and ECG data for machine learning models.
. Implement and train multimodal variational autoencoders (VAEs) using the ultrasound, ECG data, and clinical variables to predict pulmonary hypertension.
. Assess the interpretability of VAEs by evaluating the importance of each modality (ultrasound, ECG, and clinical variables) in the prediction task.
. Perform intra-modality experiments to analyze the contribution of different data types (e.g., different ultrasound views) towards the prediction of PH.
. Implement diffusion models for data augmentation to generate realistic synthetic ultrasound and ECG data, enhancing the training dataset.
. Explore a hybrid approach by combining diffusion models with multimodal VAEs for improved feature representation and prediction accuracy.
. Compare the new hybrid models with previously used spatio-temporal convolutional architectures and majority voting approach to assess improvements in generalization.
. Document findings, analyze results, and highlight potential advancements in early detection and understanding of PH in newborns.
. Prepare a final report and presentation summarizing the methodology, experiments, results, and conclusions.
If you are interested please send: (1) your CV, (2) your grades, (3) one-paragraph motivation and, (4) timeline and availability, to ece.oezkanelsen@inf.ethz.ch and samuel.ruiperezcampillo@inf.ethz.ch. Looking forward to hearing from you!
If you are interested please send: (1) your CV, (2) your grades, (3) one-paragraph motivation and, (4) timeline and availability, to ece.oezkanelsen@inf.ethz.ch and samuel.ruiperezcampillo@inf.ethz.ch. Looking forward to hearing from you!