Register now After registration you will be able to apply for this opportunity online.
This opportunity is not published. No applications will be accepted.
Identifiability of shared content factors in real-world multi-modal datasets
In this project, we wish to investigate whether contrastive objectives can be used to identify shared content information in real-world multi-modal datasets.
While the representative power of pre-trained contrastive representation is already attested [1][2], groundwork is still needed to better understand the principles underlying this success. Recent theoretical and experimental investigations have shown that contrastive objectives can successfully isolate sources of information that are systematically shared between data instances (i.e. _content_) from the remaining sources of information [3]. These identifiability experimental results have so far focused on numerical simulations and carefully engineered synthetic datasets. It is realistic to believe that real-world datasets violate some of the generative design choices used in these synthetic experiments. It is therefore of great interest to understand how contrastive objectives can be used in practice to efficiently isolate sources of content information.
This semester/master’s project aims at investigating the challenges mentioned here above and helping increase our understanding of contrastive learning objectives. More detailed information regarding the topics of contrastive learning and identifiability is reported here. **If interested, please contact Alice Bizeul (alice.bizeul@inf.ethz.ch) and send your transcript, CV as well as explain your motivation for the project in 3-4 sentences.**
An ideal candidate should fulfill the following pre-requisites:
- Proficiency in Python and Tensorflow or PyTorch
- Solid background in Machine Learning: have taken Introduction to Machine
- Learning, Advanced Machine Learning, Deep Learning
- Be an independent reader and information seeker
- Be curious about topics related to representation learning, information theory, and machine learning
- Experience with high-performance computing clusters is an advantage
Reading list:
1. Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International Conference on Machine Learning. PMLR, (2021).
2. Oord, Aaron van den, Yazhe Li, and Oriol Vinyals. "Representation learning with contrastive predictive coding." arXiv preprint arXiv:1807.03748 (2018).
3. Von Kügelgen, Julius, et al. "Self-supervised learning with data augmentations provably isolates content from style." Advances in neural information processing systems 34 (2021).
4. Yu, Fisher, et al. "Bdd100k: A diverse driving dataset for heterogeneous multitask learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2020).
While the representative power of pre-trained contrastive representation is already attested [1][2], groundwork is still needed to better understand the principles underlying this success. Recent theoretical and experimental investigations have shown that contrastive objectives can successfully isolate sources of information that are systematically shared between data instances (i.e. _content_) from the remaining sources of information [3]. These identifiability experimental results have so far focused on numerical simulations and carefully engineered synthetic datasets. It is realistic to believe that real-world datasets violate some of the generative design choices used in these synthetic experiments. It is therefore of great interest to understand how contrastive objectives can be used in practice to efficiently isolate sources of content information.
This semester/master’s project aims at investigating the challenges mentioned here above and helping increase our understanding of contrastive learning objectives. More detailed information regarding the topics of contrastive learning and identifiability is reported here. **If interested, please contact Alice Bizeul (alice.bizeul@inf.ethz.ch) and send your transcript, CV as well as explain your motivation for the project in 3-4 sentences.**
An ideal candidate should fulfill the following pre-requisites:
- Proficiency in Python and Tensorflow or PyTorch - Solid background in Machine Learning: have taken Introduction to Machine - Learning, Advanced Machine Learning, Deep Learning - Be an independent reader and information seeker - Be curious about topics related to representation learning, information theory, and machine learning - Experience with high-performance computing clusters is an advantage
Reading list:
1. Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International Conference on Machine Learning. PMLR, (2021).
2. Oord, Aaron van den, Yazhe Li, and Oriol Vinyals. "Representation learning with contrastive predictive coding." arXiv preprint arXiv:1807.03748 (2018).
3. Von Kügelgen, Julius, et al. "Self-supervised learning with data augmentations provably isolates content from style." Advances in neural information processing systems 34 (2021).
4. Yu, Fisher, et al. "Bdd100k: A diverse driving dataset for heterogeneous multitask learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2020).
The goal of the project includes, but is not limited to:
**Minimal expectations:** Perform relevant literature research on the topics of non-linear ICA, identifiability and contrastive learning; define generative design choices that are likely to be violated in real-world datasets; select at least one violation to explore; engineer synthetic datasets, using graphics software such as Blender, extending the generative design choices; evaluate identifiability of content information in the synthetic context.
**Expectations fulfilled:** Develop an experimental setup using appropriate real-world datasets (e.g. multiple object tracking datasets such as the BDD100K dataset [4]) that enables the evaluation of identifiability of content information in a real-world scenario.
**Very successful thesis:** Evaluate identifiability of content informationin the real-world scenario; conduct ablation experiments to identify key parameters affecting identifiability results; for sensitive parameters, develop model selection criteria;
**Depending on the student’s implication and results, this project could lead the way toward a research publication.**
The goal of the project includes, but is not limited to:
**Minimal expectations:** Perform relevant literature research on the topics of non-linear ICA, identifiability and contrastive learning; define generative design choices that are likely to be violated in real-world datasets; select at least one violation to explore; engineer synthetic datasets, using graphics software such as Blender, extending the generative design choices; evaluate identifiability of content information in the synthetic context.
**Expectations fulfilled:** Develop an experimental setup using appropriate real-world datasets (e.g. multiple object tracking datasets such as the BDD100K dataset [4]) that enables the evaluation of identifiability of content information in a real-world scenario.
**Very successful thesis:** Evaluate identifiability of content informationin the real-world scenario; conduct ablation experiments to identify key parameters affecting identifiability results; for sensitive parameters, develop model selection criteria;
**Depending on the student’s implication and results, this project could lead the way toward a research publication.**
Identifiability_of_shared_factors_in_real_world_multi_modal_datasets.pdf
