Register now After registration you will be able to apply for this opportunity online.
This opportunity is not published. No applications will be accepted.
Learning representations that know what they don’t know
The ability to quantify uncertainty in discriminative tasks is a fundamental requirement for the widespread deployment of neural networks. Currently, this demonstrates one of the most challenging research questions in the field.
Keywords: deep learning, computer vision, uncertainty, representation learning, density estimation
The machine learning community distinguishes between two types of uncertainty - uncertainty arising from the data (aleatoric) and uncertainty in the choice of model parameters (epistemic) [4]. Bayesian Neural Networks (BNNs) are the predominant holistic approach for modeling both types of uncertainty in deep learning. However, scalability issues encourage the development of alternatives. One such alternative are density estimates of the hidden representations of a neural network [1, 2, 3]. It has been recently demonstrated that learning the distribution of hidden representations conditioned on the network’s predictions also allows for estimating both types of uncertainty separately. This is a promising result, as it allows bypassing the scalability issues of BNNs in many cases. However, the quality of the resulting uncertainty estimate is linked to the nature of the distribution of hidden representations. In this thesis, we will investigate how to adjust the training procedure of a neural network in order to yield good distributions of hidden representations. We therefore (1) initially define metrics for the “goodness” of a distribution of hidden representations and (2) empirically explore theoretically motivated training heuristics regarding the quality of the latent distribution.
[1] van Amersfoort, J., Smith, L., Teh, Y.W. and Gal, Y., 2020. Simple and Scalable Epistemic Uncertainty Estimation Using a Single Deep Deterministic Neural Network.
[2] Lee, Kimin, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks.
[3] Alemi, A.A., Fischer, I. and Dillon, J.V., Uncertainty in the Variational Information Bottleneck.
[4] Kendall, A. and Gal, Y., 2017. What uncertainties do we need in bayesian deep learning for computer vision?
The machine learning community distinguishes between two types of uncertainty - uncertainty arising from the data (aleatoric) and uncertainty in the choice of model parameters (epistemic) [4]. Bayesian Neural Networks (BNNs) are the predominant holistic approach for modeling both types of uncertainty in deep learning. However, scalability issues encourage the development of alternatives. One such alternative are density estimates of the hidden representations of a neural network [1, 2, 3]. It has been recently demonstrated that learning the distribution of hidden representations conditioned on the network’s predictions also allows for estimating both types of uncertainty separately. This is a promising result, as it allows bypassing the scalability issues of BNNs in many cases. However, the quality of the resulting uncertainty estimate is linked to the nature of the distribution of hidden representations. In this thesis, we will investigate how to adjust the training procedure of a neural network in order to yield good distributions of hidden representations. We therefore (1) initially define metrics for the “goodness” of a distribution of hidden representations and (2) empirically explore theoretically motivated training heuristics regarding the quality of the latent distribution.
[1] van Amersfoort, J., Smith, L., Teh, Y.W. and Gal, Y., 2020. Simple and Scalable Epistemic Uncertainty Estimation Using a Single Deep Deterministic Neural Network. [2] Lee, Kimin, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. [3] Alemi, A.A., Fischer, I. and Dillon, J.V., Uncertainty in the Variational Information Bottleneck. [4] Kendall, A. and Gal, Y., 2017. What uncertainties do we need in bayesian deep learning for computer vision?
Familiarize with adjacent literature
Define metrics for the quality of the latent distribution
Define an initial set of theoretically motivated heuristics
Empirical evaluation of heuristics
Familiarize with adjacent literature Define metrics for the quality of the latent distribution Define an initial set of theoretically motivated heuristics Empirical evaluation of heuristics
Knowledge of at least one common deep learning framework (e.g. pytorch, tensorflow)
First experience with training neural networks
Machine learning basics
Knowledge of at least one common deep learning framework (e.g. pytorch, tensorflow) First experience with training neural networks Machine learning basics
Supervisor:
Janis Postels, ETF D112, jpostels@vision.ee.ethz.ch
Hermann Blum, LEE H223, blumh@ethz.ch
Professor: Luc Van Gool, ETF C117, Tel.: +41 44 63 26578; vangool@vision.ee.ethz.ch
Supervisor: Janis Postels, ETF D112, jpostels@vision.ee.ethz.ch Hermann Blum, LEE H223, blumh@ethz.ch Professor: Luc Van Gool, ETF C117, Tel.: +41 44 63 26578; vangool@vision.ee.ethz.ch