Seminar on Uncertainty Estimation (FSS 2022)

Neural networks are increasingly being trained as black-box predictors and being placed in larger decision systems where errors in their predictions can pose immediate threat to downstream tasks. Systematic methods for calibrated uncertainty estimation under these conditions are needed, especially as these systems are deployed in safety critical domains or in settings with large dataset imbalances. When neural networks are being deployed in safety critical domains, where the ability to infer model uncertainty is crucial for eventual wide-scale adoption, precise and calibrated uncertainty estimates are useful for interpreting confidence, capturing domain shift of out-of-distribution (OOD) test samples, and recognizing when the model is likely to fail.

In this seminar, you will familiarize yourself with recent developments in the field of uncertainty estimation and evidential learning to infer model uncertainty. You will read research papers, conduct own experiments and you will discuss the insights with the other participants of the seminar. 

In this seminar, you will

• Read, understand, and explore scientific literature
• Summarize a current research topic in a concise report using the LaTeX template (10 Pages + unlimited References)
• Give one presentation about your topic (20 minutes presentation + 10 minutes discussion)
• Review a (draft of a) report of a fellow student

• This seminar is organized by Dr. Tobias Weller
• Available for Master students (2 SWS, 4 ECTS)
• Prerequisites: Basic knowledge in Machine Learning (Data Mining I)

• Registration will be available in Portal2 (tba) until Feb 14.
• After the kick off session, please send a ranked list of three topics by Feb 20 you would like to read and present in the seminar to Bianca Lermer
• The actual topic assignment takes place soon afterwards; we will notify you via email. Our goal is to assign one of your preferred topics to you. 

Each student works on a topic within the area of the seminar along with an accompanying reference paper.

Topic list

1. Deep Evidential Regression
   Alexander Amini, Wilko Schwarting, Ava Soleimany, Daniela Rus
   Advances in Neural Information Processing Systems 2020
2. Evidential Deep Learning to Quantify Classification Uncertainty
   Murat Sensoy, Lance Kaplan, Melih Kandemir
   Advances in Neural Information Processing Systems 2018
3. Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness
   Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, Balaji Lakshminarayanan
   Advances in Neural Information Processing Systems 2020
4. Predictive Uncertainty Estimation via Prior Networks
   Andrey Malinin, Mark Gales
   Advances in Neural Information Processing Systems 2018
5. A Simple Baseline for Bayesian Uncertainty in Deep Learning
   Wesley J. Maddox, Pavel Izmailov, Timur Garipov, Dmitry P. Vetrov, Andrew Gordon Wilson
   Advances in Neural Information Processing Systems 2019
6. Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts
   Bertrand Charpentier, Daniel Zügner, Stephan Günnemann
   Advances in Neural Information Processing Systems 2020
7. Graph Posterior Network: Bayesian Predictive Uncertainty for Node Classification
   Maximilian Stadler, Bertrand Charpentier, Simon Geisler, Daniel Zügner, Stephan Günnemann
   Advances in Neural Information Processing Systems 2021
8. Evidential Softmax for Sparse Multimodal Distributions in Deep Generative Models
   Phil Chen, Masha Itkina, Ransalu Senanayake, Mykel J. Kochenderfer
   Advances in Neural Information Processing Systems 2021
9. Discriminative Jackknife: Quantifying Uncertainty in Deep Learning via Higher-Order Influence Functions
   Ahmed Alaa, Mihaela Van Der Schaar​​​​​​​
   International Conference on Machine Learning 2020
10. Uncertainty Estimation Using a Single Deep Deterministic Neural Network
    Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, Yarin Gal​​​​​​​
    International Conference on Machine Learning 2020
11. Maximizing Overall Diversity for Improved Uncertainty Estimates in Deep Ensembles​​​​​​​
    Siddhartha Jain, Ge Liu, Jonas Mueller​​​​​​​, David Gifford
    Proceedings of the AAAI Conference on Artificial Intelligence 2020
12. Quantifying Uncertainties in Natural Language Processing Tasks
    Yijun Xiao, William Yang Wang
    International Conference on Machine Learning 2019​​​​​​​