aleximmer / marglik

Marginal likelihood estimation for model selection in deep learning

Marglik: Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning

Code for the paper:

Immer, Bauer, Fortuin, Rätsch, Khan. Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning. ICML 2021.

In this repository, the Algorithm proposed in our paper is implemented in marglik.py. The code uses the laplace-torch package, a stand-alone package that originates from our implementation for this paper but provides additional features related to predictions (see also linearized-predictives) and last-layer Laplace approximations.

Setup

Python 3.8 is required although it might work with later or earlier versions potentially. Install minimal dependencies pip install -r requirements.txt.

Usage

The method marglik_optimization(...) in marglik.py requires at least a pytorch model and training loader. The method trains the neural network and optimizes the prior precision and observation noise (for regression). For details, see the signature of the method or consult the documentation of the separately published laplace-torch library here.

marglik's Issues

About hyperparameter optimization

Hi, I really enjoyed your impressive paper.

I read marglik.py and tried to replicate the optimization of the temperature parameter written in the paper as eq (7). But I found the returned value of lap.log_marginal_likelihood is not differentiable w.r.t. parameters other than the prior precision, that is, the temperature parameter cannot be optimized. It would be appreciated if you instruct me on how to do this.