tillahoffmann / summaries

Comparison of summary statistic selection methods with a unifying perspective.

summaries

This repository accompanies the preprint "Minimizing the expected posterior entropy yields optimal summary statistics" and can be used to reproduce all figures and reported results.

You can reproduce the results in three steps:

1. Set up a new, clean python virtual environment (you can also skip this step to use your host python environment, but your mileage may vary).
2. Install all the requirements by running pip install -r requirements.txt.
3. Generate all result files by running the following code from the command line.

# Ignore the figures until we have generated all data files.
doit ignore figures
# Generate the data files (use -n to parallelize if desired).
doit -n [number of cores]
# Generate the figures and a summary file `workspace/figures/figures.html`.
doit forget figures
doit figures

You will find all figures in the folder workspace/figures together with a HTML report workspace/figures/figures.html that contains additional information. This process takes about 40 minutes on an M1 MacBook Pro when parallelizing across six cores.

The code has been tested with python 3.9 on macOS 12.4 (Monterey) running on Apple silicon and on Ubuntu 20.04. The summaries package has complete test coverage, and you can run pytest from the repository root to verify that your installation is working.

summaries's Issues

Design benchmark and evaluation.

⚙️ Benchmark model and data

We would like to design a benchmark model that

• is sufficiently varied to be interesting.
• has a tractable (but non-trivial) likelihood.
• allows for the posterior to evaluated (probably using numerical integration for the normalisation constant).

🤔 Inference

Any inference method applied to the synthetic benchmark dataset should take as input

• the training data (e.g. to be used as a reference table for ABC, training data for feature learning, or conditional density estimation).
• the validation data (e.g. to be used for selecting a good acceptance threshold or early stopping in training).

and return a function def sample_posterior(x: np.ndarray) -> np.ndarray to draw samples from the posterior.

The ultimate, unified output of the inference pipeline is a set of posterior samples for each test dataset and each inference/feature selection method. Each method may also produce other output, such as the gridded posterior obtained from numerical integration.

📈 Evaluation

For each method, we will compare posterior samples against the "true" posterior obtained by numerical integration. Candidate evaluation approaches include

• Kullback-Leibler divergence between the true and approximate posteriors (but that requires density estimation on the unit square--or some other KL estimator).
• discrepancy (e.g. RMSE) between some point estimate of the posterior (e.g. posterior mean) and the true parameter value (but that doesn't really capture how well we're approximating, only how good the point estimate is). Maybe the discrepancy between the true and approximate posterior mean is better.

Make sure we're not "cheating" by leaving out noise statistics for the benchmark MDN model.

The current MDN model for the benchmark problem does not include the noise features. That means we're giving the algorithm an unfair advantage, and we should include the noise in the training.

Consolidate lingering numpy/pytorch inconsistencies.

#11 changes a lot of code from numpy to torch, but there are a number of outstanding lines that are "mixed".

Clean up the Makefile targets for benchmark inference

Add non-linear mean square error minimization algorithm.

Fearnhead and Prangle proposed minimising the mean square error using a linear model. This was extended by Jiang et al. to non-linear models. We can probably drop the original Fearnhead algorithm and replace it using the non-linear model because a simple dense layer is a linear model.