awehenkel / umnn

Implementation of Unconstrained Monotonic Neural Network and the related experiments. These architectures are particularly useful for modelling monotonic transformations in normalizing flows.

Unconstrained Monotonic Neural Networks (UMNN)

Official implementation of Unconstrained Monotonic Neural Networks (UMNN) and the experiments presented in the paper:

Antoine Wehenkel and Gilles Louppe. "Unconstrained Monotonic Neural Networks." (2019). [arxiv]

Other implementations:

• Check the Zuko library for a clean and complete implementation of UMNNs based normalizing flows.
• Check here if you are interested by modeling functions that are monotonic with respect to more than one input variable. (Do not hesitate to contact me for more details)

Dependencies

The code has been tested with Pytorch 1.1 and Python3.6. Some code to draw figures and load dataset are taken from FFJORD and Sylvester normalizing flows for variational inference.

Usage

Simple Monotonic Function

This experiment is not described in the paper. We create the following dataset: x = [x_1, x_2, x_3] is drawn from a multivariate Gaussian, y = 0.001(x_1^3 + x_1) + x_2 + sin(x_3). We suppose that we are given the information about the monotonicity of y with respect to x_1.

python MonotonicMLP.py 

In this experiment we show that a classical MLP won't be able to model a function that is monotonic with respect to x_1 because its effect is small in comparison to the other variables. The UMNN performs better than an MLP while ensuring that the output is monotonic with respect to x_1.

Toy Experiments

python ToyExperiments.py 

See ToyExperiments.py for optional arguments.

MNIST

python MNISTExperiment.py

See MNISTExperiment.py for optional arguments.

UCI Dataset

You have to download the datasets with the following command:

python datasets/download_datasets.py 

Then you can execute:

python UCIExperiments.py --data ['power', 'gas', 'hepmass', 'miniboone', 'bsds300']

See UCIExperiments.py for optional arguments.

VAE

You have to download the datasets:

• MNIST:

python datasets/download_datasets.py

• OMNIGLOT: the dataset can be downloaded from link;
• Caltech 101 Silhouettes: the dataset can be downloaded from link.
• Frey Faces: the dataset can be downloaded from link.

python TrainVaeFlow.py -d ['mnist', 'freyfaces', 'omniglot', 'caltech']

Other Usage

All the files related to the implementation of UMNN (Conditionner network, Integrand Network and Integral) are located in the folder models/UMNN.

• NeuralIntegral.py computes the integral of a neural network (with 1d output) using the Clenshaw-Curtis(CC) quadrature, it computes sequentially the different evaluation points required by CC.
• ParallelNeuralIntegral.py processes all the evaluation points at once making the computation almost as fast as the forward evaluation the net but to the price of a higher memory cost.
• UMNNMAF.py contains the implementation of the different networks required by UMNN.
• UMNNMAFFlow.py contains the implementation of flows made of UMNNs.

Cite

If you make use of this code in your own work, please cite our paper:

@inproceedings{wehenkel2019unconstrained,
  title={Unconstrained monotonic neural networks},
  author={Wehenkel, Antoine and Louppe, Gilles},
  booktitle={Advances in Neural Information Processing Systems},
  pages={1543--1553},
  year={2019}
}

umnn's Issues

Is Lipschitz constraint enforced in the code?

I found the function force_lipschitz in the code, but I did not see where it was used. Is this constraint used or really necessary in the codebase?

Lack of license

Hello,

Could you please add a license, for example MIT, to make your code reusable?

Best,
Hadrien

UMNNMAFFlow.invert argument missmatch in ToyExperiments.py

When running ToyExperiments.py, I run into an issue with the UMNNMAFFlow.invert function. As defined in UMNNMAFFlow.py it only takes arguments "z" and "context". However, the arguments given are z, 5, and "ParallelSimpler". Removing the second argument (5) makes the required plot, however, I am not sure what it is supposed to do.

I can send a pull request deleting the second argument.

Creating model...
Model created.
epoch: 0 - Train loss: 6.782076 - Test loss: 6.447329 - Elapsed time per epoch 0.064852 (seconds)
Traceback (most recent call last):
File "ToyExperiments.py", line 186, in
train_toy(toy, load=args.load, folder=args.folder)
File "ToyExperiments.py", line 163, in train_toy
summary_plots(x, x_test, folder, epoch, model, ll_tot, ll_test)
File "ToyExperiments.py", line 70, in summary_plots
x_pred = model.invert(z, 5, "ParallelSimpler")
TypeError: invert() takes from 2 to 3 positional arguments but 4 were given

Link to code not in paper pdf

Hello Antoine, I was just looking at your paper and noticed that the latest arXiv version still says "The source code to reproduce our experiments will be made available on Github at the end of the reviewing process". It might be nice to put the link to this (very nice) repo in the paper so that it's slightly easier to find.

Question : is it possible to enforce monotonicity on the whole feature space ?

Hello,
I would want to understand if with this approach is possible to apply monotonicity on more than 1 feature at a time from the input space.
Thank you.

download_datasets.py doesn't seem to download all datasets

Running python ToyExperiments.py does not seem to automatically download the UCI datasets for me.