bokorn / kgp

Keras + Gaussian Processes: Learning scalable deep and recurrent kernels.

Gaussian Processes for Keras

KGP extends Keras with Gaussian Process (GP) layers. It allows one to build flexible GP models with kernels structured with deep and recurrent networks built with Keras. The structured part of the model (the neural net) runs on Theano or Tensorflow. The GP layers use a custom backend based on GPML 4.0 library, and builds on KISS-GP and extensions. The models can be trained in stages or jointly, using full-batch or semi-stochastic optimization approaches (see our paper). For additional resources and tutorials on Deep Kernel Learning and KISS-GP see https://people.orie.cornell.edu/andrew/code/

KGP is compatible with: Python 2.7-3.5.

In particular, this package implements the method described in our paper:
Learning Scalable Deep Kernels with Recurrent Structure
Maruan Al-Shedivat, Andrew Gordon Wilson, Yunus Saatchi, Zhiting Hu, Eric P. Xing
arXiv:1610.08936, 2016.

Getting started

KGP allows to build models in the same fashion as Keras, using the functional API. For example, a simple GP-RNN model can be built and compiled in just a few lines of code:

from keras.layers import Input, SimpleRNN
from keras.optimizers import Adam

from kgp.layers import GP
from kgp.models import Model
from kgp.losses import gen_gp_loss

input_shape = (10, 2)  # 10 time steps, 2 dimensions
batch_size = 32
nb_train_samples = 512
gp_hypers = {'lik': -2.0, 'cov': [[-0.7], [0.0]]}

# Build the model
inputs = Input(shape=input_shape)
rnn = SimpleRNN(32)(inputs)
gp = GP(gp_hypers,
        batch_size=batch_size,
        nb_train_samples=nb_train_samples)
outputs = [gp(rnn)]
model = Model(inputs=inputs, outputs=outputs)

# Compile the model
loss = [gen_gp_loss(gp) for gp in model.output_layers]
model.compile(optimizer=Adam(1e-2), loss=loss)

Note that KGP models support arbitrary off-the-shelf optimizers from Keras.

Further resources:

• A quick tutorial that walks you through the key components of the library.
• A few more examples.

Installation

KGP depends on Keras and requires either Theano or TensorFlow being installed. The GPML backend requires either MATLAB or Octave and a corresponding Python interface package: Oct2Py for Octave or the MATLAB engine for Python. Generally, MATLAB backend seems to provide faster runtime. However, if you compile the latest version of Octave with JIT and OpenBLAS support, the overhead gets reduced to minimum.

If you are using Octave, you will need the statistics package. You can install the package using Octave-Forge:

$ octave --eval "pkg install -forge -verbose io"
$ octave --eval "pkg install -forge -verbose statistics"

The requirements can be installed via pip as follows (use sudo if necessary):

$ pip install -r requirements.txt

To install the package, clone the repository and run setup.py as follows:

$ git clone --recursive https://github.com/alshedivat/kgp
$ cd kgp
$ python setup.py develop [--user]

The --user flag (optional) will install the package for a given user only.

Note: Recursive clone is required to get GPML library as a submodule. If you already have a copy of GPML, you can set GPML_PATH environment variable to point to your GPML folder instead.

Contribution

Contributions and especially bug reports are more than welcome.

Citation

@article{alshedivat2016srk,
  title={Learning Scalable Deep Kernels with Recurrent Structure},
  author={Al-Shedivat, Maruan and Wilson, Andrew G and Saatchi, Yunus and Hu, Zhiting and Xing, Eric P},
  journal={arXiv preprint arXiv:1610.08936},
  year={2016}
}

License

For questions about the code and licensing details, please contact Maruan Al-Shedivat and Andrew Gordon Wilson.