danielkelshaw / concretedropout

PyTorch implementation of 'Concrete Dropout'

Hello! I am currently a PhD student @ImperialCollege. My research lies at the intersection of machine learning, physics, and differential geometry. The goal of this research is to leverage reduced-order models for high-dimensional, complex physical phenomena - utilising tools from differential geometry and physics to operate directly on low-dimensional manifolds.

Previously, I worked as a machine learning research intern @NasaJPL, @ESA, @MindFoundry, @Dyson, and @BAESystems.

concretedropout's Issues

Update README

At the moment the README.md provides no particular indication as to what this project is about - it would be a good idea to update this to describe the purpose of the code that has been developed.

Configure Repository

In order to provide a suitable development environment the repository should be set up with an appropriate directory structure as well as functionality for automated continuous integration.

Rethink Class Decorator Approach

In the initial development of this code I was experimenting with the use of class decorators to add functionality, namely the concrete_regulariser. Coming back to this code several months later I realise that, although this works, it is not the nicest way of implementing a solution.

I would like to rework my implementation of the regularisation by making a base-class which accounts for the implementation, rather than requiring the user to wrap their model implementation with a pre-defined decorator.

Provide MNIST Example

An example of using ConcreteDropout in training a network for MNIST classification should be included - this provides a demonstration of the capabilities of the network.

is softmax required?

The model passes the logits through softmax in the output layer, but then the F.cross_entropy() function is used which combines log_softmax and nll. I think maybe the softmax in the output layer is not necessary?

Weight regulariser and dropout regulariser

Hi ! I read Yarin Gal's paper and I did not understand how the weight regulariser and dropout regulariser are initialized. The author provided a formula, but it is not very clear (e.g what means prior length scale ? and which value to assign for this variable ?). Could you explain how you find the values used to inizialize the weight regulariser and the dropout regulariser ?

Demonstrate Usage in README.md

It would be good to demonstrate how the modules developed can be user in a very simple way. This could help users implement the code found in this repository.

Implement ConcreteDropout Class

Implementation of the ConcreteDropout class should inherit from nn.Module to allow the functionality to be used inline with other PyTorch modules. Upon implementation the ConcreteDropout module can be used in a regular neural network to be tested.