317070 / twitch-plays-lsd-neural-net

What is this?

This is the source code which is used to generate the stream http://www.twitch.tv/317070 The program can recreate images from the neural networks vgg-16 and vgg-19, both found here What's even better, it can do this interactively and in real time.

What do I need?

For clarity, this code is for linux only. It is best run on a beefy computer: At least a hexacore CPU At least a graphics card with 4GB of memory (e.g. the GTX 680, 980 and the Tesla K40 have been tested) At least 12 GB of RAM (not tested), 32GB is recommended and tested.

You will need the following libraries installed:

1. Cudnn
2. Pylearn2
3. Theano
4. Lasagne

Warning: setting these up is unfortunately not `sudo apt-get' trivial.

You will also need to download the vgg networks from their website here http://www.vlfeat.org/matconvnet/pretrained/ Put the resulting .mat files in the data folder, and run the script

python mat2npy.py

to convert to a data-structure lasagne can use.

What do I do?

You can run the default configuration for streams with

python train.py

and for images

python train_image.py

You can also run custom configurations using:

python train.py custom_configuration

How do I set this up?

Go to models/default.py to edit the default configuration. You will definitely need to fix the Twitch parameters, these are used for logging into twitch and setting up the stream.

... and now?

This source is provided as is. It will be hard to set up (because of its dependencies) by artists or others who are not familiar with these machine learning libraries. Sorry for that. I think the biggest benefit will be in the reading of the code and using the ideas behind it.

twitch-plays-lsd-neural-net's Issues

what does the argument "ds" means?

In default_image.py:

max_pool = partial(dnn.MaxPool2DDNNLayer,
                            ds=(2,2), 
                           strides=(2,2))

I found ds param, however I can't find this in lasagne documents.

BTW, I encounter out of memory error: 'Error allocating X bytes of device memory (out of memory).' Is this caused by my insufficient RAM? My computer equipped with 2GB RAM and a 4GB GPU.

dependency versions

I have all dependencies installed and I am able to run train_image.py using the hd1.jpg image successfully. However if I change the starting image to anything with a width or height less than 896px I get an error:

image shape: (1, 3, 576, 1024)
Traceback (most recent call last):
File "train_image.py", line 218, in
loss, l = iter_train(0)
File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 606, in call
storage_map=self.fn.storage_map)
File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 595, in call
outputs = self.fn()
RuntimeError: Could not set tensor4d descriptor: CUDNN_STATUS_BAD_PARAMshapes=1 4096 0 4 strides=0 0 4 1
Apply node that caused the error: GpuDnnConv{workmem='small', inplace=True}(GpuContiguous.0, GpuContiguous.0, >GpuAlloc{memset_0=True}.0, GpuDnnConvDesc{border_mode='valid', subsample=(1, 1), conv_mode='cross'}.0, Constant{1.0}, Constant{0.0})
Inputs types: [CudaNdarrayType(float32, 4D), CudaNdarrayType(float32, 4D), CudaNdarrayType(float32, 4D), <theano.gof.type.CDataType object at 0x7efe862a7990>, Scalar(float32), Scalar(float32)]
Inputs shapes: [(1, 512, 6, 10), (4096, 512, 7, 7), (1, 4096, 0, 4), 'No shapes', (), ()]
Inputs strides: [(0, 60, 10, 1), (25088, 49, 7, 1), (0, 0, 4, 1), 'No strides', (), ()]
Inputs values: ['not shown', 'not shown', <CudaNdarray object at 0x7efe4ee2a830>, <PyCObject object at 0x7efe4eea7918>, 1.0, 0.0]`

Another problem comes up when running train.py:

features activated:
[331]
Buffer underflow? 0
Traceback (most recent call last):
File "train.py", line 202, in
newframe = stream.send_frame( np.transpose(x.get_value()[0]/255.0,(1,2,0)), text=string )
File "/usr/local/Twitch-plays-LSD-neural-net/zoomingstream.py", line 303, in send_frame
return zoom(frame, frames_before_next_image)
File "/usr/local/Twitch-plays-LSD-neural-net/zoomingstream.py", line 125, in zoom
frame_rotated = fast_warp(frame, (tf_shift_inv + (tf_rotate + tf_shift)).params,
AttributeError: 'SimilarityTransform' object has no attribute 'params'`

I suspect these problems may be due to dependency version differences.
What versions of Theano and skimage are you using? I'd like to test with the same versions you are using, or at least have the older version to compare the code and help debug.

I have a newer version of Lasagne and was able to fix the obvious problems of field names. I am also running the latest NVidia drivers and CUDA libraries to support the gtx980 ti, hopefully those do not have any problems.

VLFeat hosted mat files changed format

The format of the VGG network model files has changed and the mat2npy.py processing file no longer works with the new format.

What is the original shapes of data/vgg16.npy, data/mean-16.npy, and data/classes-16.npy? I attempted to reprocess the new format and below is the shapes I get for the various Numpy arrays. I want to know if this is consistent with the old format before I debug any further.

Verified 'imagenet-vgg-verydeep-16.mat' (md5 = f72d927587ca4c97fbd165ec9cb9997f)
Verified 'imagenet-vgg-verydeep-19.mat' (md5 = 106118b7cf60435e6d8e04f6a6dc3657)

Processing: 16
['layers', '__version__', 'meta', '__header__', '__globals__']
Params:
     (64, 3, 3, 3) (64, 1)
     (64, 64, 3, 3) (64, 1)
     (128, 64, 3, 3) (128, 1)
     (128, 128, 3, 3) (128, 1)
     (256, 128, 3, 3) (256, 1)
     (256, 256, 3, 3) (256, 1)
     (256, 256, 3, 3) (256, 1)
     (512, 256, 3, 3) (512, 1)
     (512, 512, 3, 3) (512, 1)
     (512, 512, 3, 3) (512, 1)
     (512, 512, 3, 3) (512, 1)
     (512, 512, 3, 3) (512, 1)
     (512, 512, 3, 3) (512, 1)
     (4096, 512, 7, 7) (4096, 1)
     (4096, 4096, 1, 1) (4096, 1)
     (1000, 4096, 1, 1) (1000, 1)
Mean:
     (1, 3)
Classes:
     (1000,)

Processing: 19
['layers', '__version__', 'meta', '__header__', '__globals__']
Params:
     (64, 3, 3, 3) (64, 1)
     (64, 64, 3, 3) (64, 1)
     (128, 64, 3, 3) (128, 1)
     (128, 128, 3, 3) (128, 1)
     (256, 128, 3, 3) (256, 1)
     (256, 256, 3, 3) (256, 1)
     (256, 256, 3, 3) (256, 1)
     (256, 256, 3, 3) (256, 1)
     (512, 256, 3, 3) (512, 1)
     (512, 512, 3, 3) (512, 1)
     (512, 512, 3, 3) (512, 1)
     (512, 512, 3, 3) (512, 1)
     (512, 512, 3, 3) (512, 1)
     (512, 512, 3, 3) (512, 1)
     (512, 512, 3, 3) (512, 1)
     (512, 512, 3, 3) (512, 1)
     (4096, 512, 7, 7) (4096, 1)
     (4096, 4096, 1, 1) (4096, 1)
     (1000, 4096, 1, 1) (1000, 1)
Mean:
     (1, 3)
Classes:
     (1000,)

Alternatively, would you be willing to provide a mirror for the old model files?

Selecting initial image for dream

Awesome project!

I looked through train_image.py but didn't find where I can specify an initial image for a dream. I want to make a fork which takes as input initial image and priors then outputs a short dream.