huwcampbell / grenade Goto Github PK

Deep Learning in Haskell

License: BSD 2-Clause "Simplified" License

Shell 0.87% Haskell 95.31% C 3.82%

machine-learning deep-neural-networks haskell deep-learning generative-adversarial-networks convolutional-neural-networks

grenade's People

Contributors

Stargazers

Watchers

Forkers

angerman silky wavewave ocramz yuhangwang mstksg iraikov delanoe aromazyl shabesoglu vdt janbussieck cpehle gdemelo jreina popara neo4reo varosi neuroradiology drwebb halhenke vajrabisj deepakkapiswe theunixman cpennington kerscher tmcgilchrist yichao-liang abailly uuhan hgldj1966 barakb schnecki sholland1 hardentoo sebeaumont atidot csvdb xodarap almays zzmjohn sinsjr2 ndcroos holm-xie sahuism imuli barista-team soraismus lazuraslong freefrancisco tarsbase faezs dme2 leopiney madriska kikaitech tkemps andrewdmeier bourbakii acganesh awesomemachinelearning jbrot jrp2014 zeta1999 morpheus-incorporated leftaroundabout jhb563 nolrai curiouscorrelation pedroelbanquero chiruk stjordanis csaltos standardgalactic luischdev ilyakooo0 haedosa rogerlarsson psilospore sorki akifavcu yssource

grenade's Issues

Missing file for bench-lstm?

> mafia build
Building grenade-0.0.1...
Preprocessing library grenade-0.0.1...
Preprocessing executable 'feedforward' for grenade-0.0.1...
Preprocessing executable 'mnist' for grenade-0.0.1...
Preprocessing executable 'recurrent' for grenade-0.0.1...
Preprocessing executable 'shakespeare' for grenade-0.0.1...
Preprocessing test suite 'test' for grenade-0.0.1...
Preprocessing benchmark 'bench' for grenade-0.0.1...
Preprocessing benchmark 'bench-lstm' for grenade-0.0.1...
cabal: can't find source for bench-lstm in
dist/build/bench-lstm/bench-lstm-tmp, bench
Process failed: cabal build -j --ghc-options=-Werror (exit code: 1)

You probably just forgot to git add it.

Better Wengert tape storage

At the moment during training we give the input for reverse mode at the layer level.

This is pretty good and efficient for most layers, but for some layers like LSTM it's not as granular as it could be.

Maybe Layer should be something like

class UpdateLayer x => Layer x (i :: Shape) (o :: Shape) where
  -- | The Wengert tape for this layer. Includes all that is required
  --   to generate the back propagated gradients efficiently. As a
  --   default, `S i` is fine.
  type Tape x i o :: *
  -- | Used in training and scoring. Take the input from the previous
  --   layer, and give the output from this layer.
  runForwards    :: x -> S i -> (Tape x i o, S o)
  -- | Back propagate a step. Takes the current layer, the input that the
  --   layer gave from the input and the back propagated derivatives from
  --   the layer above.
  --   Returns the gradient layer and the derivatives to push back further.
  runBackwards   :: x -> Tape x i o -> S o -> (Gradient x, S i)

I can't seem to get the examples to produce decent results

I can't seem to get the examples to produce decent results.

OS: Ubuntu 16.04 LTS 64-bit

I installed blas and lapack with

sudo apt-get install libblas-dev
sudo apt-get install liblapack-dev

I ran ../mafia build from the examples directory.

This gist contains the output I got from running the mnist, gan-mnist, and shakespeare examples: https://gist.github.com/sholland1/e4c9047ea683f4bf8dbc6516006f85ad

MNIST data: https://pjreddie.com/projects/mnist-in-csv/
Shakespeare data: https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt

I also tried checking out the code at v0.1.0 and running the mnist example, but I got similar results.

Please assist me in figuring out what the problem is.

I noticed that the serialisation of the type Dropout was missing on the release 0.1.0, preventing from saving and loading a trained network.
The workaround is to apply the trick described here. The problem is that it works only for ghc>=7.2.1.
Tell me if you prefer to add a fix for any ghc release or if I may pull my fix.

mafia build failure

I cloned the latest code on master branch and ran ./mafia build to build grenade
When Preprocessing test suite, I got the following error message.

Preprocessing test suite 'test' for grenade-0.1.0...
[ 1 of 15] Compiling Test.Hedgehog.Hmatrix ( test/Test/Hedgehog/Hmatrix.hs, dist/build/test/test-tmp/Test/Hedgehog/Hmatrix.o )
[ 2 of 15] Compiling Test.Hedgehog.Compat ( test/Test/Hedgehog/Compat.hs, dist/build/test/test-tmp/Test/Hedgehog/Compat.o )
[ 3 of 15] Compiling Test.Hedgehog.TypeLits ( test/Test/Hedgehog/TypeLits.hs, dist/build/test/test-tmp/Test/Hedgehog/TypeLits.o )

test/Test/Hedgehog/TypeLits.hs:22:1: warning: [-Wdeprecations]
    Module ‘GHC.TypeLits.Witnesses’ is deprecated:
      Use singletons package instead

test/Test/Hedgehog/TypeLits.hs:64:14: warning: [-Wdeprecations]
    In the use of ‘natDict’ (imported from GHC.TypeLits.Witnesses):
    Deprecated: "Use singletons package instead"

test/Test/Hedgehog/TypeLits.hs:64:25: warning: [-Wdeprecations]
    In the use of ‘%*’ (imported from GHC.TypeLits.Witnesses):
    Deprecated: "Use singletons package instead"

test/Test/Hedgehog/TypeLits.hs:64:28: warning: [-Wdeprecations]
    In the use of ‘natDict’ (imported from GHC.TypeLits.Witnesses):
    Deprecated: "Use singletons package instead"

<no location info>: error:
Failing due to -Werror.
Process failed: cabal build -j --ghc-options=-Werror (exit code: 1)
ubuntu@ip-172-31-29-191:~/grenade$

index out of bounds -- in Data.Vector.Generic

just tried shakespeare on shakespear 100k sample and got the following...

seb@psi(0) [grenade](2646) 14:06:55> shakespeare ~/Data/misc/shakespeare.txt
TRAINING STEP WITH SIZE: 50
shakespeare: ./Data/Vector/Generic.hs:245 ((!)): index out of bounds (38,37)
CallStack (from HasCallStack):
  error, called at ./Data/Vector/Internal/Check.hs:87:5 in vector-0.12.0.1-3FWV4ejAWV0FsmvNvoLaed:Data.Vector.Internal.Check

What did I do wrong?

Remove momentum from Layers

Momentum shouldn't be stored in the layers any more. This will free us up to use a broader set of optimisation algorithms. We will however need to provide a class for fast updates and manipulations of learnable parameters.
Gradient associated type family shouldn't exist, we'll just return a Network with gradient weights.
randomNetwork shouldn't exist. Networks where all layers have a Random instance will also have a Random instance.

Separate optimisation from gradient calculation.

At the moment, we have the gradient calculations interwoven with the optimisation algorithm inside runBackwards. This is a bit terrible, as it means we can't give users a choice of algorithm.

It would be better for runBackwards to return just the gradients, and have training take a (hopefully minibatched) optimisation strategy.

Allowing Nesterov and SGD (with momentum) should be trivial, adagrad and friends might need a rethink in how we keep track of training updates.

Minibatch type family

Matrix-matrix multiplications are significantly faster than many matrix-vector ones. So minibatching over layers which do this is definitely worthwhile.

With improvements to how LSTM layers update, I think we could get a 20x speed increase for decent sized layers.

To do this, my thought is to have an injective type family for mini batches, and allow for either runForwards, or runBatchForwards (or both) to be written for each layer, with a default of each either lifting into a batch, or just running many in parallel sparks.

Biggest questions are, how to efficiently store the tapes (what is Tapes? this easier if I don't make the tape change, but I still think I should).

type family MiniBatch (n :: Nat) (s :: Shape) = (b :: Shape) | b -> n s where
  MiniBatch n ('D1 x) = ('D2 x n)
  MiniBatch n ('D2 x y) = ('D3 x y n)
  MiniBatch n ('D3 x y z) = ('D4 x y z n) | Vec n ('D3 x y n)

class UpdateLayer x => Layer x (i :: Shape) (o :: Shape) where
  ...
  runBatchForwards :: x -> S (MiniBatch n i)
    -> (Tapes n x i o, S (MiniBatch n o))
  runBatchBackwards :: x -> Tapes n x i o
    -> S (MiniBatch n o) -> S (MiniBatch n i)

Investigate using Hblas instead of hmatrix

From:

https://www.reddit.com/r/haskell/comments/4q5fwh/grenade_is_a_dependently_typed_practical_and/d4rvlwa

Using Recurrent and Concat together

I'm trying to use Recurrent and Concat together in the same network. In particular, I'm trying to run two LSTMs in parallel against different subsets of the input, and then want to Concat the results together.

I have something like this, so far:

type R = Recurrent
type F = FeedForward

type ShapeInput = 'D1 164

type CropOpponent = Crop 0 0 0 55
type CropPlayer = Crop 0 55 0 164

type LearnPlayer = RecurrentNetwork
    '[ F Reshape
    , F CropPlayer
    , F Reshape
    , R (LSTM 109 20)
    ]
    '[ ShapeInput
    , D2 1 164
    , D2 1 109
    , D1 109
    , D1 20
    ]

type LearnOpponent = RecurrentNetwork
    '[ F Reshape
    , F CropOpponent
    , F Reshape
    , R (LSTM 55 10)
    ]
    '[ ShapeInput
    , D2 1 164
    , D2 1 55
    , D1 55
    , D1 10
    ]

type RecNet = Network
    '[ Concat
        ShapeInput
        LearnPlayer
        ShapeInput
        LearnOpponent
    ]
    '[ ShapeInput
    , 'D1 30
    ]

randomNet :: MonadRandom m => m RecNet
randomNet = randomNetwork

On compilation, the error I'm getting is:

LearningBot.hs:69:13: error:
    • Couldn't match type ‘'False’ with ‘'True’
        arising from a use of ‘randomNetwork’
    • In the expression: randomNetwork
      In an equation for ‘randomNet’: randomNet = randomNetwork

Is there any way to accomplish what I'm looking for with Grenade right now?

P.S. I also tried this way, and got the same error:


type RecNet = RecurrentNetwork
    '[ F (
        Concat
            ShapeInput
            LearnPlayer
            ShapeInput
            LearnOpponent
        )
    ]
    '[ ShapeInput
    , 'D1 30
    ]

type RecInput = RecurrentInputs
    '[ F (
        Concat
            ShapeInput
            LearnPlayer
            ShapeInput
            LearnOpponent
        )
    ]

randomNet :: MonadRandom m => m (RecNet, RecInput)
randomNet = randomRecurrent

FYI: Builds, test and benchmarks work OK on OS X (Sierra) with stack-1.6.1 and ghc-8.2.2

All test and benchmarks run with my very relaxed stack.yaml (see below).

The following allow-newer bumps resulted:

WARNING: Ignoring out of range dependency (allow-newer enabled): criterion-1.2.6.0. grenade requires: ==1.1.*
WARNING: Ignoring out of range dependency (allow-newer enabled): optparse-applicative-0.14.0.0. grenade-examples requires: ==0.13.*

stack.yaml

system-ghc: true
resolver: nightly-2017-12-14
compiler-check: newer-minor
allow-newer: true

packages:
  - .
  - examples

Possible invalid memory access?

I think the im2col (and maybe col2im) can potentially read uninitialized memory: Running Grenade.Layers.Internal.Convolution.im2col 10 10 2 2 $ (5><6) [0..] gives a bunch of small float values, which seem likely to be because of reading invalid memory locations.

License?

Hi Huw,

The cabal file currently says "AllrightsReserved", but I don't think that's what you meant. Would you be able to specify a more liberal license and add an explicit LICENSE file?

Thanks!

Hackage release with composable recurrent nets

Upload grenade to Hackage (when its ready)

To be able to upload it to Hackage, I think we need to de-mafia-ize it.

Not sure if a Hackage upload was on your radar.

Shuffling data

Hi,

I'm fairly new to Haskell, so I apologize for the level of my question. I have been trying to build a network following the model from the convolutional example, but I cannot find the way to make it work with just one file, instead of one for training and another for testing.

My idea is to read the file, shuffle it and split it into the two different datasets.

Could you help me out with this, please?

Cannot get mnist example working

I am trying to get the mnist example working.

I have got all the tests passing and am using data downloaded from kaggle.

The output when I run it:
stack runghc examples/main/mnist.hs data/train.csv data/test.csv
Training convolutional neural network...
Failed reading: takeWhile1

Have I got the right dataset? Am I just missing something or just being stupid?

Switch to a standard Random class

At the moment the random initialisation is baked into the UpdateLayer class.

We could replace this with either Variates from System.Random.MWC or Random.

Gradient decent in C?

Why did you choose to write the gradient descent code in C, rather than using the library you used for the other matrix computations? Would you get a speedup by doing the descent in hblas?

Dropout unimplemented

Looking at the source of the dropout layer, it looks like it just passes through its input (both directions), and its seed and rate are unused.

Is it possible to create neural networks in run-time?

For example, I want to compare the performances of neural networks with different number of hidden layers. Instead of defining neural networks in compile-time, can I create neural networks in run-time by just inputting a list of number of hidden layers from a JSON file to a network generator function ?

Question about backprop

First, thanks for this cool library!

After reading Justin Le's articles about functional models (I really like the idea) I wonder why the grenade library is not using Justin Le's backprop library.

Connectionist Temporal Classification support?

Glad to see deeplearning implemented in my favourate language! Are there any plans to support that ? Or how can I use existing things provided by grenade to contribute?

Object detection

My apologies in advance for abusing the issues to ask a rather naive question.

I'm trying to gauge how far off some reasonably simple object detection would be in grenade? (Perhaps something in a similar vein to the tensorflow object detection api or YOLO)

Bug: network for XOR doesn't train correctly

I'm experiencing a bug and can't trace where it's coming from: my networks aren't actually training,
they just seem to randomly change the network parameters.
I simplified my code until now, all I'm using are the 'randomNetwork' and 'train' functions,
yet the bug seems to persist.

Details: You can find the code for the bug on https://github.com/Nickske666/grenade-examples/tree/bug
in app/main.hs. This executable only depends on your grenade (master branch, latest version).
I'm trying to train a two layer fully connected NN to approximate XOR. Here is the output before and after 100000 train-loops:

Before training:
S1D (-0.6153713089038197 :: R 1)
S1D (-0.6227542569188731 :: R 1)
S1D (-0.6152355742354048 :: R 1)
S1D (-0.6286478521211926 :: R 1)
After training:
S1D (0.3985667983943745 :: R 1)
S1D (0.4880564046752094 :: R 1)
S1D (0.5148131666098358 :: R 1)
S1D (0.5420005167827723 :: R 1)

This shows the network's predictions on the vectors [0, 0], [0, 1], [1, 0] and [1, 1],
which should be 0 1 1 0.
However, as you can see, it's not even close. I turned off the momentum and
regulator for this, and optimised the learning rate.

Is this a bug, or did I make a mistake here?

mafia build failure

There's a conflict with base==4.10.0.0 which is rejected because of the constraint that memory-0.12 is giving: base >= 4 && < 4.10.

How can I fix this?

I have tested this with ghc-8.2.1

full error message:

Cloning into 'mafia'...
remote: Counting objects: 2063, done.
remote: Total 2063 (delta 0), reused 0 (delta 0), pack-reused 2062
Receiving objects: 100% (2063/2063), 623.94 KiB | 391.00 KiB/s, done.
Resolving deltas: 100% (1117/1117), done.
HEAD is now at 3607163 Merge pull request #153 from ambiata/topic/project-terminology
Downloading the latest package list from hackage.haskell.org
Submodule 'lib/disorder' (https://github.com/ambiata/disorder.hs) registered for path 'lib/disorder'
Submodule 'lib/p' (https://github.com/ambiata/p) registered for path 'lib/p'
Submodule 'lib/twine' (https://github.com/ambiata/twine) registered for path 'lib/twine'
Submodule 'lib/x' (https://github.com/ambiata/x) registered for path 'lib/x'
Cloning into '/private/var/folders/vt/_b_21qtd12v2bl7qtl031zw00000gn/T/tmp.ekImw9m4/mafia/lib/disorder'...
Cloning into '/private/var/folders/vt/_b_21qtd12v2bl7qtl031zw00000gn/T/tmp.ekImw9m4/mafia/lib/p'...
Cloning into '/private/var/folders/vt/_b_21qtd12v2bl7qtl031zw00000gn/T/tmp.ekImw9m4/mafia/lib/twine'...
Cloning into '/private/var/folders/vt/_b_21qtd12v2bl7qtl031zw00000gn/T/tmp.ekImw9m4/mafia/lib/x'...
Submodule path 'lib/disorder': checked out 'c8625df58098d78cfa71cfb578bf72ccc2836706'
Submodule path 'lib/p': checked out 'fd88f397585914b1d897a5ec5dc5862a4fea9833'
Submodule path 'lib/twine': checked out '23d611a131785da696760000f148e7f14f26e277'
Submodule path 'lib/x': checked out '29d9c8c188f6f02a59228bd7649f13abceb1311a'
Writing a default package environment file to
/private/var/folders/vt/_b_21qtd12v2bl7qtl031zw00000gn/T/tmp.ekImw9m4/mafia/cabal.sandbox.config
Creating a new sandbox at
/private/var/folders/vt/_b_21qtd12v2bl7qtl031zw00000gn/T/tmp.ekImw9m4/mafia/.cabal-sandbox
Resolving dependencies...
cabal: Could not resolve dependencies:
trying: ambiata-mafia-0.0.1 (user goal)
trying: base-4.10.0.0/installed-4.1... (dependency of ambiata-mafia-0.0.1)
next goal: memory (dependency of ambiata-mafia-0.0.1)
rejecting: memory-0.14.16, memory-0.14.15, memory-0.14.14, memory-0.14.13,
memory-0.14.12, memory-0.14.11, memory-0.14.10, memory-0.14.9, memory-0.14.8,
memory-0.14.7, memory-0.14.6, memory-0.14.5, memory-0.14.4, memory-0.14.3,
memory-0.14.2, memory-0.14.1, memory-0.14, memory-0.13 (conflict:
ambiata-mafia => memory==0.12.*)
rejecting: memory-0.12 (conflict: base==4.10.0.0/installed-4.1..., memory =>
base>=4 && <4.10)
rejecting: memory-0.11, memory-0.10, memory-0.9, memory-0.8, memory-0.7,
memory-0.6, memory-0.5, memory-0.4, memory-0.3, memory-0.2, memory-0.1
(conflict: ambiata-mafia => memory==0.12.*)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: ambiata-mafia, base, memory
Note: when using a sandbox, all packages are required to have consistent
dependencies. Try reinstalling/unregistering the offending packages or
recreating the sandbox.

RNN or LSTM seq2seq

It would be great to have an example or implementation of some RNN model.

cabal collisions during install

Hey guys,

I am trying to build grenade by running ./mafila build and I am getting the following error:

$ ./mafia build
...
transformers-bifunctors-0.1 (via: mafia-0.0.1 mafia-0.0.1) (new package)
transformers-either-0.0.1 (latest: 0.1.1) (via: mafia-0.0.1 mafia-0.0.1) (new package)
mafia-0.0.1 (new package)
cabal: The following packages are likely to be broken by the reinstalls:
yaml-0.10.1.1
hlint-2.0.15
ghc-mod-5.8.0.0
libmpd-0.9.0.8
xmonad-extras-0.14
darcs-2.14.1
aeson-1.1.2.0
graphviz-2999.20.0.2
Use --force-reinstalls if you want to install anyway.
mafia version (fd85bf858719324e256ec178fd889a6752935b1d) could not be built.

How can I force the build script to use --force-reinstalls, or is there another workaround?

Thanks,
Vlad

Recurrent — cannot deduce typeclasses

Whatever I do, I cannot get working recurrent nets.

It keeps giving me two errors:

[ghcmod]
• Could not deduce (Grenade.Recurrent.Core.Network.CreatableRecurrent
                      MyLayers MyShapes)
    arising from a use of ‘randomRecurrent’

[ghcmod]
• No instance for (Num (RecurrentInputs MyLayers))
    arising from a use of ‘trainRecurrent’

where

type RR = Recurrent

type FF i o = RecurrentNetwork
  '[ RR (LSTM i o)]
  '[ 'D1 i, 'D1 o]

type MyLayers = '[ FF 3 6, FF 6 4, FF 4 1]
type MyShapes = '[ 'D1 3, 'D1 6, 'D1 4, 'D1 1]

bug: foreign import type mismatch

There is:

void descend_cpu(int len, double rate, ...);

and

foreign import ccall unsafe descend_cpu :: Int -> Double -> ... -> IO ()

but these don't match on amd64 Linux (int is 32bit and Int is 64bit).

Should probably be size_t in the C and CSize in the FFI import.

(Also to be pedantic it should be CDouble, which you can cast efficiently using coerce which should fail to compile on systems where Double /= CDouble, but I don't know if any exist. realToFrac is the wrong thing to use.)

Wrong category of family instance; declaration was for a type family

When I cabal build against the current base, and ghc 8.8.2, I get

src/Grenade/Core/Shape.hs:90:1: error:
    • Wrong category of family instance; declaration was for a type family
    • In the data instance declaration for ‘Sing’
   |
90 | data instance Sing (n :: Shape) where

Please advise.

Support ONNX?

Any idea if this would be a feasible thing to do? Nice idea to specify, train and verify models in Haskell then export for runtime in embedded environment, mobile or wherever.

grenade more easily built with stack than mafia

i tried the mafia build, but it failed immediately:

08:17 PM noon ∈ grenade (master*) ♫ ./mafia 
INFO: No explicit mafia version requested installing latest (39ec1dff2fbc9a0dad49e26f203f696d929ad316).
Building mafia-39ec1dff2fbc9a0dad49e26f203f696d929ad316 in /tmp/tmp.NF0BVAdorK
Cloning into '/tmp/tmp.NF0BVAdorK'...
remote: Enumerating objects: 27, done.
remote: Counting objects: 100% (27/27), done.
remote: Compressing objects: 100% (20/20), done.
remote: Total 2434 (delta 11), reused 15 (delta 4), pack-reused 2407
Receiving objects: 100% (2434/2434), 721.96 KiB | 207.00 KiB/s, done.
Resolving deltas: 100% (1352/1352), done.
HEAD is now at 39ec1df Merge pull request #239 from tmcgilchrist/topic/entwine
./bin/bootstrap: 12: ./bin/bootstrap: cabal: not found
mafia version (39ec1dff2fbc9a0dad49e26f203f696d929ad316) could not be built.

i then did stack init && stack build and that worked :)

Shakespeare - no decent results.

I ran shakespeare example overnight and got some awkward results on both hackage release packet:
shakespeare_output_from_clean_cabal_installation.txt
and also from freshly downloaded source code:
shakespeare_output_from_github_source.txt

I used the training data proposed in the source code:
https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt

How should I run the example to get this code to work and produce more realistic output (like the generated sequence from example)?

OpenCL support

I know its too much to ask, but can i expect OpenCL support for AMD/nVidia gpu cards?

I think haskell is best for pretty much anything, so I wish to have a haskell library that supports gpu computations and can ease the level of experimentation/research that I can do with neural nets!

TL, DR: Can grenade provide me performance on par with PyTorch using GPU?

Test suite fails

Hi, I'm trying to build grenade with nix, the test suite fails however:

━━━ Test.Grenade.Recurrent.Layers.LSTM ━━━
  ✗ prop_lstm_reference_forwards failed after 54 tests and 55 shrinks.

       ┏━━ test/Test/Grenade/Recurrent/Layers/LSTM.hs ━━━
    44 ┃ prop_lstm_reference_forwards =
    45 ┃   property $ do
    46 ┃     input :: S.R 3                       <- forAll randomVector
       ┃     │ (vector54319561065,0.7925737992825796,-6.308042121263235e-2] :: R 3)
    47 ┃     cell :: S.R 2                        <- forAll randomVector
       ┃     │ (vector102552342276,-0.5733412138993579] :: R 2)
    48 ┃     net@(LSTM lstmWeights _) :: LSTM 3 2 <- forAll genLSTM
       ┃     │ LSTM
    49 ┃
    50 ┃     let actual          = runRecurrentForwards net (S1D cell) (S1D input)
    51 ┃     case actual of
    52 ┃       (_, (S1D cellOut) :: S ('D1 2), (S1D output) :: S ('D1 2)) ->
    53 ┃         let cellOut'        = Reference.Vector . H.toList . S.extract $ cellOut
    54 ┃             output'         = Reference.Vector . H.toList . S.extract $ output
    55 ┃             refNet          = Reference.lstmToReference lstmWeights
    56 ┃             refCell         = Reference.Vector . H.toList . S.extract $ cell
    57 ┃             refInput        = Reference.Vector . H.toList . S.extract $ input
    58 ┃             (refCO, refO)   = Reference.runLSTM refNet refCell refInput
    59 ┃         in do toList refCO ~~~ toList cellOut'
       ┃         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       ┃         │ Failed (- lhs ~/~ + rhs)
       ┃         │ - [ 8.600932134870043e-2 , 0.10405281905297237 ]
       ┃         │ + [ 8.016375201087303e-2 , 7.653272311821224e-2 ]
    60 ┃               toList refO ~~~ toList output'

    This failure can be reproduced by running:
    > recheck (Size 53) (Seed 10790320388850313198 5063647523751707663) prop_lstm_reference_forwards

  ✗ prop_lstm_reference_backwards failed after 58 tests and 36 shrinks.

       ┏━━ test/Test/Grenade/Recurrent/Layers/LSTM.hs ━━━
    63 ┃ prop_lstm_reference_backwards =
    64 ┃   property $ do
    65 ┃     input :: S.R 3                       <- forAll randomVector
       ┃     │ (vector55046569861,0.7353593240191039,0.10220169420456582] :: R 3)
    66 ┃     cell :: S.R 2                        <- forAll randomVector
       ┃     │ (vector94712659202,8.446059985294041e-2] :: R 2)
    67 ┃     net@(LSTM lstmWeights _) :: LSTM 3 2 <- forAll genLSTM
       ┃     │ LSTM
    68 ┃     let (tape, _ :: S ('D1 2), _ :: S ('D1 2))
    69 ┃                                           = runRecurrentForwards  net (S1D cell) (S1D input)
    70 ┃         actualBacks                       = runRecurrentBackwards net tape (S1D (S.konst 1) :: S ('D1 2)) (S1D (S.konst 1) :: S ('D1 2))
    71 ┃     case actualBacks of
    72 ┃       (actualGradients, _, _ :: S ('D1 3)) ->
    73 ┃         let refNet          = Reference.lstmToReference lstmWeights
    74 ┃             refCell         = Reference.Vector . H.toList . S.extract $ cell
    75 ┃             refInput        = Reference.Vector . H.toList . S.extract $ input
    76 ┃             refGradients    = Reference.runLSTMback refCell refInput refNet
    77 ┃         in toList refGradients ~~~ toList (Reference.lstmToReference actualGradients)
       ┃         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       ┃         │ Failed (- lhs ~/~ + rhs)
       ┃         │   [
       ┃         │ -   -0.2833762365900625
       ┃         │ +   -0.2772291966623182
       ┃         │ - , 0.2792387920721928
       ┃         │ + , 0.273181502212966
       ┃         │ - , 3.8809159964732e-2
       ┃         │ + , 3.796730583209162e-2
       ┃         │ - , -1.8789963351874522e-2
       ┃         │ + , -1.9046776736865983e-2
       ┃         │ - , 1.851561984376432e-2
       ┃         │ + , 1.8768683616069006e-2
       ┃         │ - , 2.5733374902189056e-3
       ┃         │ + , 2.60850879413327e-3
       ┃         │ - , 0.3522269593369693
       ┃         │ + , 0.34458639918017514
       ┃         │ - , 3.2072315003399895e-2
       ┃         │ + , 3.1376597524498194e-2
       ┃         │ - , 2.3355281081871578e-2
       ┃         │ + , 2.36744913261781e-2
       ┃         │ - , 2.1266342964228386e-3
       ┃         │ + , 2.1557002473284216e-3
       ┃         │ - , 0.3797310824129001
       ┃         │ + , 0.3714938986832906
       ┃         │ - , 2.51790100961354e-2
       ┃         │ + , 2.5523146308241297e-2
       ┃         │ - , -0.14407943439166543
       ┃         │ + , -0.14095404163950748
       ┃         │ - , 0.14197579763956925
       ┃         │ + , 0.13889603729210484
       ┃         │ - , 1.9732077340779888e-2
       ┃         │ + , 1.930404615252405e-2
       ┃         │ - , 0.10233133248392495
       ┃         │ + , 0.10372995447129899
       ┃         │ - , -0.10083724033390323
       ┃         │ + , -0.1022154416927028
       ┃         │ - , -1.401455922896567e-2
       ┃         │ + , -1.4206104381415103e-2
       ┃         │ - , 0.17908580369842605
       ┃         │ + , 0.1752010475203034
       ┃         │ - , 1.6306804906884104e-2
       ┃         │ + , 1.5953075243230058e-2
       ┃         │ - , -0.12719434247358946
       ┃         │ + , -0.12893278171536413
       ┃         │ - , -1.1581785295885263e-2
       ┃         │ + , -1.1740080308514117e-2
       ┃         │ - , 0.19306996321689512
       ┃         │ + , 0.1888818605480774
       ┃         │ - , -0.13712648638597202
       ┃         │ + , -0.13900067402973101
       ┃         │ - , -0.11252340904695016
       ┃         │ + , -0.131574427655387
       ┃         │ - , 0.11088050713147728
       ┃         │ + , 0.12965337068552493
       ┃         │ - , 1.5410392325159424e-2
       ┃         │ + , 1.80194820553455e-2
       ┃         │ - , 4.787291093695013e-2
       ┃         │ + , 4.6151157876618276e-2
       ┃         │ - , -4.71739408493592e-2
       ┃         │ + , -4.5477326303979465e-2
       ┃         │ - , -6.556327661366858e-3
       ┃         │ + , -6.320528814073785e-3
       ┃         │ - , 0.1398627446668094
       ┃         │ + , 0.16354250849410923
       ┃         │ - , 1.2735317059880666e-2
       ┃         │ + , 1.489149739912809e-2
       ┃         │ - , -5.9504389135933686e-2
       ┃         │ + , -5.736430903441441e-2
       ┃         │ - , -5.418221013079177e-3
       ┃         │ + , -5.22335426217051e-3
       ┃         │ - , 0.1507841180627999
       ┃         │ + , 0.17631294858261248
       ┃         │ - , -6.415087061319925e-2
       ┃         │ + , -6.184367943473307e-2
       ┃         │ - , -0.6893313089947273
       ┃         │ + , -0.6743782306038787
       ┃         │ - , 0.6792667034381152
       ┃         │ + , 0.6645319479841436
       ┃         │ - , 9.440583078310466e-2
       ┃         │ + , 9.235796530850202e-2
       ┃         │ - , -0.7491175754191286
       ┃         │ + , -0.7593562021102653
       ┃         │ - , 0.7381800584171601
       ┃         │ + , 0.7482691956860943
       ┃         │ - , 0.10259372545373369
       ┃         │ + , 0.10399593371881949
       ┃         │ - , 0.9237208005000677
       ┃         │ + , 0.9036833100206666
       ┃         │ - , 1.00383585861486
       ┃         │ + , 1.0175558686009876
       ┃         │   ]

    This failure can be reproduced by running:
    > recheck (Size 57) (Seed 15557724053009315116 9838685224472302247) prop_lstm_reference_backwards

  ✗ prop_lstm_reference_backwards_input failed after 73 tests and 51 shrinks.

       ┏━━ test/Test/Grenade/Recurrent/Layers/LSTM.hs ━━━
    79 ┃ prop_lstm_reference_backwards_input =
    80 ┃   property $ do
    81 ┃     input :: S.R 3                       <- forAll randomVector
       ┃     │ (vector132148291512,0.2850784935453341,-1.40887764348131e-2] :: R 3)
    82 ┃     cell :: S.R 2                        <- forAll randomVector
       ┃     │ (vector429500642426,0.7481729899291754] :: R 2)
    83 ┃     net@(LSTM lstmWeights _) :: LSTM 3 2 <- forAll genLSTM
       ┃     │ LSTM
    84 ┃     let (tape, _ :: S ('D1 2), _ :: S ('D1 2))
    85 ┃                                           = runRecurrentForwards  net (S1D cell) (S1D input)
    86 ┃         actualBacks                       = runRecurrentBackwards net tape (S1D (S.konst 1) :: S ('D1 2)) (S1D (S.konst 1) :: S ('D1 2))
    87 ┃     case actualBacks of
    88 ┃       (_, _, S1D actualGradients :: S ('D1 3)) ->
    89 ┃         let refNet          = Reference.lstmToReference lstmWeights
    90 ┃             refCell         = Reference.Vector . H.toList . S.extract $ cell
    91 ┃             refInput        = Reference.Vector . H.toList . S.extract $ input
    92 ┃             refGradients    = Reference.runLSTMbackOnInput refCell refNet refInput
    93 ┃         in toList refGradients ~~~ H.toList (S.extract actualGradients)
       ┃         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       ┃         │ Failed (- lhs ~/~ + rhs)
       ┃         │   [
       ┃         │ -   0.41783498961490245
       ┃         │ +   -7.144093770897443e-2
       ┃         │ - , -0.643583384088483
       ┃         │ + , -0.5033482252382088
       ┃         │ - , 0.24741239536602957
       ┃         │ + , 0.2196259366479876
       ┃         │   ]

    This failure can be reproduced by running:
    > recheck (Size 72) (Seed 15511439570864369243 14061565670218007919) prop_lstm_reference_backwards_input

  ✗ prop_lstm_reference_backwards_cell failed after 45 tests and 31 shrinks.

        ┏━━ test/Test/Grenade/Recurrent/Layers/LSTM.hs ━━━
     95 ┃ prop_lstm_reference_backwards_cell =
     96 ┃   property $ do
     97 ┃     input :: S.R 3                       <- forAll randomVector
        ┃     │ (vector957622892669,-0.26370740740732634,-0.8739678849810585] :: R 3)
     98 ┃     cell :: S.R 2                        <- forAll randomVector
        ┃     │ (vector544704658233,-0.82774989764567] :: R 2)
     99 ┃     net@(LSTM lstmWeights _) :: LSTM 3 2 <- forAll genLSTM
        ┃     │ LSTM
    100 ┃     let (tape, _ :: S ('D1 2), _ :: S ('D1 2))
    101 ┃                                           = runRecurrentForwards  net (S1D cell) (S1D input)
    102 ┃         actualBacks                       = runRecurrentBackwards net tape (S1D (S.konst 1) :: S ('D1 2)) (S1D (S.konst 1) :: S ('D1 2))
    103 ┃     case actualBacks of
    104 ┃       (_, S1D actualGradients, _ :: S ('D1 3)) ->
    105 ┃         let refNet          = Reference.lstmToReference lstmWeights
    106 ┃             refCell         = Reference.Vector . H.toList . S.extract $ cell
    107 ┃             refInput        = Reference.Vector . H.toList . S.extract $ input
    108 ┃             refGradients    = Reference.runLSTMbackOnCell refInput refNet refCell
    109 ┃         in toList refGradients ~~~ H.toList (S.extract actualGradients)
        ┃         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        ┃         │ Failed (- lhs ~/~ + rhs)
        ┃         │ - [ 0.3555130556015041 , 4.6592664606872156e-2 ]
        ┃         │ + [ 0.4057801473816573 , -1.403610494537344e-2 ]
    110 ┃
    111 ┃ (~~~) :: (Monad m, Eq a, Ord a, Num a, Fractional a, Show a, HasCallStack) => [a] -> [a] -> PropertyT m ()
    112 ┃ (~~~) x y =
    113 ┃   if all (< 1e-8) (zipWith (-) x y) then
    114 ┃     success
    115 ┃   else
    116 ┃     case valueDiff <$> mkValue x <*> mkValue y of
    117 ┃       Nothing ->
    118 ┃         withFrozenCallStack $
    119 ┃           failWith Nothing $ unlines [
    120 ┃               "━━━ Not Simliar ━━━"
    121 ┃             , showPretty x
    122 ┃             , showPretty y
    123 ┃             ]
    124 ┃       Just diff ->
    125 ┃         withFrozenCallStack $
    126 ┃           failWith (Just $ Diff "Failed (" "- lhs" "~/~" "+ rhs" ")" diff) ""

    This failure can be reproduced by running:
    > recheck (Size 44) (Seed 17734744653642261044 4653266027254713753) prop_lstm_reference_backwards_cell

  ✗ 4 failed.
Test suite test: FAIL
Test suite logged to: dist/test/grenade-0.1.0-test.log
0 of 1 test suites (0 of 1 test cases) passed.
builder for '/nix/store/g2ishp6kxmakfxpxxl0l58m0n6w7dfzi-grenade-0.1.0.drv' failed with exit code 1
cannot build derivation '/nix/store/djy93n6n0wjvbv85a1g9s6nc7hh0nxq9-ghc-8.4.3-with-packages.drv': 1 dependencies couldn't be built
error: build of '/nix/store/djy93n6n0wjvbv85a1g9s6nc7hh0nxq9-ghc-8.4.3-with-packages.drv' failed
make[1]: *** [makefile:70: enter] Fout 100
make[1]: Map '/home/jappie/projects/raster' wordt verlaten

I used cabal2nix to just grab the latest version from github, this one to be precise:

{ mkDerivation, ad, base, bytestring, cereal, constraints
, containers, criterion, deepseq, fetchgit, hedgehog, hmatrix
, MonadRandom, mtl, primitive, random, reflection, singletons
, stdenv, text, transformers, typelits-witnesses, vector
}:
mkDerivation {
  pname = "grenade";
  version = "0.1.0";
  src = fetchgit {
    url = "https://github.com/HuwCampbell/grenade";
    sha256 = "1nzzacpiz86z3svvi2lpczzkv4algjfw3m2h2pqsw5n44xp8b320";
    rev = "f729b291e40a9044d95d49d31bce3149a5b9d599";
    fetchSubmodules = true;
  };
  libraryHaskellDepends = [
    base bytestring cereal containers deepseq hmatrix MonadRandom
    primitive singletons vector
  ];
  testHaskellDepends = [
    ad base constraints hedgehog hmatrix MonadRandom mtl random
    reflection singletons text transformers typelits-witnesses vector
  ];
  benchmarkHaskellDepends = [ base bytestring criterion hmatrix ];
  description = "Practical Deep Learning in Haskell";
  license = stdenv.lib.licenses.bsd2;
}

I'll disable the tests for now so it's not a big deal, but I just wanted to let you know.

Do you have non-mafia install tips?

Amazing library. Great work.

I tried installing with stack and had some version conflicts. Is it possible there is a non-mafia installer?

Concurrency?

I'm trying to speed up a program that's using Grenade, and a) ran into MVar errors (my fault) and b) noticed that it seems to only run single-threaded, even if I'm using GHC concurrency. Is there something under the hood that would limit network execution to a single core at a time?

ghc-8.6 compatibility

There are a few things that need to be fixed for ghc-8.6:

-XNoStarIsType -- otherwise GHC thinks a * b is using the * as type, not type operator. Also need to import Data.Kind (Type) to replace real uses of * as type
-XUndecidableInstances -- otherwise GHC complains about nested KnownNat (a * b)
containers-0.6
hmatrix-0.19

I need to check how best to handle older GHC versions with these changes, will do a pull request soon - how far back do you want to support?