shadiakiki1986 / keras-models-factory

Factory generating keras models, with unit tests proving they work

keras-models-factory

Factory generating keras models, with unit tests proving they work.

The repository includes integration tests showing that the networks perform well on certain datasets.

The tests include caching so that if a test is interrupted and re-launched, it resumes from where it had reached.

The idea is also available in the keras integration tests

LICENSE

Installation

pip3 install pew
pew new TEST_KERAS_MODELS_FACTORY
pip install git+https://github.com/shadiakiki1986/keras-models-factory.git

For GPU instances (e.g. AWS EC2 p2.xlarge): run init-gpu.sh

Usage

To test a keras model (copy one of the integration tests in tests)

To use a model from the factory:

from keras_models_factory import lstm

# stacked lstm
m1 = lstm.model_1(in_neurons=10, lstm_dim=[8,6], look_back=5)
m1.summary()
# ...

# single lstm followed by dense layers
m2 = lstm.model_2(in_neurons=10, lstm_dim=[8,6], look_back=5)
m2.summary()
# _________________________________________________________________
# Layer (type)                 Output Shape              Param #
# =================================================================
# lstm_1 (LSTM)                (None, 8)                 608
# _________________________________________________________________
# dense_1 (Dense)              (None, 6)                 54
# _________________________________________________________________
# dense_2 (Dense)              (None, 1)                 7
# =================================================================
# Total params: 669
# Trainable params: 669
# Non-trainable params: 0
# _________________________________________________________________

Testing

sudo apt-get install python3-tk

pip3 install pew
pew new KERAS_MODELS_FACTORY

# http://stackoverflow.com/questions/28509965/ddg#28842733
pip install -e .[dev]

# for GPU usage, check
# https://gist.github.com/shadiakiki1986/0c9ea999113691fb9a7ae64e3541fe29#file-init-gpu-sh

# run tests
nosetests --logging-level INFO --nocapture

# or run individual tests
# http://pythontesting.net/framework/specify-test-unittest-nosetests-pytest/
# https://nose.readthedocs.io/en/latest/plugins/logcapture.html
nosetests --logging-level INFO --nocapture -v tests/test_lstm_ds1.py:TestLstmDs1.test_fit_model_1
nosetests --logging-level INFO --nocapture -v tests/test_lstm_ds1.py:TestLstmDs1.test_fit_model_2

Note that ATM I moved some LSTM slow tests into

• tests/data/params_lstm_1_slow.yml
• and tests/data/params_lstm_2_slow.yml

To run tests in parallel

sudo apt-get install parallel
ls tests/test*py -1|pew in KERAS_MODELS_FACTORY parallel nosetests --logging-level INFO --nocapture -v {}

Note that the nosetests parameter --processes doesnt work and yields a CUDA_ERROR_NOT_INITIALIZED

Dev notes

In order for import keras_models_factor as kmf; help(kmf.datasets) to work, I use the keras_models_factory/__init__.py to have the imports. Ref: https://stackoverflow.com/a/46242108/4126114

keras-models-factory's Issues

batch_size = 10e3 => bad MSE ?

• in params_1.yml (or 2), I tried one of the tests with batch_size = 10e3
• that yielded a horrible result
• I would have thought that it wouldn't have any effect on the MSE
• could investigate if I have some silly problem in my code
  • or if it indeed makes sense?

EDIT: can also try test_lstm_ds3 with batch_size = 100 and see if can achieve same RMSE

add test showing test-ml/t5-lstm/p5c

This sample showed that input + LSTM + Dense + output do not work together

Add test for this

For reference

# https://cmsdk.com/python/lstm--learn-a-simple-lag-in-the-data.html
def build_lstm_vanilla(in_neurons:int, out_neurons:int, lstm_dim:int, enc_dim:int=None):
  model = Sequential()
  model.add(LSTM(lstm_dim,
    return_sequences=False,
    input_shape=(None, in_neurons),
    stateful=False,                                     
    activation='tanh'
    ))
  # can use "dropout" parameter of LSTM(...) constructor instead of the below
  # model.add(Dropout(0.25))                            
                                                        
  # DOESNT WORK as demonstrated in test-ml/t5-lstm/p5c  
  if enc_dim is not None:                               
    model.add(Dense(enc_dim, activation='tanh'))        
  
  model.add(Dense(out_neurons, activation='linear'))    
    
  model.compile(loss="mean_squared_error", optimizer="rmsprop")
  
  model.summary()
  return model

new dataset: conx cosine

https://conx.readthedocs.io/en/latest/LSTM.html

test for over-fitting?

• perhaps I can add to assert_fit_model a few lines that check that the val_loss isn't much larger than the loss
  • the case of which would signify over-fitting

add integration tests for datasets.py / ds_2

• move tests/test_lstm.py to tests/test_lstm_ds1.py
• add new file tests/test_lstm_ds2.py, possibly inheriting from the ds1 test file
• factor out common code if possible

add dataset and integration test for airline passengers

• tutorial here

lstm stacked model: horizontal to vertical stacking

• the current model isa horizontally stacked model (on time axis)
• need to use vertical stacking (with return_values=False)

new architecture: Generative Adversarial Network Deep Learning Framework

• gandlf seems like an easy-to-add architecture

new architecture: lstm-based autoencoder

• the code for this is already in utils*.py

new architecture: autoencoders

• the code for these networks is already in utils.py or utils2.py

add `assert_fit_model_to_all_datasets`

• no need for me to specify the dataset
• test against all
• obviously LSTM datasets may be different than AE datasets
  • need to think of how to preprocess the AE dataset to become a LSTM dataset (by striding)
  • maybe move the striding out of the ds1 and into some function for lstm

integrate cmsdk example on simple lag

https://cmsdk.com/python/lstm--learn-a-simple-lag-in-the-data.html