cerlymarco / keras-hypetune

A friendly python package for Keras Hyperparameters Tuning based only on NumPy and hyperopt.

keras-hypetune

A friendly python package for Keras Hyperparameters Tuning based only on NumPy and Hyperopt.

Overview

A very simple wrapper for fast Keras hyperparameters optimization. keras-hypetune lets you use the power of Keras without having to learn a new syntax. All you need it's just create a python dictionary where to put the parameter boundaries for the experiments and define your Keras model (in any format: Functional or Sequential) inside a callable function.

def get_model(param):
        
    model = Sequential()
    model.add(Dense(param['unit_1'], activation=param['activ']))
    model.add(Dense(param['unit_2'], activation=param['activ']))
    model.add(Dense(1))
    model.compile(optimizer=Adam(learning_rate=param['lr']), 
                  loss='mse', metrics=['mae'])
    
    return model

The optimization process is easily trackable using the callbacks provided by Keras. At the end of the searching, you can access all you need by querying the keras-hypetune searcher. The best solutions can be automatically saved in proper locations.

Installation

pip install --upgrade keras-hypetune

Tensorflow and Keras are not needed requirements. keras-hypetune is specifically for tf.keras with TensorFlow 2.0. The usage of GPU is normally available.

Fixed Validation Set

This tuning modality operates the optimization on a fixed validation set. The parameter combinations are evaluated always on the same set of data. In this case, it's allowed the usage of any kind of input data format accepted by Keras.

KerasGridSearch

All the passed parameter combinations are created and evaluated.

param_grid = {
    'unit_1': [128,64], 
    'unit_2': [64,32],
    'lr': [1e-2,1e-3], 
    'activ': ['elu','relu'],
    'epochs': 100, 
    'batch_size': 512
}

kgs = KerasGridSearch(get_model, param_grid, monitor='val_loss', greater_is_better=False)
kgs.search(x_train, y_train, validation_data=(x_valid, y_valid))

KerasRandomSearch

Only random parameter combinations are created and evaluated.

The number of parameter combinations that are tried is given by n_iter. If all parameters are presented as a list, sampling without replacement is performed. If at least one parameter is given as a distribution (from scipy.stats random variables), sampling with replacement is used.

param_grid = {
    'unit_1': [128,64], 
    'unit_2': stats.randint(32, 128),
    'lr': stats.uniform(1e-4, 0.1), 
    'activ': ['elu','relu'],
    'epochs': 100, 
    'batch_size': 512
}

krs = KerasRandomSearch(get_model, param_grid, monitor='val_loss', greater_is_better=False, 
                        n_iter=15, sampling_seed=33)
krs.search(x_train, y_train, validation_data=(x_valid, y_valid))

KerasBayesianSearch

The parameter values are chosen according to bayesian optimization algorithms based on gaussian processes and regression trees (from hyperopt).

The number of parameter combinations that are tried is given by n_iter. Parameters must be given as hyperopt distributions.

param_grid = {
    'unit_1': 64 + hp.randint('unit_1', 64),
    'unit_2': 32 + hp.randint('unit_2', 96),
    'lr': hp.loguniform('lr', np.log(0.001), np.log(0.02)), 
    'activ': hp.choice('activ', ['elu','relu']),
    'epochs': 100, 
    'batch_size': 512
}

kbs = KerasBayesianSearch(get_model, param_grid, monitor='val_loss', greater_is_better=False, 
                          n_iter=15, sampling_seed=33)
kbs.search(x_train, y_train, trials=Trials(), validation_data=(x_valid, y_valid))

Cross Validation

This tuning modality operates the optimization using a cross-validation approach. The CV strategies available are the same provided by scikit-learn splitter classes. The parameter combinations are evaluated on the mean score of the folds. In this case, it's allowed the usage of only numpy array data. For tasks involving multi-input/output, the arrays can be wrapped into list or dict like in normal Keras.

KerasGridSearchCV

All the passed parameter combinations are created and evaluated.

param_grid = {
    'unit_1': [128,64], 
    'unit_2': [64,32],
    'lr': [1e-2,1e-3], 
    'activ': ['elu','relu'],
    'epochs': 100, 
    'batch_size': 512
}

cv = KFold(n_splits=3, random_state=33, shuffle=True)

kgs = KerasGridSearchCV(get_model, param_grid, cv=cv, monitor='val_loss', greater_is_better=False)
kgs.search(X, y)

KerasRandomSearchCV

Only random parameter combinations are created and evaluated.

The number of parameter combinations that are tried is given by n_iter. If all parameters are presented as a list, sampling without replacement is performed. If at least one parameter is given as a distribution (from scipy.stats random variables), sampling with replacement is used.

param_grid = {
    'unit_1': [128,64], 
    'unit_2': stats.randint(32, 128),
    'lr': stats.uniform(1e-4, 0.1), 
    'activ': ['elu','relu'],
    'epochs': 100, 
    'batch_size': 512
}

cv = KFold(n_splits=3, random_state=33, shuffle=True)

krs = KerasRandomSearchCV(get_model, param_grid, cv=cv, monitor='val_loss', greater_is_better=False,
                          n_iter=15, sampling_seed=33)
krs.search(X, y)

KerasBayesianSearchCV

The parameter values are chosen according to bayesian optimization algorithms based on gaussian processes and regression trees (from hyperopt).

The number of parameter combinations that are tried is given by n_iter. Parameters must be given as hyperopt distributions.

param_grid = {
    'unit_1': 64 + hp.randint('unit_1', 64),
    'unit_2': 32 + hp.randint('unit_2', 96),
    'lr': hp.loguniform('lr', np.log(0.001), np.log(0.02)), 
    'activ': hp.choice('activ', ['elu','relu']),
    'epochs': 100, 
    'batch_size': 512
}

cv = KFold(n_splits=3, random_state=33, shuffle=True)

kbs = KerasBayesianSearchCV(get_model, param_grid, cv=cv, monitor='val_loss', greater_is_better=False,
                            n_iter=15, sampling_seed=33)
kbs.search(X, y, trials=Trials())

keras-hypetune's Issues

Multi-input functional model: ValueError inconsistent number of samples

Hi Marco,

This is a really great package, but I'm having trouble getting it to work. I have a model which takes text and audio features (numpy arrays) which I am passing to search, as with the Keras fit() method:

print(tf_train.shape, af_train.shape, y_train.shape)
kgs.search({'text_inputs': tf_train, 'audio_inputs': af_train}, {'labels': y_train})

I am getting the output and error below:

(61, 400) (61, 400, 13) (61, 3)
...
ValueError: Found input variables with inconsistent numbers of samples: [2, 1]

I have tried passing the features in a list and the labels as-is (this resembles your notebook example), but with the same error:

print(tf_train.shape, af_train.shape, y_train.shape)
kgs.search([tf_train, af_train], y_train)

ValueError: Found input variables with inconsistent numbers of samples: [2, 61]

Any ideas?

Thanks!

André

How to output the best parameter values

Hi Cerly,

This package is great, but how to output the best parameter values?

Thank you,
Ling

Sample weights

Hi, maybe this is not an issue but something I am missing. Is there a way to enter sample weights in the search? Any help would be appreciated. Thanks,

ModuleNotFoundError: No module named 'keras_hypetune'

I'm having issues loading the package in my scripts. Is keras_hypetune the correct name to import under?