Coder Social home page Coder Social logo

huntermcgushion / hyperparameter_hunter Goto Github PK

View Code? Open in Web Editor NEW
702.0 24.0 103.0 7.45 MB

Easy hyperparameter optimization and automatic result saving across machine learning algorithms and libraries

License: MIT License

Python 99.28% Makefile 0.05% Shell 0.68%
artificial-intelligence machine-learning hyperparameter-optimization hyperparameter-tuning neural-network keras scikit-learn xgboost lightgbm catboost deep-learning data-science python rgf sklearn optimization experimentation feature-engineering ai ml

hyperparameter_hunter's Issues

Keras learning rate recorded incorrectly when decay/scheduling callbacks used

  • See :meth:recorders.DescriptionRecorder.format_result
  • Model's get_config() returns the final learning rate, rather than the initial one, so experiment description files are misleading by not displaying the actual value used
  • Leads to failed similar experiment matches
  • Experiment started with Adam at lr=0.001, and ReduceLROnPlateau, which dropped the lr down to 0.0001
    • 0.0001 was recorded as the Experiment's lr, but it should be 0.001
  • Probably need to call parameterize_compiled_keras_model immediately after initializing it, then store the results, then use them in the DescriptionRecorder
    • Midway through experiments.BaseCVExperiment.cv_run_workflow, or in models.KerasModel.initialize_model/fit

Remove Keras dependence in `key_handler`

  • Remove dependence on keras.callbacks.Callback
  • Only usage in key_handler.KeyMaker.handle_complex_types.visit function
  • Probably need to wire in import hooks, since Keras actually should be used here if a Keras model_initializer is given

Leaderboard conflict with aliased, and non-aliased metrics

  • Resolve issue noted in leaderboards.GlobalLeaderboard.add_entry, where aliased metrics names should be merged together based on their equivalent hashes
  • Currently, using SKLearn's roc_auc_score, then using the same function under an alias (like 'roc') would produce two separate columns: 'roc_auc_score', and 'roc'
    • This is despite the fact that the two metrics are, in fact, the same thing

Finish `leaderboards` documentation

  • Add documentation description for leaderboards.Leaderboard.__init__
  • Add documentation for leaderboards.GlobalLeaderboard.add_entry (See Leaderboard implementation)

UninformedOptimizationProtocols need `current_hyperparameters_list`

  • Add current_hyperparameters_list equivalent to optimization_core.UninformedOptimizationProtocol
  • See usages in optimization_core.InformedOptimizationProtocol for proper implementation
  • Only used by optimization_core.BaseOptimizationProtocol for logging in the _optimization_loop method (in which pertinent flag comments are located)
  • This bug breaks the children of UninformedOptimizationProtocol, which is very not good

`n_random_starts` broken in `optimization_core.SKOptimizationProtocol.__init__`

  • Make optimization_core.SKOptimizationProtocol.__init__.n_random_starts actually do something when specified
  • The kwarg is currently ignored if a sufficient number of experiment results have already been read in
    • This makes the SKOptimizationProtocol think the requirement has already been satisfied
  • Random starts are only actually executed when n_random_starts-many result files cannot be located

Hide internally-used `experiments.BaseExperiment` methods

  • Hide the following methods of experiments.BaseExperiment that generally shouldn't be used by class instances:
    • additional_preparation_steps
    • initial_preprocessing
    • validate_parameters
    • validate_environment
    • clean_up
    • generate_experiment_id
    • generate_hyperparameter_key
    • create_script_backup
    • initialize_random_seeds
    • random_seed_initializer
    • update_model_params

Add default hyperparameter search ranges

  • Declare default hyperparameter ranges/selections for certain libraries/algorithms in files named for each library in the hyperparameter_hunter/library_helpers directory
  • These should be used by optimization_core.BaseOptimizationProtocol.add_default_options when completed by #31

Perform Keras layer interception in project's `__init__.py`

  • Perform call to importer.hook_keras_layer near top of __init__.py
  • Currently called before any other imports
    • See examples.keras_example.py for current usage - This will need to be removed
  • Verify hook_keras_layer does not raise any exceptions if Keras has not been installed

Documentation for `models.KerasModel`

  • Finish documentation for the following methods of models.KerasModel:
    • __init__.
      • initialization_params, and extra_params kwargs
    • initialize_model
    • fit,
    • get_input_dim
    • validate_keras_params
    • initialize_keras_neural_network

`tell` optimizer positive/negative utility values depending on `target_metric`

  • Update the following methods of optimization_core.InformedOptimizationProtocol:
    • _execute_experiment
    • _find_similar_experiments
  • The two aforementioned methods are the two locations at which optimization_core.InformedOptimizationProtocol.optimizer is "tell-ed" the utility value of a set of hyperparameters
  • Currently, a negative utility value is provided to optimizer, which will cause problems if target_metric should be minimized
    • This is the case when target_metric is some loss measure
  • Need to add a means of specifying that positive utility values should be used, instead of negative, or of detecting that target_metric measures loss

Separate input/target data for `environment.Environment.__init__`

  • Add ability to provide separate input/target DataFrames for following environment.Environment.__init__ kwargs: train_dataset, holdout_dataset, and test_dataset
  • Accept NumPy arrays, instead of DataFrames
  • Alternative to providing the whole DataFrame, containing a target column

`models.XGBoostModel.fit` `eval_set` behavior

  • Remove the default inclusion of eval_set in models.XGBoostModel.fit per todo comment
  • This results in unexpectedly long execution times
  • models.XGBoostModel.fit has been commented out, meaning models.Model.fit is being used
  • The updated version of models.XGBoostModel.fit should still accommodate eval_set and eval_metric arguments

Documentation for `optimization_core.BaseOptimizationProtocol`

  • Add documentation for following optimization_core.BaseOptimizationProtocol methods:
    • _optimization_loop
    • _update_current_hyperparameters
    • _set_hyperparameter_space
    • _get_current_hyperparameters
    • search_space_size (See InformedOptimizationProtocol implementation)

Implement `optimization_core.BaseOptimizationProtocol.add_default_options`

  • Complete the optimization_core.BaseOptimizationProtocol.add_default_options method
  • This will need to play nice with the BaseOptimizationProtocol.hyperparameter_space attribute
    • Likely requires updating space.Space to reflect new default options being added to original dimensions (if InformedOptimizationProtocol)
  • The implemented add_default_options should leverage the default hyperparameter search ranges added in #30 for the hyperparameter provided as input and optimization_core.BaseOptimizationProtocol.model_initializer

Clean up `optimization_utils.AskingOptimizer.__init__`

  • Problem: skopt.optimizer.Optimizer.__init__ is copied almost verbatim by optimization_utils.AskingOptimizer.__init__, which is far from ideal
    • This is copied in order to make AskingOptimizer use hyperparameter_hunter.space.Space, rather than skopt.space.Space
  • Need way to tell skopt.optimizer.Optimizer.__init__ to use updated Space, or need to override the particular section of skopt.optimizer.Optimizer.__init__, in which skopt.space.Space is used
  • In its current state, any changes to skopt.optimizer.Optimizer.__init__ will be completely lost, and will need to be manually recreated
  • Solution still needs to accommodate __repeated_ask_kwargs, as noted in the pertinent todo comments and the original optimization_utils.AskingOptimizer.__init__, which is commented out above the current monstrosity

Finish `experiments.BaseExperiment.__init__` documentation

  • Add documentation for the target_metric kwarg of experiments.BaseExperiment.__init__
  • Label the following experiments.BaseExperiment.__init__ kwargs as experimental while in development: preprocessing_pipeline, preprocessing_params

Documentation for `reporting.ReportingHandler`

  • Add documentation for the following reporting.ReportingHandler methods:
    • validate_parameters
    • configure_reporting_type
    • initialize_logging_logging
    • configure_console_logger_handler
    • configure_heartbeat_logger_handler
    • _logging_log
    • _logging_debug
    • _logging_warn

Keras dependence in `models`

  • Remove Keras dependence in models, unless keras.models.load_model required by models.KerasModel.fit
    • This will only be the case if models.KerasModel is actually in use
  • May need to use Keras import hooks from importer inside hyperparameter_hunter.__init__

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.