Comments (2)
Thanks for raising this!
You’re absolutely right. Currently, we don’t support optimizing both a Keras optimizer
object and its parameters at the same time. As you say, optimizing nested parameters in general currently isn’t supported. However the Keras optimizer
s are specifically problematic because they each have different parameters.
While all optimizers do have an lr
kwarg, some have other kwargs that different optimizers don’t have. For example, an attempt to search through optimizers “sgd” and “rmsprop”, as well as momentum
gets weird because RMSprop
doesn’t have a momentum
kwarg. Furthermore, many optimizers have different default lr
values.
As you mentioned, this could probably be addressed by adding support for nested/conditional hyperparameter optimization, but I still think it gets messy rather quickly in the case of Keras optimizers. This messiness is compounded when we consider that HH is going to automatically read all of our old Experiments and try to match them with the new search space to jump-start optimization. All the old Experiment results, of course, will also have the default values for hyperparameters that weren’t explicitly given.
All that said, I would very much like to support nested/conditional dimensions…
Could you please clarify what you mean by,
This way i can return optimizers that can’t be called by name (Ie. Adamax or Nadam)
If you’re referring to using Categorical
on the string names of optimizers, then “adamax” and “nadam” do work; however, I think I may have misunderstood.
…
The problem is i don’t think you can call any non-native function inside your build_fn…
That’s correct. I won’t bore you with all the technical details, but HH actually rewrites the Keras build_fn
its given before evaluating it, and this currently leaves out any non-native or non-Keras stuff, which is the reason for the NameError you’re getting.
Doing this may seem unnecessarily complicated, but the reason for it is to enable providing search dimensions directly to Keras layers. Otherwise it wouldn’t be possible to do something like Dense(Integer(50, 150))
because Dense
is expecting a native Python int
, not an Integer
instance, as its first argument.
If you have any ideas or would actually like some more technical details, I’d be more than happy to discuss it further, but that’s the “short” answer to why you’re getting that NameError.
Your wrapper function approach does seem really interesting, though, and I’m trying to figure out how we could integrate it into HH to support nested/conditional dimensions.
from hyperparameter_hunter.
If you’re referring to using Categorical on the string names of optimizers, then “adamax” and “nadam” do work; however, I think I may have misunderstood.
adamax and nadam are callable by name? I wasn't aware. Either way i think we'd have the same problem when using custom optimizers like adabound, that really have to be defined in a wrapper function.
Your wrapper function approach does seem really interesting, though, and I’m trying to figure out how we could integrate it into HH to support nested/conditional dimensions.
Maybe it is possible to allow a list of methods as params in the build_fn function? Then if we annotated our wrappers HH could possibly supply the appropriate function by name. This way we can supply our own wrapper functions. I'm probably over simplifying things...
from hyperparameter_hunter.
Related Issues (20)
- Support for various advanced functionality? HOT 9
- Unable to access docs HOT 2
- How to do predict_proba in catboost classifier? HOT 3
- How to handle variable number of layers?
- How can I set class weights in a multiclass classification with imbalance dataset? HOT 1
- How to use the Experiments? HOT 6
- OSError: could not get source code HOT 9
- Q: what is RandomForestOptPro exactly? HOT 3
- Doc improvement suggestion HOT 4
- logloss issue with multiclass task HOT 4
- Metaclass conflict with Keras 2.3.0 HOT 1
- Timeseries Cross Validation HOT 6
- Any way to send hyperparameter_hunter.Integer to custom feature_engineer function? HOT 3
- pd.DataFrame.sparse supported in environment?
- Interface with pytorch or tensorflow models HOT 2
- Problem with library scikit-optimize in Python 3.6 HOT 9
- ROC AUC scores don't match to those of sklearn HOT 1
- Failed to import packages from hyperparameter_hunter HOT 3
- ImportError: cannot import name 'Log10' HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hyperparameter_hunter.