Conditional/Inactive hyperparameters about syne-tune

[geoalgo]

Hi Sebastian,

Syne Tune does not support conditional hyperparameters natively yet.

I see several several options:

1. you can write your own TrialSheduler (see this example). There you can write your own suggestion function and apply whatever logic you want (you could then use ConfigSpace for instance which supports hierarchical spaces).

2. You can override a scheduler and change _postprocess_config to set to None some hyperparameters conditioned on the sampled value.

3. Add support for those type of sampling in config_space as you suggested.

The downside of 1. is that you will have to rewrite your optimizer however, this is something you likely would have to do as our BO does not support such spaces for instance. The upside is that you have all flexibility to add your method.

2. Is a bit in-between, you may be able to reuse some of the schedulers with this approach.

3. Is the cleanest but also an investment to be made to add clean support for conditional space.

Hope this helps, thanks for the issue!

In any case, let us know your thoughts and if you implement one of the approach we would be glad to receive your contribution as an example (easier) or an update to config-space (nice but probably longer and harder).

[mseeger]

Hello, the reason why Syne Tune does not (yet) support conditional spaces, is because this is hard to support generically with Bayesian optimization.

[mseeger]

If you only have a small conditional part, I recommend the "union" approach. Just add all parameters, even if the function does not depend on some of them, given values of others. This probably still works better than just doing random search. Your example is a bit incomplete, but you could add a uniform([-1, 1]) and a binary choice, with different names.

[mseeger]

What I mean:

   'depth': uniform(-1, 1),
   'depth_active': choice(["true", "false"]),

And if depth_active == "false", you just do not use the value of depth.

[mseeger]

For random search you get the same as if you encoded some complex conditional sampler for yourself (I'd not do that), and for BO, you may still outperform methods based on random search, even though the GP surrogate model does not model the conditional dependence of 'depth' on 'depth_active'.

[mseeger]

Of course, if you have a good idea how to support conditional relationships in a BO surrogate model, let us talk! There is some work, but this has no good general solution so far.