Allow for list-valued hyperparameters about syne-tune

[mseeger]

The idea would be that we allow for list-valued hyperparameters in a config space. Each component can be a Domain or a fixed value.

Internally, this would be flattened when it comes to searchers and encodings.

This can certainly be done, but we should warn folks that high-D HPO is often failing if no special structure is present and assumed by the surrogate model.

[mseeger]

One technical blocker is that command line arguments for SageMaker training jobs are limited in length. What is possible is to pass parameters to a script as JSON file config.json. We could maybe provide some tooling to load this file inside a training script.

This is not a simple change.

[mseeger]

https://github.com/awslabs/syne-tune/blob/main/benchmarking/nursery/tuning_with_hyperparameter_file

[mseeger]

Example of how to read config from JSON file. This works for the local backend only at the moment. I can try and support this for the SageMaker backend as well.

[mseeger]

#577 is adressing this issue to some extent. It allows to pass fixed lists to the training script, but all hyperparameters to be optimized still have to be elementary types.

[mseeger]

I am closing this, since #577 is addressing the request partly. Since HPO methods are not ready to jointly tune many hyperparameters, we hesitate to offer list-valued domains, as this may send the wrong signal.