Slow Execution of MOBSTER Algorithm for Tabulated Benchmarks about syne-tune

[mseeger]

Hello,

I don't think your script has any issues.

The real-world runtime depends on two factors:

• Training time for trials
• Decision-making time of the HPO algorithms

The simulator backend on a tabulated benchmark eliminates the first factor, but the second factor remains. While simple methods like RandomSearch and ASHA have very cheap decision making (just random sampling), BayesianOptimization and MOBSTER need a substantial amount of time for decision making.

You can decrease this decision making time in various ways, as said in our docs. But in the end, you need to be careful, because what matters in the end are the results against real wall-clock time, where each trial has a runtime as in the tabulated benchmark.

The fact that simulations run faster for one method than another, has no real significance for comparing them. BO often does much better than RS, unless you have extremely cheap trials.

[mseeger]

Your second question: The real wall-clock time for an experiment is what is logged in the results, in the column of name ST_TUNER_TIME. That is why if you plot results, you get the correct results (as if trials really took the time in the table).

[mseeger]

BTW: I strongly encourage you to look at this tutorial:
https://syne-tune.readthedocs.io/en/latest/tutorials/benchmarking/README.html

Using this is much simpler than writing launcher scripts. At the moment, benchmarking is not available from pip, but only when installed from source, but we'll change this soon.

[piiiiiiie]

Thank you for your thorough explanations! I am studying it and will reach out again by opening a new issue if having other questions.