[Feature Proposal] Bayesian Optimisation Implementation about gama

[PGijsbers]

The initial pull request will concentrate on migrating to ConfigSpace and removing the param check as ConfigSpace would allow so. The second PR will implement the BO's search algorithm based on SMAC.

Sounds good to me! We may need to have a look at how to exactly integrate the BO PR, I think something like optional dependencies might probably be best since I understand SMAC is not always easy to install across all systems (mostly referring to Windows here). That way users can make of BO search if they have the requisite dependencies installed, but otherwise can still use GAMA without. Otherwise I think we should be good to go.

during the summer I will...

Just a heads up that I will be out of office from the end of July through the end of August. I may or may not choose to work on (some of) my open source projects, but I will not make any commitment for that time period. Otherwise I will be happy to have a look once I get back in office.

[PGijsbers]

Just dropping a message to let you know I am currently working night and day on the automl benchmark because we've got a deadline coming up. It might take a little longer than usual for me to respond, but I think I should be able to do so sometime next week at the latest.

[PGijsbers]

Would also appreciate it if you remove/mark text which is no longer relevant (when things change) 👍

[PGijsbers]

Oh and as far as PRs go, you could even split removing the "check_params" logic into a separate PR. Smaller is better!

[simonprovost]

Sounds amazing! Have a wonderful annual leave, Pieter, and if not before, I will see you in September 👌

I will manage the PRs properly, do not worry :)

Cheers!

[simonprovost]

EDIT 1.0: I will provide a more detailed update later; for now, it is time to take the win for today. I am not entirely certain that everything is functioning properly, but I am currently, at the time of writing this down, able to perform an 8-core multi-processed SMAC BO run with the GAMA async component, and it appears to function quite well.

There is a lot much more work to be done, and on various aspects, but I am now confident that a POC is feasible; within a week, I will be able to elaborate whether I have a fully workable POC. Therefore, if this POC seems to be promising, over the summer we would have a fully generalised BO's search algorithm for the GAMA's community.

In the interim, it would be preferable to await , if you planned answering, the more in-depth update later in the day-or-so; In a nutshell, I have taken a slightly different route from time to time than the one(s) proposed above, but the overall plan remains kind of the same anyway 💪

Cheers,

[simonprovost]

Just dropping a message to let you know I am currently working night and day on the automl benchmark because we've got a deadline coming up. It might take a little longer than usual for me to respond, but I think I should be able to do so sometime next week at the latest.

Indeed ! Thank you for informing me and I wish you and your team the best of luck with this deadline. I will keep this thread as current as feasible till then so that you have all the tools-in-hands to (dis-)approve / guide.

Cheers Pieter 👌

[simonprovost]

As it has been a couple of days since I first released this proposal, and due to the fact that as you may be aware, I worked things on and updated the mentioned path proposed a little, I advise against responding to this issue until I have provided an update. I'd recommend you guys wait for my go-update signal so I do not squander your valuable time. If we could have paused a Github issue, I would have done so.

Cheers for your understanding,

[simonprovost]

Re @PGijsbers,

It is now my turn to meet a deadline for my first year of Ph.D! The upcoming days, if not weeks, will be quite hectic for me. Should we 'close' this issue for the time being so that none of your Kanban issues are affected and I reopen everything as soon as I can?

In the meantime, I have a workable BO search optimiser operating under GAMA's design, so things are looking positive. Nonetheless, I will pose a few queries/dilemmas as soon as possible.

Cheers.

[PGijsbers]

No, it's fine to leave it open. I'll close it when I answer it (early next week). When you get back to it, feel free to re-open if you have follow-up questions.

[PGijsbers]

Sorry for the delayed reply. Your proposal seems very reasonable.

1. I do not object to the search algorithm itself choosing a different method of parallelization (e.g., using SMAC's). This can be advantageous if the underlying software can make better use of the parallelization (there are some BO methods designed with parallelization in mind, I don't know whether SMAC supports those). The important thing is that the provided constraints are observed. The default AsyncEvaluator is mainly there to provide an easy out-of-the-box solution.

2. Param checks are temporarily disabled and I am not 100% sure if I will turn them on again. The current implementation adds complexity with very little benefit: the failures it prevents are failures that are super quick. Feel free to aspect that part of the search space definition, SMAC/BO should pick up on the failed configurations anyway.

3. I have been wanting to have a ConfigSpace definition of the search space instead of my custom format. I would be very happy with a PR which converts the current search spaces to ConfigSpace ones so long as the current algorithms remain functional (probably mostly requires updates to how individuals are generated). You then wouldn't have to do the conversion specific for BO. Should you pursue this route, please set up a PR independently of any changes for BO. i.e., set up a PR that only changes the search space definitions/parsing/individual generation. The changes are probably substantial enough on their own, which makes reviewing hard enough already without adding more features :)

Cheers

[simonprovost]

Hello @PGijsbers,

I hope this finds you well. I hope the paper submission with your team was successful.

1. SMAC Parallelisation and GAMA Integration

I value your opinion regarding the possible benefits of utilising the parallelisation method from whichever Bayesian Optimisation (BO) library we choose. However, my current stance on this topic has changed slightly.

As you correctly noted, the default AsyncEvaluator provides a straightforward, out-of-the-box solution. I am an ardent supporter of this strategy. I believe this component is essential for fusing GAMA with any future implementation of a new search algorithm, e.g., in this current thread's case, BO.

Considering the long-term application and potential future enhancements of the AsyncEvaluator led to this conclusion. Any such enhancements would be advantageous for the implementation of (BO's)/the new search algorithm. Alternatively, if BO's/the new internal framework's multi-threading component were to be removed or malfunction, we would be forced to manually fork and modify the code or, in the worst case scenario, wait for the authors to fix the issue - which is far from ideal.

In contrast, with GAMA's multi-threading component (i.e., AsyncEvaluator), we have a clear understanding of where to investigate or place debugging statements if issues arise, while simultaneously reaping the benefits of any enhancements in the future.

In conclusion, I want to reassure you that my proof-of-concept (POC) implementation of this strategy, i.e. using SMAC's Ask&Tell Interface in conjunction with AsyncEvaluator, is operational and appears to be functioning flawlessly (both on OSX and Linux at the very least).

2. Param Check Implementation Complexity

I concur with your assessment of the current implementation's increased complexity and negligible benefits. The ability of the system to prevent typically rapid failures does not appear to outweigh the expense of maintaining such complexity. Regarding the aspect of the param check, I will share my thoughts in the following point (transition to configspace).

3. Transition to ConfigSpace

I strongly support your proposal to transition the search space from a custom format (current) to a ConfigSpace definition. I am happy to undertake this PR. Despite the fact that the PR will necessitate a few modifications to how individuals are generated, the overall process I reckon should remain largely unchanged, with the exception of the use of ConfigSpace instead of your current custom format.

Therefore, as you pointed out, this modification will allow me to eliminate the custom transformation from the BO's search algorithm, greatly simplifying matters.

In conclusion, I also concur that these changes should be split into multiple PRs for you and your team to handle them in a simpler manner. The initial pull request will concentrate on migrating to ConfigSpace and removing the param check as ConfigSpace would allow so. The second PR will implement the BO's search algorithm based on SMAC.

Could you double confirm this plan?

---

To conclude this lengthy response, I have a working POC of BO with SMAC integrated within GAMA and GAMA's multi-processing component as of today, also happily allowing me to complete my Ph.D. first year a few days ago. In addition, my supervisor and I have agreed that during the summer I will concentrate on improving the Python aspect of my work (i.e., among a tonnes of things, submitting the aforementioned two pull requests) and return to further research-oriented tasks in the fall.

Keeping this in mind, let us consider this issue to be a completed feature proposal. We appear to have reached a consensus on the underlying logic, so the next step will be to examine the practical implications, which I hope to begin soon 🙏

Thank you for your support with this Pieter ! I would appreciate it if you could confirm everything twice to make sure we are in-line 🫡

Wishing you a great week!!!