Prevent runs of same configurations on same dataset about omlbots

[PhilippPro]

This is not inherently a mistake, as 1 10 foldCV is maybe not enough and with 10 repetitions we get a much more exact result. The general issue is rather, that for algorithms with many parameters (like xgboost) we should do much more runs than for kknn. Where our surrogate models are already quite exact.

[jakob-r]

But isn't the train/test-split not always the same for one openML-Task?

[PhilippPro]

That seems to be true, I did not know that. 👎

[jakob-r]

Well it' actually 👍 for reproduciblity but 👎 for your purposes 😉

[DanielKuehn87]

We are currently not using knn anymore because of this.
I feel like this should be done by the OpenML Website by rejecting identical uploads.
If we would want to control it with the bot, we would need to download the rather large database on every machine we run the bot on.

[berndbischl]

The general issue is rather, that for algorithms with many parameters (like xgboost) we should do much more runs than for kknn. Where our surrogate models are already quite exact.

My 2cents:

1. Why dont you REALLY upweight model with many param in sampling? I mean the bot was specifically constructed in a way that that can be done?

2. For stuff like knn @jakob-r is kinda right. It is just very annoying to check whether the epxeriment already exist.
   Problems:
   a) OML will not do that for you, I am pretty sure. If you want that, I think you have to do that yourself, for now.
   b) 2nd problem: That makes the random sampling less nice. But cant you easily check from the DB whether the exact experiment already exists? Or even from the OML server?