Error runs about omlbots

[DanielKuehn87]

Hm, annoying.
But I feel like we still can handle it somehow.

1. Runs fail because of ressource restrictions: As long our predictions for runtime/memory are accurate, this shouldn't be an issue.
2. Runs fail due to invalid hyperparameter settings: We won't be able to learn this, but I think in most cases these restrictions are known upfront, so we can set the param.set accordingly.

Any other cases where the failed runs might be usefull?

[PhilippPro]

Ranger is still making a lot of problems/error runs we have to fix this.

[berndbischl]

failed runs SHOULD be uploaded if the failure is nontrivial and potentially interesting for other persons.

so dont upload: (eg)

• when you apply regr to classif
• when he learner does not have the capability to handle the task (i think)
• when your resource setting are wrong on the cluster

upload

• when the learner numerically breaks

but feel free to discuss here, i am a bit unsure

[PhilippPro]

I think it is even not possible to upload failed runs to OpenML, is it? (see openml/openml-r#331)

At the moment I am a bit unhappy with the OpenML solution, because of the accessibility of the results, maybe we should upload the results also to a local database directly.

[berndbischl]

I think it is even not possible to upload failed runs to OpenML, is it? (see openml/openml-r#331)

then talk to @giuseppec. for WEKA they have been uploading failed runs for years.

At the moment I am a bit unhappy with the OpenML solution, because of the accessibility of the results, maybe we should upload the results also to a local database directly.

i really would not do this, ONLY if ABSOLUTELY needed.
how about you leave the results on disk in the cache for now? and write a simple collector?