Cant work HyperparameterTuner GridSearch on RandomForest about mlpack

[GLmontanari]

Oh, wow! That's a bug for sure. I wasn't sure where to put num_classes = 2 (which is my case) and I put it both in the constructor and in the call of HyperParameterTuner.Optimize() because the docs says to put all the parameters in the same order as in the constructor of the classifier, but it's clearly a mistake, because as you say, the number of classes is no hyperparameter in this case.
I commented it and now I get different results on my dataset. Thank you!!!

[rcurtin]

Sorry for the slow response on this---I was traveling. Are you able to try with the fixes in #3521? Alternately, can you provide a dataset that I can try your code with to see if those fixes work?

My theory is that the random forest you are training is affected by the same bug that #3521 solves---that bug basically means that when using OpenMP, the trees created by each thread might be identical (and thus perform exactly the same).

Another way you could test here is by disabling OpenMP during compilation.

Thanks for the report!

[conradsnicta]

@GLmontanari #3521 was merged. Let us know if this addresses the issue on your setup.

[GLmontanari]

Dear all,

I don't have OpenMP, and I compiled the library with that option disabled. Nevertheless I forked the last commits and tried again.
Now I have armadillo 12.6.2 and mlpack, well, the last version. Nothing changes.
I can share my data if you wish. How do you rather have it?

[rcurtin]

Thanks for trying anyway. I tried the code with the covertype dataset (7 classes, ~580k points, 55 dimensions). I found that there is an error in the hpt.Optimize() call:

	std::tie(best_forest) = hpt.Optimize(
		mlpack::Fixed(numClasses), 
		forest_sz, // forest_sz 
		mlpack::Fixed(1),  // min_leaf_sz
		mlpack::Fixed(1e-7),	   // gain_split
		mlpack::Fixed(0)); // max_depth

You don't need to specify the number of classes here, since that is not a hyperparameter. In fact, what that line actually does is performs hyperparameter optimization with the number of trees set to the number of classes, the minimum leaf size set to forest_sz, and so forth... the arguments are off by one. When I removed numClasses from that line, I had more success:

	std::tie(best_forest) = hpt.Optimize(
		forest_sz, // forest_sz 
		mlpack::Fixed(1),  // min_leaf_sz
		mlpack::Fixed(1e-7),	   // gain_split
		mlpack::Fixed(0)); // max_depth

I think that will fix your issue. Let me know if you still see strange behavior and I can keep digging.