GridSearchCV seems to work, but how does it work internally? about cleanlab

[cgnorthcutt]

@komodovaran Your loops are in the wrong order.

You should instead write code like this:

for parameters in GridSearch using cross validation:
    Do cleanlab training # uses cross validation to get out of sample predicted probabilities)

instead your implementation will do this

After already removing data to perform cleanlab cross validation:
    Find the best parameters for that subset of data left over

It might work fine if you're dataset is relatively large compared to model complexity, but if you want to implement this for harder classification problems, you'll want as much data as possible.

Checkout this example: https://github.com/cgnorthcutt/cleanlab/blob/master/examples/classifier_comparison.ipynb
In this example, you see looping through the classifiers is the outermost loop.

So, you're code should look something like

from sklearn.model_selection import ParameterGrid
grid = [{'kernel': ['linear']}, {'kernel': ['rbf'], 'gamma': [1, 10]}]
best_result = None
for params in list(ParameterGrid(grid)):
    Do stuff with LearningWithNoisyLabels(clf = RandomForestClassifier(**params))
    Keep track of best result

[cgnorthcutt]

closing as there were no further questions