How to tune hyperparameter when use the lambdamart example about unbiased_lambdamart

[acbull]

There are mainly two hyper-parameters that can be tuned, which is all in the src/objective/rank_objective.hpp files. They are:

  double _eta               : this denotes how much regularization is posed to the position bias.
  size_t _position_bins     : this denotes the maximum positions taken into account.

If you want to re-implement the result of our table, you can change the regularization term back to 0, by modifying

double _eta = 1.0 / (1 + 0.5); 

in the src/objective/rank_objective.hpp (Line 418) to

double _eta = 1.0 / (1 + 0); 

Sorry I haven't implemented these hyperparameter tuning into the config file. Will consider adding them later.

For the second question, actually in our algorithm, estimating position bias cost relatively minor time compared to training lgb. Also the calculation of position bias will not influence the parallelization of lightgbm (you can check the code). Thus, I don't think there is a big difference of directly using our code on the large dataset with first estimating position bias then tuning lgb. Actually, our experiment is conducted on a relatively big dataset (12G) and the efficiency is equal to the original lightgbm version without debiasing.

[hbghhy]

THX.