This is an Unbiased Learning To Rank Algorithms (ULTRA) toolbox, which is still UNDER DEVELOPMENT.
Please cite the following paper when you use this toolbox:
Ai, Qingyao, Jiaxin Mao, Yiqun Liu, and W. Bruce Croft. "Unbiased learning to rank: Theory and practice." In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pp. 2305-2306. ACM, 2018.
Create virtual environment (optional):
pip install --user virtualenv
~/.local/bin/virtualenv -p python3 ./venv
source venv/bin/activate
Install ULTRA from the source:
git clone https://github.com/QingyaoAi/ULTRA.git
cd ULTRA
python setup.py install #use setup-gpu.py for GPU support
Run toy example:
cd example/toy
bash offline_exp_pipeline.sh
-
DLA: this is an implementation of the Dual Learning Algorithm in Unbiased Learning to Rank with Unbiased Propensity Estimation.
-
IPW: this model is an implementation of the Inverse Propensity Weighting algorithms in Learning to Rank with Selection Bias in Personal Search and Unbiased Learning-to-Rank with Biased Feedback
-
REM: this model is an implementation of the regression-based EM algorithm in Position bias estimation for unbiased learning to rank in personal search
-
PD: this model is an implementation of the pairwise debiasing algorithm in Unbiased LambdaMART: An Unbiased Pairwise Learning-to-Rank Algorithm.
-
PDGD: this model is an implementation of the Pairwise Differentiable Gradient Descent algorithm in Differentiable unbiased online learning to rank
-
DBGD: this model is an implementation of the Dual Bandit Gradient Descent algorithm in Interactively optimizing information retrieval systems as a dueling bandits problem
-
NA: this model is an implementation of the naive algorithm that directly train models with clicks.
Create click models for click simulations
python input_layer/click_models.py pbm 0.1 1 4 1.0 example/ClickModel
* The output is a json file containing the click mode that could be used for click simulation. More details could be found in the code.
(Optional) Estimate examination propensity with result randomization
cd utils
python propensity_estimator.py ../example/ClickModel/pbm_0.1_1.0_4_1.0.json <DATA_DIR> ../example/PropensityEstimator/
* The output is a json file containing the estimated examination propensity (used for IPW). More details could be found in the code.
If you use ULTRA in your research, please use the following BibTex entry.
@inproceedings{Ai:2018:ULR:3269206.3274274,
author = {Ai, Qingyao and Mao, Jiaxin and Liu, Yiqun and Croft, W. Bruce},
title = {Unbiased Learning to Rank: Theory and Practice},
booktitle = {Proceedings of the 27th ACM International Conference on Information and Knowledge Management},
series = {CIKM '18},
year = {2018},
isbn = {978-1-4503-6014-2},
location = {Torino, Italy},
pages = {2305--2306},
numpages = {2},
url = {http://doi.acm.org/10.1145/3269206.3274274},
doi = {10.1145/3269206.3274274},
acmid = {3274274},
publisher = {ACM},
address = {New York, NY, USA},
keywords = {click model, counterfactual learning, unbiased learning to rank, user bias},
}