asash / bert4rec_repro Goto Github PK

@asash First, thank you for your great job on the sequential recommendation.

I have noticed that in your article, the results of BERT4Rec on the Beauty dataset cannot be replicated. After investigation, I found that it may be due to inconsistent preprocessing methods on the dataset. I used the Beauty dataset processed by S3Rec (which can replicate the results of SASRec on Beauty) for the experiment. Finally, the experimental results of BERT4Rec can be reproduced.

Inspired by you, I am trying to improve SASRec. As you said in the paper, the main difference between BERT4Rec and SASRec lies in the training objectives or loss function. So I tried to improve the loss function of SASRec. Finally, SASRec using the improved loss function can surpass BERT4Rec on the ML-1M and Beauty datasets, and achieve similar results on the ML-20M and Steam datasets.

I provide code (by fork) to reproduce what I said above. Finally, I hope my findings can be helpful to you.

Thank you again for your great work!

I tried to experiment with different models and encountered the problem that GRU4Rec takes much longer to train than other NNs. So I checked the logs and saw warnings saying that GRU layers are not using GPU optimization.
WARNING:tensorflow:Layer gru will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU

As far as I understood, the model implementations for "ours" BERT4Rec and ALBERT4Rec tests on ML-1M use the model code located at recommenders/dnn_sequential_recommender/models. While the evaluation configs pass the intermediate_size parameter to models' constructors, it doesn't get propagated to the HF model config and remains default (3072 for BERT and 16384 for ALBERT). The configs also use the default head number, which is 2 for BERT4Rec (same as declared in the replicability paper) but 16 for ALBERT4Rec.