about running time about sequence-based-recommendations

[seun-wun]

Hello, the same issue occur to me. I have trained the model for a very long time, but the sps is still under 3%, much lower than the performance in your paper. Do you know how to fix this?

[rdevooght]

Hi,

It clearly shouldn't take that long, on model can be trained on movielens in an hour or so.
Can you tell me exactly what command line you used to launch your experiment ?

[xyanggu]

Hi,
I use "python train.py -d dataset/ml-1m/ -m RNN"
is that right?
thank you very much!

[rdevooght]

Hi,

With that command, the training process should compute validation cost every 2, 4, 8, 16, ... iterations (that's the default). The output should look something like that:

Compiling train...
Compilation done.
Compiling test...
Compilation done.
Opening file (1)
Opening file (1)
RNN with categorical cross entropy 2 batchs,  0.000198412698413  epochs in 1.20136499405 s
Last train cost :  8.21803
recall :  0.00255408308809
sps :  0.0
Best  sps :  0.0
blockbuster_share :  0.0737704918033
item_coverage :  118
ndcg :  0.0235363130049
user_coverage :  0.19
-----------------
2 0.000198412698413 1.20148301125 8.21803 0.00255408308809 0.0 0.0737704918033 118 0.0235363130049 0.19
Save model in ../../data/ml-1M/models/rnn_cce_db0.0_r0.0_ml30_bs16_ne0.0_GRU_gc100_h50_Ua_lr0.001_b10.9_b20.999_nt1_nf
Opening file (1)
RNN with categorical cross entropy 4 batchs,  0.000595238095238  epochs in 2.66510891914 s
Last train cost :  8.21811
recall :  0.00275834442008
sps :  0.0
Best  sps :  0.0
blockbuster_share :  0.0902255639098
item_coverage :  124
ndcg :  0.0256614255626
user_coverage :  0.202
-----------------
4 0.000595238095238 2.66527700424 8.21811 0.00275834442008 0.0 0.0902255639098 124 0.0256614255626 0.202
Save model in ../../data/ml-1M/models/rnn_cce_db0.0_r0.0_ml30_bs16_ne0.001_GRU_gc100_h50_Ua_lr0.001_b10.9_b20.999_nt1_nf
Opening file (1)
RNN with categorical cross entropy 8 batchs,  0.00138888888889  epochs in 4.16728496552 s
Last train cost :  8.21899
recall :  0.0040551560429
sps :  0.0
Best  sps :  0.0
blockbuster_share :  0.11875
item_coverage :  114
ndcg :  0.0330393612388
user_coverage :  0.236
-----------------
8 0.00138888888889 4.16740894318 8.21899 0.0040551560429 0.0 0.11875 114 0.0330393612388 0.236
Save model in ../../data/ml-1M/models/rnn_cce_db0.0_r0.0_ml30_bs16_ne0.001_GRU_gc100_h50_Ua_lr0.001_b10.9_b20.999_nt1_nf
Opening file (1)
RNN with categorical cross entropy 16 batchs,  0.00297619047619  epochs in 5.75333189964 s
Last train cost :  8.21685
recall :  0.00775130587827
sps :  0.002
Best  sps :  0.002
blockbuster_share :  0.212598425197
item_coverage :  98
ndcg :  0.0515730897683
user_coverage :  0.33

What output do you get exactly ?
Are you sure that you correctly specified the path to the dataset ? Did you use prepare.py properly to format the data ?

[xyanggu]

Hi,
yes, I get the same output as you said.

Opening file (7)
Opening file (8)
Opening file (9)
Opening file (10)
Opening file (11)
Opening file (12)
Opening file (1)
RNN with categorical cross entropy 65536 batchs,  11.215984939243539  epochs in 501.82608795166016 s
Last train cost :  377.138
recall :  0.010586256070690052
user_coverage :  0.4387755102040816
blockbuster_share :  0.1780821917808219
item_coverage :  39
sps :  0.01020408163265306
Best  sps :  0.0510204081633
ndcg :  0.0660293589142
-----------------
65536 11.215984939243539 501.82629132270813 377.138 0.010586256070690052 0.4387755102040816 0.1780821917808219 39 0.01020408163265306 0.0660293589142
Opening file (13)
Opening file (14)
Opening file (15)
Opening file (16)
Opening file (17)
Opening file (18)
Opening file (19)
Opening file (19)
Opening file (20)
Opening file (21)
Opening file (22)
Opening file (23)
Opening file (1)
RNN with categorical cross entropy 131072 batchs,  22.43214102344686  epochs in 999.8380262851715 s
Last train cost :  381.021
recall :  0.015686721566399123
user_coverage :  0.5204081632653061
blockbuster_share :  0.2
item_coverage :  42
sps :  0.02040816326530612
Best  sps :  0.0510204081633
ndcg :  0.077068678167
-----------------
131072 22.43214102344686 999.8382160663605 381.021 0.015686721566399123 0.5204081632653061 0.2 42 0.02040816326530612 0.077068678167
Opening file (25)
Opening file (26)
Opening file (27)
Opening file (28)
Opening file (29)
Opening file (30)
Opening file (31)
Opening file (32)
Opening file (33)
Opening file (34)
Opening file (35)
Opening file (36)
Opening file (37)
Opening file (38)
Opening file (39)
Opening file (40)
Opening file (41)
Opening file (42)
Opening file (43)
Opening file (44)
Opening file (45)
Opening file (1)
RNN with categorical cross entropy 262144 batchs,  44.8644531918535  epochs in 1979.2232081890106 s
Last train cost :  388.986
recall :  0.013790187977074955
user_coverage :  0.5102040816326531
blockbuster_share :  0.2988505747126437
item_coverage :  45
sps :  0.02040816326530612
Best  sps :  0.0510204081633
ndcg :  0.0844002809776
-----------------
262144 44.8644531918535 1979.223396539688 388.986 0.013790187977074955 0.5102040816326531 0.2988505747126437 45 0.02040816326530612 0.0844002809776

But I don't know when the training is over?
Last time I ran it 3 days, the console showed 'Opening file (3500)'
I use "python train.py -d dataset/ml-1m/ -m RNN", it seems as if it never to an end.
I don't know what's wrong with it.

Much appreciate!

[rdevooght]

This is actually normal, with the default settings the training never stops.
Opening file (3500) means that RNN iterated 3500 times over the whole dataset (it has certainly largely overfitted by now), and it will continue to do so until you stop the process.

When you train a complex model by stochastic gradient descent you generally go through a phase where the quality of the model increases on the training set and on the validation set, but at some point the quality starts to decrease on the validation set even though it still increases on the training set, at that point you are overfitting the model, and you should stop the training.
Unfortunately it is hard to guess a priori how long you should train before overfitting, which is why we generally use a kind of early stopping strategy.

By default, the training process uses the option --save Best which means that from time to time it will evaluate the model on the validation set, and only keep the model that performs best on the validation set. That means that even if you train for way too long, you'll still keep a model that likely did not overfit.

So if you have time, you can simply let the training run for way too long then stop it (ctrl-C) when you're sure that it won't improve anymore. You can also use the options --max_time time_in_seconds or --max_iter nb_of_iterations to set a fixed time/nb of iterations after which the training will stop.
If you're not sure about what max_time to set, and you don't want to look at the log yourself, you can also use one of the early stopping options. For example, with --es_m WorstTimesX --es_x 2 --es_min_wait 5, the training will stop if the training lasted at least five epochs, and if the number of iterations since the last best score on the validation set is larger than twice the largest number of iterations that you previously waited before an improvement.

One last thing, you can choose how often the model is evaluated, which determines both how often you'll see the scores appearing in the terminal, and also how often the model is potentially saved. If you set it too short you'll spend more time evaluating the model than training it, but if it's too long you might miss the optimal model between underfitting and overfitting.
By default it is set to evaluate the model after 2, 4, 8, 16, etc. iterations, which is good to give you an idea of the beginning of the training process (that's what I use when I'm exploring new problems or methods), but it quickly leads to too long waiting time between evaluations. There are many options to set when to evaluate the model: look at --progress, --time_based_progress and --mpi in the doc (https://github.com/rdevooght/sequence-based-recommendations#trainpy).

I hope it helps !

[xyanggu]

Hi,
It helps me a lot ! O(∩_∩)O
Much appreciated!

[DebonairLi]

Hello, the same issue occur to me. I have trained the model for a very long time, but the sps is still under 3%, much lower than the performance in your paper. Do you know how to fix this?

Hi, so have you solved it?Maybe you should do it in GPU?The same problem with me...