szilard / gbm-tune Goto Github PK

If you have enough RAM, it is always better to run parallel trainings with a low amount of training threads instead of parallelizing each training.

Example on a 20c/40t server, approximate theoretical time:

With hyperthreading, timings decrease by about 30% (in theory - in reality it is about 15-25% due to overhead). Parallel versions only have the overhead of merging results together (and copying data, if not forking), which is nearly non-existent (use a parallel lapply and not a parallel for to remove most of the overhead).

Also, it will allow to skip the negative efficiency issue you may have.

as usual great code and ideas
but
d0_train <- fread(paste0("/var/data/airline/",yr-1,".csv"))
may you share link from where to get data please

Important question :
may you clarify if you using some special algorithms for generation many test data / train data splits

to get big difference between splits
since some splits many have negligible difference for example for

samples 1 2 3 4 5 6 7 8 9 10 11 12
good split
test data 1 2 3 4 5 6 train data 7 8 9 10 11 12
test data 1 2 3 7 8 9 train data 4 5 6 10 11 12
minimum difference between all sets is 3

bad split
test data 1 2 3 4 5 6 train data 7 8 9 10 11 12
test data 1 2 3 4 5 7 train data 6 8 9 10 11 12
minimum difference between all sets 1