More frequent progress bar updates about biocparallel

[lgeistlinger]

A more fine-grained progress bar that updates whenever FUN finishes execution on an element of the vector X one is applying over with eg bplapply would indeed be a nice feature, which I hereby request. This is especially true for a rather time-consuming FUN such as a DE analysis or a GSEA analysis.

[frederikziebell]

Is there any update on this issue? An update of the progress bar whenever an element of X is finished makes the bar really informative. As it is now, the default tasks=0 setting splits X in as many chunks as there are workers and usually all workers finish simultaneously, making the progressbar essentially useless:

f <- function(i){
  Sys.sleep(1)
}

# progress bar almost directly 0% -> 100%
res <- bplapply(1:20, f, BPPARAM = MulticoreParam(workers=4, progress=T))

Of course an option is to set tasks=length(X) to get more frequent updates, but it comes at the cost of increased data transfer to the workers.

[mtmorgan]

This is challenging to implement, especially across back ends, so remains a feature not yet implemented. tasks != 0 does cost more in terms of data transfer, but then presumably the individual tasks involve a lot of computation so perhaps the additional cost is not that relevant? Strategies for reducing data transfer are always advantageous...

[mtmorgan]

I resolved this (d29c40b) by allowing tasks to be a number larger than length(X) (the length of the first argument to bplapply()), and (b944679) having progressbar = TRUE in the constructor of MulticoreParam() or SnowParam() set tasks = .Machine$integer.max if tasks is not set by the user. Thus MulticoreParam(progressbar = TRUE) updates progress on each element of X (desired behavior of this issue), while MulticoreParam(tasks = 2, progressbar = TRUE) only updates on the completion of each task. The update on each element of X come at the cost of more frequent data transfer to workers (though not set-up, e.g., loading packages), as @frederikziebell notes.