WISH: Allow for mc.cores=0 in parallel::mclapply() and friends about wishlist-for-r

[ilarischeinin]

I agree the documentation and behavior is confusing.

When run with mc.preschedule=TRUE (default) and mc.cores=1L, mclapply() will actually not fork a separate process:

Line 117 in mclapply():

    if (cores < 2L) return(lapply(X = X, FUN = FUN, ...))

i.e. same behavior you noted for pvec(). However, I guess technically this doesn't go against the documentation, since it says:

a integer giving the maximum allowed number of additional R processes allowed to be run in parallel to the current R process.

So, when mc.cores=1L, allowed additional processes are either 0 (as with mclapply(mc.preschedule=TRUE) or 1 (as with mclapply(mc.preschedule=FALSE).

But like I said, that is confusing and also inconsistent. It makes no sense to fork one additional process and have the main process simply wait.

The idea of the main process simply waiting I guess is the rational behind using mc.cores=detectCores() instead of - 1L. The point isn't so much to define how many simultaneous processes are allowed, but how many processes are allowed to compute at the same time. (Hence the two different behaviors of mclapply(mc.cores=1L) of one or two simultaneous processes, but only one actually doing anything.)

[HenrikBengtsson]

Thanks for the comments @ilarischeinin and for spotting the mcapply(mc.preschedule=TRUE) case; I've updated my top comment accordingly. To me this indicates that the developer(s) had similar ideas but at some point we ended up with a requirement of mc.cores >= 1.

I've also fixed a typo in my code suggestion: It should be if (mc.cores == 0 || (isChild() && !isTRUE(mc.allow.recursive))) and nothing else.

About the number of active versus computing processes: Yes, it's true that the main process shouldn't consume that much of the CPU since it is "just" spawning and polling for results. Having said this, I can imagine that there are obnoxiously strict compute clusters that would kick you out if you ran one process more than you requested (regardless of it's CPU usage). Moreover, this is holds for mclapply(). If you use for instance mcparallel(), there is nothing preventing you from running for computations in the main process as well. This can certainly becomes a reality with futures, where a main loop spawns of background processes and collects and post-processes results as they come back in while still polling for results of non-finished processes. This is why I think it is very important that the definition of mc.cores is very transparent and clear.