SnowParam with MPI about biocparallel

[lgatto]

And I forgot the obvious. BiocParallel was installed using biocLite.

> sessionInfo()
R Under development (unstable) (2014-02-01 r64910)
Platform: x86_64-unknown-linux-gnu (64-bit)

locale:
 [1] LC_CTYPE=en_GB.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_GB.UTF-8        LC_COLLATE=en_GB.UTF-8    
 [5] LC_MONETARY=en_GB.UTF-8    LC_MESSAGES=en_GB.UTF-8   
 [7] LC_PAPER=en_GB.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] BiocParallel_0.5.8

loaded via a namespace (and not attached):
 [1] BatchJobs_1.1-1135 BBmisc_1.5         BiocGenerics_0.9.3 brew_1.0-6        
 [5] codetools_0.2-8    compiler_3.1.0     DBI_0.2-7          digest_0.6.4      
 [9] fail_1.2           foreach_1.4.1      iterators_1.0.6    parallel_3.1.0    
[13] plyr_1.8           RSQLite_0.11.4     sendmailR_1.1-2    tools_3.1.0 

[DarwinAwardWinner]

The problem is here: https://github.com/Bioconductor/BiocParallel/blob/master/R/SnowParam-class.R#L25

I think we should eliminate the type argument to .nullCluster and just always use type="PSOCK" when creating the null cluster. So the new .nullCluster function would look like:

.nullCluster <-
    function()
{
    makeCluster(0L, "PSOCK")
}

And of course all places that call .nullCluster should be modified not to pass any args.

[lgatto]

Thanks. Testing further with a default PSOCK null cluster, I get the following:

> library("BiocParallel")
> BiocParallel:::.nullCluster
function(type)
    makeCluster(0L, "PSOCK")
<environment: namespace:BiocParallel>
> p <- SnowParam(4L, type = "MPI")
> p
class: SnowParam; bpisup: FALSE; bpworkers: 4; catch.errors: TRUE
cluster 'spec': 4; 'type': MPI
> bplapply(1:10, sqrt, BPPARAM=p)
Loading required package: Rmpi
    4 slaves are spawned successfully. 0 failed.
Error in (function (classes, fdef, mtable)  : 
  unable to find an inherited method for function ‘bpbackend<-’ for signature ‘"SnowParam", "spawnedMPIcluster"’

[DarwinAwardWinner]

I don't think SnowParam was written with MPI clusters in mind. Looking that the bpbackend<- method, I see it is only defined for replacement values of class SOCKcluster:

https://github.com/Bioconductor/BiocParallel/blob/master/R/SnowParam-class.R#L97

setReplaceMethod("bpbackend", c("SnowParam", "SOCKcluster"),
    function(x, ..., value)
{
    x$cluster <- value
    x
})

Perhaps we should simply relax this restriction to class cluster, which I believe all Snow cluster objects have in their class vector.

[lgatto]

> I don't think SnowParam was written with MPI clusters in mind. 

I hadn't realised that. Would adding this require a lot of work, other than updating the bpbackend-SnowParam,... replace method?

[DarwinAwardWinner]

I'm not sure what else would be required. You can try adding the replacement method as described and see if ti just works.

[DarwinAwardWinner]

Alternatively, as a workaround for now, I would recomment that you use the foreach backend for BiocParallel, and then use the MPI backend for foreach's parallelization: http://cran.r-project.org/web/packages/doMPI/index.html

[lgatto]

Adding setOldClass(c("spawnedMPIcluster", "cluster")) and defining bpbackend<- for sig c("SnowParam", "cluster") looks promising.

> library("BiocParallel")
> showMethods("bpbackend<-")
Function: bpbackend<- (package BiocParallel)
x="missing", value="ANY"
x="SnowParam", value="cluster"
x="SnowParam", value="SOCKcluster"
    (inherited from: x="SnowParam", value="cluster")
x="SnowParam", value="spawnedMPIcluster"
    (inherited from: x="SnowParam", value="cluster")

> getClass("spawnedMPIcluster")
Virtual Class "spawnedMPIcluster" [package "BiocParallel"]

Slots:

Name:   .S3Class
Class: character

Extends: 
Class "cluster", directly
Class "oldClass", by class "cluster", distance 2
> (p <- SnowParam(2L, type = "MPI"))
class: SnowParam; bpisup: FALSE; bpworkers: 2; catch.errors: TRUE
cluster 'spec': 2; 'type': MPI
> x <- bplapply(1:10, sqrt, BPPARAM=p)
    2 slaves are spawned successfully. 0 failed.

I quickly check the default socket type and multicore with the above changes and it works as expected. I will send a pull request for you to review the changes.

Will get a go with the foreach/doMPI backend.

Thanks for your help!

[DarwinAwardWinner]

> x <- bplapply(1:10, sqrt, BPPARAM=p)
    2 slaves are spawned successfully. 0 failed.

I assume that the above code produced the correct result?

[lgatto]

> library("BiocParallel")
> (p <- SnowParam(2L, type = "MPI"))
class: SnowParam; bpisup: FALSE; bpworkers: 2; catch.errors: TRUE
cluster 'spec': 2; 'type': MPI
> (p2 <- SnowParam(2L, type = "PSOCK"))
class: SnowParam; bpisup: FALSE; bpworkers: 2; catch.errors: TRUE
cluster 'spec': 2; 'type': PSOCK
> x <- bplapply(1:10, sqrt, BPPARAM=p)
Loading required package: Rmpi
    2 slaves are spawned successfully. 0 failed.
x2 <- bplapply(1:10, sqrt, BPPARAM=p2)
x0 <- lapply(1:10, sqrt)
> all.equal(x0, x)
> [1] TRUE
> all.equal(x0, x2)
[1] TRUE

[mtmorgan]

Ryan is correct that the original intention was that SnowParam support the socket clusters, and that other Params be developed to support other back-ends. But I am changing my mind on this.

The role of .nullCluster was in part to retain state across calls to bpstart, so the same type of cluster would be available. I think this needs to be re-implemented in a more straight-forward way, with fields in SnowParam remembering (forced) arguments of the original constructor.

For these reasons I will not incorporate the pull request earlier in this thread, but will happily review a (not too much more) comprehensive pull request. I'll try to work on this during the next several days myself.

[mtmorgan]

I've added support for SnowParam type="MPI", thanks for the suggestion and please let me know if there are other issues.