Comments (12)
And I forgot the obvious. BiocParallel
was installed using biocLite
.
> sessionInfo()
R Under development (unstable) (2014-02-01 r64910)
Platform: x86_64-unknown-linux-gnu (64-bit)
locale:
[1] LC_CTYPE=en_GB.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_GB.UTF-8 LC_COLLATE=en_GB.UTF-8
[5] LC_MONETARY=en_GB.UTF-8 LC_MESSAGES=en_GB.UTF-8
[7] LC_PAPER=en_GB.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] BiocParallel_0.5.8
loaded via a namespace (and not attached):
[1] BatchJobs_1.1-1135 BBmisc_1.5 BiocGenerics_0.9.3 brew_1.0-6
[5] codetools_0.2-8 compiler_3.1.0 DBI_0.2-7 digest_0.6.4
[9] fail_1.2 foreach_1.4.1 iterators_1.0.6 parallel_3.1.0
[13] plyr_1.8 RSQLite_0.11.4 sendmailR_1.1-2 tools_3.1.0
from biocparallel.
The problem is here: https://github.com/Bioconductor/BiocParallel/blob/master/R/SnowParam-class.R#L25
I think we should eliminate the type
argument to .nullCluster
and just always use type="PSOCK"
when creating the null cluster. So the new .nullCluster
function would look like:
.nullCluster <-
function()
{
makeCluster(0L, "PSOCK")
}
And of course all places that call .nullCluster
should be modified not to pass any args.
from biocparallel.
Thanks. Testing further with a default PSOCK
null cluster, I get the following:
> library("BiocParallel")
> BiocParallel:::.nullCluster
function(type)
makeCluster(0L, "PSOCK")
<environment: namespace:BiocParallel>
> p <- SnowParam(4L, type = "MPI")
> p
class: SnowParam; bpisup: FALSE; bpworkers: 4; catch.errors: TRUE
cluster 'spec': 4; 'type': MPI
> bplapply(1:10, sqrt, BPPARAM=p)
Loading required package: Rmpi
4 slaves are spawned successfully. 0 failed.
Error in (function (classes, fdef, mtable) :
unable to find an inherited method for function ‘bpbackend<-’ for signature ‘"SnowParam", "spawnedMPIcluster"’
from biocparallel.
I don't think SnowParam was written with MPI clusters in mind. Looking that the bpbackend<-
method, I see it is only defined for replacement values of class SOCKcluster
:
https://github.com/Bioconductor/BiocParallel/blob/master/R/SnowParam-class.R#L97
setReplaceMethod("bpbackend", c("SnowParam", "SOCKcluster"),
function(x, ..., value)
{
x$cluster <- value
x
})
Perhaps we should simply relax this restriction to class cluster
, which I believe all Snow cluster objects have in their class vector.
from biocparallel.
> I don't think SnowParam was written with MPI clusters in mind.
I hadn't realised that. Would adding this require a lot of work, other than updating the bpbackend-SnowParam,...
replace method?
from biocparallel.
I'm not sure what else would be required. You can try adding the replacement method as described and see if ti just works.
from biocparallel.
Alternatively, as a workaround for now, I would recomment that you use the foreach backend for BiocParallel, and then use the MPI backend for foreach's parallelization: http://cran.r-project.org/web/packages/doMPI/index.html
from biocparallel.
Adding setOldClass(c("spawnedMPIcluster", "cluster"))
and defining bpbackend<-
for sig c("SnowParam", "cluster")
looks promising.
> library("BiocParallel")
> showMethods("bpbackend<-")
Function: bpbackend<- (package BiocParallel)
x="missing", value="ANY"
x="SnowParam", value="cluster"
x="SnowParam", value="SOCKcluster"
(inherited from: x="SnowParam", value="cluster")
x="SnowParam", value="spawnedMPIcluster"
(inherited from: x="SnowParam", value="cluster")
> getClass("spawnedMPIcluster")
Virtual Class "spawnedMPIcluster" [package "BiocParallel"]
Slots:
Name: .S3Class
Class: character
Extends:
Class "cluster", directly
Class "oldClass", by class "cluster", distance 2
> (p <- SnowParam(2L, type = "MPI"))
class: SnowParam; bpisup: FALSE; bpworkers: 2; catch.errors: TRUE
cluster 'spec': 2; 'type': MPI
> x <- bplapply(1:10, sqrt, BPPARAM=p)
2 slaves are spawned successfully. 0 failed.
I quickly check the default socket type and multicore with the above changes and it works as expected. I will send a pull request for you to review the changes.
Will get a go with the foreach
/doMPI
backend.
Thanks for your help!
from biocparallel.
> x <- bplapply(1:10, sqrt, BPPARAM=p) 2 slaves are spawned successfully. 0 failed.
I assume that the above code produced the correct result?
from biocparallel.
> library("BiocParallel")
> (p <- SnowParam(2L, type = "MPI"))
class: SnowParam; bpisup: FALSE; bpworkers: 2; catch.errors: TRUE
cluster 'spec': 2; 'type': MPI
> (p2 <- SnowParam(2L, type = "PSOCK"))
class: SnowParam; bpisup: FALSE; bpworkers: 2; catch.errors: TRUE
cluster 'spec': 2; 'type': PSOCK
> x <- bplapply(1:10, sqrt, BPPARAM=p)
Loading required package: Rmpi
2 slaves are spawned successfully. 0 failed.
x2 <- bplapply(1:10, sqrt, BPPARAM=p2)
x0 <- lapply(1:10, sqrt)
> all.equal(x0, x)
> [1] TRUE
> all.equal(x0, x2)
[1] TRUE
from biocparallel.
Ryan is correct that the original intention was that SnowParam support the socket clusters, and that other Params be developed to support other back-ends. But I am changing my mind on this.
The role of .nullCluster
was in part to retain state across calls to bpstart, so the same type of cluster would be available. I think this needs to be re-implemented in a more straight-forward way, with fields in SnowParam remembering (forced) arguments of the original constructor.
For these reasons I will not incorporate the pull request earlier in this thread, but will happily review a (not too much more) comprehensive pull request. I'll try to work on this during the next several days myself.
from biocparallel.
I've added support for SnowParam type="MPI", thanks for the suggestion and please let me know if there are other issues.
from biocparallel.
Related Issues (20)
- Error in serialize(data, node$con, xdr = FALSE) : ignoring SIGPIPE signal HOT 5
- Performance (speed) degradation in MulticoreParam with default force.G = TRUE HOT 3
- Nested parallellization question HOT 6
- stop.on.error = FALSE for DoParam doesn't work as expected HOT 2
- Troubles with bplapply within function (using SnowParam on Windows) HOT 6
- MulticoreParam bplapply unable to restart upon interrupt HOT 1
- Increase depth of traceback beyond `tryCatch()` for bp* functions - possible enhancement HOT 1
- The running time isn't reduced when using bplapply()? HOT 3
- BiocParallel errors HOT 4
- Handle worker abort better HOT 1
- move Rmpi to Enhances: HOT 1
- "foreach" %in% loadedNamespaces() instead of "package:foreach" %in% search()? HOT 1
- BiocParallel errors:could not find function ".OLD_read_block" HOT 11
- I meet a error when I use BiocParallel HOT 6
- BiocParallel : long vectors are not supported in .C() HOT 5
- BiocParallel for parallelization in BEER: Error and GPU Compatibility HOT 8
- BatchtoolsParam fails to propagate errors in bpiterate HOT 2
- Extremely minor: typo in docs? man/MulticoreParam-class.Rd HOT 1
- BiocParallel socketConnection error HOT 5
- strategy of tasks in MulticoreParam HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from biocparallel.