Comments (12)
Actually, I just noticed that @HenrikBengtsson appears to have already done so: https://github.com/HenrikBengtsson/BiocParallel.FutureParam. So maybe BiocParallel already supports batchtools through the future package.
from biocparallel.
@DarwinAwardWinner Nice! I hadn't seen that library previously. Rather new, but looks promising.
I still think it would be useful to integrate such functionality directly into BiocParallel rather than having an additional dependency.
from biocparallel.
A pull request would be great incentive for this, yes.
from biocparallel.
Based on the existence of future.batchtools, I suggest it might be better to write a backend for future and get batchtools support for free. The downside is that such a solution might not support the more advanced features of batchtools, so it depends on what batchtools features BiocParallel needs.
from biocparallel.
I don't have time to migrate this myself in the next 2-3 month, but I could export functions and complement missing helpers in batchtools
if someone wants to start working on a PR.
from biocparallel.
I'm willing to contribute the code of BiocParallel.FutureParam to the BiocParallel package. That way BiocParallel would support all backends that implement the Future API, including future.batchtools (and hence batchtools). All the end user would do is register(FutureParam)
and select what future backend to use via future::plan()
. One can of course come up with a register()
call that does both in one go, but to keep it simple, I haven't done so for now.
BTW, in lieu of FutureParam, a working solution to utilize (future.)batchtools in BiocParallel is to go via doFuture and DoParParam, e.g.
library("BiocParallel")
register(DoparParam(), default = TRUE)
doFuture::registerDoFuture()
library("future.batchtools")
plan(batchtools_local)
y <- bplapply(X = 1:3, FUN = function(x) list(x = x, pid = Sys.getpid()))
str(y)
## List of 3
## $ :List of 2
## ..$ x : int 1
## ..$ pid: int 26553
## $ :List of 2
## ..$ x : int 2
## ..$ pid: int 26554
## $ :List of 2
## ..$ x : int 3
## ..$ pid: int 26555
PS. This solution illustrates the goal of the future package; to bridge parallel frontends with parallel backends by providing a backend Future API that parallel backends implement and a frontend Future API that higher-level APIs implement.
from biocparallel.
I guess the question is, is there any special functionality provided by batchtools that BiocParallel needs/wants to access and which is not exposed through the Future API? If not, then I would consider this pretty much solved.
from biocparallel.
It seemed like the future api was also committing to a particular programming style (of unevaluated expressions)?
from biocparallel.
There's
f <- future({ expr })
and
f <- futureCall(FUN, args)
Is that what you're looking for, or did you mean something else? The other two key components of the frontend Future API are
resolved(f)
to check if future is resolved or not, and
v <- value(f)
to collect the value (or signal error if it occurred). The idea is that those are the building blocks for higher level needs. There's also resolved(fs)
and values(fs)
which operate on a set of futures, e.g. a list of futures.
Certain features such as stopping ("killing") futures are not part of the frontend Future API. This might be added in the future, cf. HenrikBengtsson/future#93 (long story).
UPDATE 2017-11-16: The civis package implements a CivisFuture for their services and added their own cancel()
method in lieu of an official one for the Future API. So, even if I haven't figured out exactly how this should look like in the Future API, it can certainly be added under some contrains/assumptions.
from biocparallel.
Coming back to this one. Not sure if my most recent reply helped or caused confusion.
From the end user's point of view, futures just provide another backend that the user can register for BiocParallel, i.e. register(FutureParam)
. Everything else will work the same. It should be 100% compatible with the BiocParallel API.
For the actual internals of the FutureParam backend, there are a few alternatives, but my existing working proposal shows how it can be done. There might be features in BiocParallel that I'm not aware of and that's missing, but the package tests against the existing BiocParallel API that I'm aware of.
Let me know if you want me to do a PR of what I've got.
from biocparallel.
Hi,
There is now a BatchtoolsParam-class
implemented with some basic functionality. It is on pull request #71 , which was just merged with master.
Please take a look.
Best,
Nitesh
from biocparallel.
Hi, the latest extension has been merged. Please take a look at #72.
@mfansler If you feel like this addresses your issue. Please close it.
from biocparallel.
Related Issues (20)
- Error in serialize(data, node$con, xdr = FALSE) : ignoring SIGPIPE signal HOT 5
- Performance (speed) degradation in MulticoreParam with default force.G = TRUE HOT 3
- Nested parallellization question HOT 6
- stop.on.error = FALSE for DoParam doesn't work as expected HOT 2
- Troubles with bplapply within function (using SnowParam on Windows) HOT 6
- MulticoreParam bplapply unable to restart upon interrupt HOT 1
- Increase depth of traceback beyond `tryCatch()` for bp* functions - possible enhancement HOT 1
- The running time isn't reduced when using bplapply()? HOT 3
- BiocParallel errors HOT 4
- Handle worker abort better HOT 1
- move Rmpi to Enhances: HOT 1
- "foreach" %in% loadedNamespaces() instead of "package:foreach" %in% search()? HOT 1
- BiocParallel errors:could not find function ".OLD_read_block" HOT 11
- I meet a error when I use BiocParallel HOT 6
- BiocParallel : long vectors are not supported in .C() HOT 5
- BiocParallel for parallelization in BEER: Error and GPU Compatibility HOT 8
- BatchtoolsParam fails to propagate errors in bpiterate HOT 2
- Extremely minor: typo in docs? man/MulticoreParam-class.Rd HOT 1
- BiocParallel socketConnection error HOT 5
- strategy of tasks in MulticoreParam HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from biocparallel.