Comments (9)
I'm taking a stab at this here: https://github.com/DarwinAwardWinner/BiocParallel/tree/bpexport
So far I've added stubs for all the params, and I've added a clusterExport
-based implementation for SnowParam. But thinking about it, that will only work if the cluster is running when clusterExport is called, so even that is not fully implemented.
from biocparallel.
One issue to consider is, what if we call bpexport
on a SerialParam or a MulticoreParam? They already have access to all the parent's variables, including any changes to those variables' values that occur after the call to bpexport
. Should we make an attempt to have these params match the behavior of e.g. SnowParam by storing a snapshot of the variables when bpexport
is called and then using that snapshot in place of the current value when the param is used?
Also, what should happen when you call bpexport on a stopped cluster? What should happen when you stop a cluster after exporting a variable?
from biocparallel.
I'd like to suggest creating a simple class/list storing objects exported via bpexport. As soon as bplapply/bpmapply is called the objects can then be put into the function's environment. Something like
exported = list(x = 12, y = rnorm(10))
mapply(assign, x = names(exported), values = exported, MoreArgs=list(envir = environment(FUN))
You would just have to check that environmentName(FUN) != "R_GlobalEnv"
and in this case just give the function a new environment with the GlobalEnv as parent.
from biocparallel.
I think it's probably a good idea to always give the function a new environment with the exported values and with the function's previous environment as parent. Are you suggesting this for the SerialParam and MulticoreParam classes?
from biocparallel.
Yes, Serial and Multicore. I also see no drawbacks for BatchJobs over its internal export mechanism. I don't know if this is applicable for DoPar. You could pass them to .export in foreach, but I was unable to find a way to turn the heuristic auto-export off.
On more thing to consider is the expected behavior if a variable is explicitly exported and also defined in the function's environment. Variables in the function's env have precedence in the lookup which deviates from the lookup using parallel/clusterExport (which assigns to GlobalEnv on the slaves).
from biocparallel.
Well, I think the goal would be in all cases to keep the behavior consistent across all param classes. So to answer what happens when you export a variable and the same variable is defined in the function's environment, we ask what happens naturally in the case of ShowParam where you use clusterExport
to implement bpexport
, and then make sure we do the same thing for the other params, right? I actually don't know how (or if) function environments get transferred between processes by snow and others.
from biocparallel.
Actually, to be honest, I'm probably not the best person to implement this, because the vast majority of the time I want to do parallel stuff in R, I use multicore, so I never have to worry about exporting variables and I have no real idea how to do it.
from biocparallel.
Thinking about it, we should probably take this same "just-in-time export" approach for SnowParam as well. This will solve the problem of the cluster not being running when bpexport is called.
from biocparallel.
Ok, I am finding myself using BatchJobsParam a lot and wanting export functionality, so I will try to work on this some time soon.
from biocparallel.
Related Issues (20)
- Error in serialize(data, node$con, xdr = FALSE) : ignoring SIGPIPE signal HOT 5
- Performance (speed) degradation in MulticoreParam with default force.G = TRUE HOT 3
- Nested parallellization question HOT 6
- stop.on.error = FALSE for DoParam doesn't work as expected HOT 2
- Troubles with bplapply within function (using SnowParam on Windows) HOT 6
- MulticoreParam bplapply unable to restart upon interrupt HOT 1
- Increase depth of traceback beyond `tryCatch()` for bp* functions - possible enhancement HOT 1
- The running time isn't reduced when using bplapply()? HOT 3
- BiocParallel errors HOT 4
- Handle worker abort better HOT 1
- move Rmpi to Enhances: HOT 1
- "foreach" %in% loadedNamespaces() instead of "package:foreach" %in% search()? HOT 1
- BiocParallel errors:could not find function ".OLD_read_block" HOT 11
- I meet a error when I use BiocParallel HOT 6
- BiocParallel : long vectors are not supported in .C() HOT 5
- BiocParallel for parallelization in BEER: Error and GPU Compatibility HOT 8
- BatchtoolsParam fails to propagate errors in bpiterate HOT 2
- Extremely minor: typo in docs? man/MulticoreParam-class.Rd HOT 1
- BiocParallel socketConnection error HOT 5
- strategy of tasks in MulticoreParam HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from biocparallel.