Comments (4)
BatchJobsParam is mustly just telling BiocParallel to use the BatchJobs configuration, which is specified in the config file. I believe bpworkers doesn't return anything meaningful for BatchJobsParam because in the "normal" use case of submitting jobs to a cluster across multiple machines and shared by multiple users, BatchJobs doesn't control the level of parallelism, the cluster's job scheduler does.
from biocparallel.
Hi,
Both of you are right, the bpworkers
setting has no purpose for BatchJobs. I'm unsure about consistency, I guess I would propose NA_integer_
as return value.
Was it intended that the user would set most parameters through the config file and not the BatchJobsParam() constructor interface?
Actually, no. We opted for configuration files to allow sysadmins to set global, system-wide options. You can always overrule these settings in your own config file or from R via setConfig()
. Furthermore, configuration settings depend primarily on the system, and not on the user -- therefore, to keep the R scripts portable, configuration options should not be set in script files of the user. Ideally you should not have to touch most options once you've figured out a working configuration for your system.
Maybe we were careless in the past and some conf options should have been function arguments. These can still be exposed as arguments in BiocParallel. Do you have some specific options in mind?
from biocparallel.
Hi,
In general I was hoping to lower the overhead involved in setting up the config and template files. Thinking of the less-experienced user, it would be great if they could do this completely within R.
BatchJobsParam fields:
- I'll modify the show method for BatchJobsParam so 'workers' isn't shown (and not documented).
- getConfig() is not mentioned on ?BatchJobsParam so I think it's difficult to know what config options are available. I'm not sure if the best approach is to (a) expose all of these as args or (b) document that they can be set with setConfig() or passed to BatchJobsParam() in the '...'. I'm leaning towards (b).
config and template files:
Interactive, Local and Multicore only need config file info (not template) so we're set for these.
I believe LSF, SLURM, Torque, SGE and SSH need both a template and config. Template files aren't included in BatchJobs and must be downloaded from github. For these cluster types the 'cluster.functions' arg only gets us part way there. To enable a complete R approach maybe we need a function that generates cluster-specific template files? Does something like this exist?
from biocparallel.
Closing this issue.
from biocparallel.
Related Issues (20)
- Error in serialize(data, node$con, xdr = FALSE) : ignoring SIGPIPE signal HOT 5
- Performance (speed) degradation in MulticoreParam with default force.G = TRUE HOT 3
- Nested parallellization question HOT 6
- stop.on.error = FALSE for DoParam doesn't work as expected HOT 2
- Troubles with bplapply within function (using SnowParam on Windows) HOT 6
- MulticoreParam bplapply unable to restart upon interrupt HOT 1
- Increase depth of traceback beyond `tryCatch()` for bp* functions - possible enhancement HOT 1
- The running time isn't reduced when using bplapply()? HOT 3
- BiocParallel errors HOT 4
- Handle worker abort better HOT 1
- move Rmpi to Enhances: HOT 1
- "foreach" %in% loadedNamespaces() instead of "package:foreach" %in% search()? HOT 1
- BiocParallel errors:could not find function ".OLD_read_block" HOT 11
- I meet a error when I use BiocParallel HOT 6
- BiocParallel : long vectors are not supported in .C() HOT 5
- BiocParallel for parallelization in BEER: Error and GPU Compatibility HOT 8
- BatchtoolsParam fails to propagate errors in bpiterate HOT 2
- Extremely minor: typo in docs? man/MulticoreParam-class.Rd HOT 1
- BiocParallel socketConnection error HOT 5
- strategy of tasks in MulticoreParam HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from biocparallel.