Comments (4)
I understand you want to run several separate processes simultaneously, each with different model setting. Look at the standard package parallel (it is in the default installation of R). If you used lapply
to launch these sequential processes, you can use similarly parLapply
(in Windows) or mclapply
(in other systems than Windows), and there are also other alternatives. See the help for these commands and the parallel package info in general, including its vignette.
You should take care of a couple of details:
- Your sampling runs are of very different lengths, and this means that they will end in very different times. If you launch more processes than cores (cpus) in your system, the default is to wait for the last one to end before launching new processes simultaneously in all cpus. Short process will wait till the longest one ends. So you should set off prescheduling (see the manual pages for parallel commands), or you may be running slower than in sequential processing (where the last process will take ~90% of running time, and all previous were run in 10% before that).
- You should think how to set random number seeds for each parallel process. Look at the documentation of the parallel package, in particular the vignette.
- If you launch several model runs in parallel, you may have trouble if you also run chains in parallel in each of these processes. Running parallel processes within parallel processes may be suppressed in your system, and you may have no gain. On the other hand, if parallel processing within parallel processes is allowed, you may run out of free processors and have a race situation and your system can stall, and runs can be much slower than in sequential runs of models.
- All other things of parallel processing hold. See the parallel vignette, and in particular take care that you have sufficient memory for handling several models in parallel, or you may stall.
from hmsc.
I do think that parallel processing of your models makes little sense, and sequential processing is a sensible strategy. You increase thin 10-fold in each model and this gives approximately similar 10-fold increase in running time of each model. So the relative running times are 1, 10, 100, 1000, 10000 etc, and all previous models took only 11% of the time needed to run your current model. This means that theoretical maximum saving in computing time is 11% given by the longest running model. Further, the purpose of this sequence of models is not to run them all, but find sufficient thinning. When you run models sequentially from faster to slower, you can analyse the last finished model while the next is running, and if the diagnostics indicate that thinning is sufficient, you can keep that model and terminate the running process (which would take nearly 10 times longer than all models so far). The purpose is not to run all these models, but stop as soon as you can. This makes little sense in parallel processing.
Parallel processing can make sense if you have alternative model structures (different fixed effects, random effects, etc.), and you really need to run all these models to compare them later. In that case you can use parallelization tools that are provided by your operating environment and launch several models in parallel. That is a service that is above and outside our Hmsc package. So read guides to parallel package.
from hmsc.
Thanks, Jari, for the directions. Actually, my system has a moderate setup. So, it was taking a colossal time. I asked about this because I want to remotely move the thinning in a supercomputer facility with multiple cores to appoint for parallel processing.
from hmsc.
Parallel processing can be performed at various levels in Hmsc
- You can launch several
sampleMcmc
models with different definitions. This must be done at the level of R shell and the starting point is to use functions in parallel package to launchsampleMcmc
calls. - You can run several chains in parallel in
sampleMcmc
using argumentnParallel
. Each chain must be run sequentially, but chains run in prallel. - You may have parallelized linear algebra libraries (BLAS) in your system. In supercomputer you probably have such optimized libraries, but you can also use them in smaller systems (and they may already be in use depending on your sysadmin – that may be you...). Most of the time in
sampleMcmc
is spent in matrix algebra, and these parallelized libraries can give considerable speed-up even when you nothing about them or their existence.
from hmsc.
Related Issues (20)
- Hmsc Error: 8 nodes produced errors; first error: the leading minor of order 4 is not positive definite HOT 9
- When to use expected vs realized predictions
- sampleMcmc Error in checkForRemoteErrors(val) : one node produced an error: NA/NaN/Inf in foreign function call HOT 1
- HSMC usage to infer microbial communities? [discussion] HOT 1
- Spatial random variable with 9,738 coordinates causes R to crash HOT 2
- Interpretation of model coefficients in a multivariate poisson GLM with spatial random effect
- incorrect number of dimensions HOT 3
- Can not predict at the same coordinates used to train the model
- Missing help for `importPosteriorFromHPC` function
- Error in cross validation: missing value where TRUE/FALSE needed
- predict with Yc instead of constructGradient to avoid "Error: vector memory exhausted (limit reached?)" ?
- Interpretation of `predictEtaMean` / `predictEtaMeanField` arguments of the predict function
- In cor(lbeta[[i]][k, ], lmu[[i]][k, ]) : the standard deviation is zero HOT 2
- Unexpected trace plots for alpha parameters of a GPP model HOT 4
- Error in `importPosteriorFromHPC` for GPP/Hmsc-hpc models with `alignPost = TRUE` HOT 1
- Spatial Model running extremely slow HOT 6
- Error while converting Hmsc model object to JSON: `Error in rcpp_to_json(x, unbox, digits, numeric_dates, factors_as_string, : negative length vectors are not allowed` HOT 3
- im getting this error in running the Uhlig code
- Question about making predictions when using a hurdle approach
- Inconsistency in spatial model variance partitioning
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hmsc.