Error in load_packages(packages) : task 1 failed - "could not find function "load_packages"" about simdesign

[philchalmers]

Thanks for opening this issue. I believe this was just a case of SimDesign not being attached early enough in the distribution of the objects/functions, which for some reason in my tests was not detected (I run an MPI cluster locally for testing, so perhaps that's why?). In any event, I've add SimDesign to the .packages argument in the foreach() calls, so hopefully that fixes the issue. Would you mind testing this on your end before closing?

[jgman86]

Works perfectly fine ! I still have to figure out some details regarding our Cluster and MPI, but as far as I can see, the bug is fixed and it works:

cl <- startMPIcluster()

Attaching package: ‘mvtnorm’

The following objects are masked from ‘package:SimDesign’:

    rmvnorm, rmvt


Attaching package: ‘Matrix’

The following objects are masked from ‘package:tidyr’:

    expand, pack, unpack



Design row: 1/8;   Started: Tue Aug  2 21:33:08 2022;   Total elapsed time: 0.00s 
CmdStan path set to: /home/jgoettma/R/x86_64-pc-linux-gnu-library/4.1/cmdstan-2.30.0

Thanks for this amazing package, I abandoned the nested foreach loops !

[jgman86]

Hey thank you very much !! I will test it asap and call back !

[jgman86]

I tested now on the Cluster (reinstalled from GitHub previously) got this msg:

> cl <- startMPIcluster()

Attaching package: ‘mvtnorm’

The following objects are masked from ‘package:SimDesign’:

    rmvnorm, rmvt


Attaching package: ‘Matrix’

The following objects are masked from ‘package:tidyr’:

    expand, pack, unpack

Error in load_packages(packages) : 
  task 1 failed - "could not find function "load_packages""
Calls: runSimulation -> %dopar% -> <Anonymous>
Execution halted
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[51400,1],0]
  Exit code:    1
--------------------------------------------------------------------------

seems the same Issue - is it necessary to attach simDesign itself in packages?

[philchalmers]

It isn't necessary for the package itself to be exported, but I do so so that the other data generation functions are within the correct scope (e.g., avoids the need for SimDesign::rHeadrick(), for instance). It also makes the package experience the same when using a single computer or MPI (at least in theory.....)

The odd thing about the error message you are seeing is that SimDesign is somehow getting out of the correct scope to the point that none of the internal functions are being located, and it's not clear to me why. I'll poke around some more to see what I can find.

[philchalmers]

I was able to reproduce the problem on my end and have sent a suitable patch. Mind testing once more when you have a chance?