Comments (12)
Too high. It would be much clearer if just doing @everywhere using Module
worked. Rather than automatically doing code loading from the head node, how to load code should be configurable and you can say @workers command_that_tells_workers_to_load_code_from_node_1()
. Otherwise they would just load code locally like a normal Julia process does by default.
from distributed.jl.
This seems a bit too magical.
from distributed.jl.
At least the worker should not terminate abruptly, it should give a clean error message.
The module PTools was not even required on the workers. The closure had some references - not obvious looking at the code - which caused the problem. It took me a while to figure it out.
The contra argument for auto-loading modules, of course, is that typically the visualization packages are only required on process id 1.
from distributed.jl.
It should certainly not crash.
from distributed.jl.
How about an additional keyword argument to addprocs
called pkgsync
addprocs(n, pkgsync=true) # The default. All packages currently loaded on 1 are loaded on the worker
addprocs(n, pkgsync=false) # Nothing is loaded on the worker
addprocs(n, pkgsync=[pkgs...]) # Only specified packages are loaded
The packages loaded are the ones listed in Base.package_list
I assume. We strip the path and filename extension and try to load these on the worker.
If you loaded scripts using include
or packages not on the standard search path, they will not be sync'ed, you will need to do so explicitly on the new workers.
from distributed.jl.
Julia users link regarding the same issue - https://groups.google.com/d/msg/julia-users/-Y1rc8gkrgo/r6w5f144BkMJ
from distributed.jl.
Perhaps
using PTools
addprocs(1)
using PTools # currently this has no effect on the worker
should cause PTools
to be loaded on the worker?
(Discovered while testing JuliaLang/julia#8745 with the Images test, CC @vtjnash.)
from distributed.jl.
To fix the Images test, I added an @ everywhere before the import statement. With JuliaLang/julia#8745 changes to require, syncing modules during addprocs might be easier now.
from distributed.jl.
I got a module reload when I fixed it that way; I just pushed a different fix (to Images master, not tagged). But the fix uses a PR against JuliaLang/julia#8745 I'll be submitting shortly.
from distributed.jl.
@amitmurthy I think the level of magic in using
/require
here is either too low or too high.
- Too low?
Ifusing
automatically loads modules on workers then either modules loaded beforeaddprocs
should propagate orusing
, as in the example in https://github.com/JuliaLang/Distributed.jl/issues/17`, should load the module on the workers. Right now, you'll have to@everywhere using Module
if the module is already loaded on master but the@everywhere
solution ends up in a mess if the module hasn't already been loaded on master. - Too high?
If we removed all therequire
magic where the code is also executed on workers then you could just always use@everywhere using Module
. It would be quite simple to reason about.
from distributed.jl.
Related - #20
from distributed.jl.
Status of this?
from distributed.jl.
Related Issues (20)
- Distributed.jl - possibility to use other Serialization libraries? HOT 6
- MKL_NUM_THREADS
- Uncaught failure or noisy test in julia CI HOT 1
- Spurious `@spawnat` parallelism with single worker, single thread HOT 2
- SIGTERM test leaks stderr interrupt trace HOT 3
- improve pmap code for arrays, with type/shape of result the same as map (with PR)
- RFC: "for-loop" compliant @parallel for.... take 3 (with PR)
- Dynamic @distributed scheduling HOT 3
- Use a custom hashing function for remotecall_fetch to mitigate #48 (with PR).
- add RemoteLogger for distributed logging (with PR code)
- do not lock up addprocs on worker setup errors (with PR)
- RFC: Make addprocs() safe for reuse (i.e. doesn't request more processes if re-called) (with PR)
- Underministic behavior of `addprocs()` of `SSHManager` HOT 1
- Allow @everywhere include(...) to override default path behavior
- Distributed worker manager doesn't use socket connection to infer worker ip HOT 2
- Add a wait(::[Abstract]WorkerPool) (with PR code)
- `isready(::AbstractWorkerPool)` is inconsistent with whether `take!` will block
- Can we have `bind(::RemoteChannel, ::Process)`? HOT 1
- [Distributed.jl] inconsistent serialization of closures over global vars HOT 2
- broken error handling in message_handler_loop
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from distributed.jl.