Comments (11)
Intel et al' parallel_for
etc. do not seem to assume cheap operations and in c++ at least force refactor of some code to allow parallelism. I wonder if there is a link between being completely automatic and specific. I see this lib and think wow imagine if it were just a drop in for iter()
or similar than think a bit and realise folk need to code algorithms really to take advantage.
I know this is not a direct correlation to the question, but I feel it is linked. In re-factoring for parallel many times code gets cleaner. It forces/allows the programmer to reason. This is my quandary at the moment to be honest. I love the idea of automating parallelism but have found in the past being told by the compiler I need to refactor to be handy.
In essence I feel an opt-in to be the way forward here, possibly later giving hints at optimisation of algorithms for parallelism at compile tim . Hope that helps a little anyway
from rayon.
@dirvine thanks for the input. A few thoughts:
- I'm not trying to make anything completely automatic -- you have to choose to write
par_iter
. But I do want to make it easy. - Rust's type system has the nice benefit of basically encouraging you (though not forcing you) to write parallel-safe from the get-go, so I do hope that only minimal refactoring is needed.
- Depending on your loop and what it does, though, you may still need to do some manual refactoring. The more that you are relying on the iterator adapters, vs writing ad-hoc code in your for loop body, the better off you are, basically (though some adapters, like
fold
, are not very parallelizable, but hopefully you can make do withreduce
).
from rayon.
Hmm. I have been experimenting with this in a branch. One interesting result that I found was that, when I ported the nbody demo, the par-reduce variant (which can generate quite a lot of inexpensive tasks...) ran ridiculously slow until I raised up the sequential threshold. This isn't really surprising I guess -- the defaults are very wrong for this case -- but it did point out of course the danger of changing our weights.
from rayon.
I guess if we did more work on making task spawning cheap (work that would be very profitable in any case) that might help out here. (For that matter, par-reduce is still always slower than the more coarse-grained version.)
from rayon.
The branch (for the record) is no-more-weight
.
from rayon.
See https://github.com/nikomatsakis/rayon/pull/81
from rayon.
Definitely significant progress here with @cuviper's https://github.com/nikomatsakis/rayon/pull/106. I still think we want to remove the existing weight stuff before 1.0 -- and maybe add back with some other APIs.
from rayon.
Maybe rayon could sample how long leaf nodes take to run and dynamically adjust? Of course some elements may require much more processing than others, but starting with fine grained splitting and dynamically increasing splits may get the best of both worlds.
from rayon.
@edre we already do this, effectively, via the mechanism of work stealing as well as the adaptive changes. What we are talking about is tuning that mechanism.
from rayon.
In particular I think the current mechanism should work pretty well except for when things are both highly variable between tasks and bigger tasks are clumped together.
from rayon.
Now that @cuviper added dynamic balancing, I think this is basically all done. Or done enough. Closing in favor of #111.
from rayon.
Related Issues (20)
- Error reporting in scoped tasks
- cooperative yield in ThreadPool::install() causes unexpected behavior in nested pools HOT 4
- rayon-core tests fail to build. HOT 6
- Matrix multiplication with Rayon doesn't see perf improvements HOT 3
- general purpose WASM support? HOT 1
- `fold` creates identity for each sublist HOT 10
- Support for some sort of collect()/extend() into `LinkedList<Vec<T>>` HOT 1
- A yield_now/yield_local alternative that allows the thread to go to sleep HOT 3
- Rayon slower than single threaded performance when mixing iter and par_iter HOT 9
- Potential to modify ordering for split_count and tour_counter HOT 2
- rayon 1.9.0 doctest failure - stack overflow on aarch64-unknown-linux-gnu HOT 2
- par_bridge_recursion test sometimes hanging indefinitely HOT 3
- rayon works very slow after MacOS Sonoma 14.4 has beed updated HOT 6
- `&mut [T]` should implement `rayon::slice::ParallelSliceMut` HOT 4
- Suggestion for stack size when calling par_sort_unstable HOT 5
- Tag missing for release 1.10.0 HOT 1
- Unable to parallelize properly using `par_iter` or `par_bridge` HOT 8
- Documentation for `ParallelIterator::fold` and/or `ParallelIterator::reduce` seam to be incorrect HOT 5
- Slowdown when using deeply nested vector HOT 6
- Potential `len` method ambiguity in `src/range.rs` HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rayon.