Comments (7)
Well, right now there isn't really a much better way (though you could handwrite a recursive function with join
), but I'd like to add a kind of "work queue" abstraction (basically similar to scoped threads) that would probably address this use case.
from rayon.
@grahame See also https://github.com/QuietMisdreavus/polyester, but I'm not sure that the cost of allocating String
s makes this worth it.
from rayon.
This seems like a fairly popular feature judging on the number of issues people have created that are slight variations on this general Iterator
to ParallelIterator
adapter idea. Has there been any progress on this?
My particular use case is that I've got an iterator which yields every permutation of 4 numbers from 18 to 200 (around 167,961,600,000,000 permutations), then does a bunch of filters and maps to find a bunch of "ideal" combinations. In this case it's not feasible to save the combinations into a Vec
, yet the filters and maps are all trivially parallelizable.
from rayon.
It's a popular request, but there's not an obvious way to implement it. We've had a few ideas, but not any progress to report for it, sadly.
My particular use case is that I've got an iterator which yields every permutation of 4 numbers from 18 to 200 (around 167,961,600,000,000 permutations),
Perhaps I misunderstand, but wouldn't the number of permutations be (200-18)4 -- roughly 109? If you can set up that generator with ranges, you can parallelize that.
(18..200).into_par_iter().for_each(|x| (180..200).into_par_iter().for_each(|y| ...))
In this case it's not feasible to save the combinations into a Vec,
Another strategy is batch this work. Generate a manageable amount into a Vec
, process those in parallel, and repeat in a loop until you've done them all.
from rayon.
I'm using Rayon to parallelise a fairly CPU heavy operation on a bunch of String
instances coming to me via the csv crate. I wound up here after I did exactly what @cuviper suggested; batching into a Vec
. That gave me a good performance gain, but not as much as should be possible – the csv crate has to do enough work that I lose a lot of time building the batches.
Would a generic solution be to allow Rayon to .par_map()
on the rx side of a channel, or have a templated function that makes that happen given the type of the channel? I could then stick the iterator bit in a thread, have it spit String
instances down the channel, and Rayon could do its thing. This would be even more general than supporting Read
or Iterator
as a provider.
(I'm fairly new to Rust, so apologies if I'm missing something and this is not actually easier than just supporting Iterator
.)
from rayon.
It looks like this is still open as #550 which added ParallelBridge
was considered only "half" of this issue. But does that really capture the requirement expressed in the original report? Read
in contrast with iterators based on it like Lines
or the above ParseIter
does not have a fixed item count or rather buffer size. To me, it appears that how to go from bytes the items is out of scope for Rayon and this could be closed due to providing ParallelBridge
?
from rayon.
Yes, I think it's fair to say that ParBridge
solves Iterator
input, and we have no plans for Read
.
from rayon.
Related Issues (20)
- Potential `len` method ambiguity in `src/range.rs` HOT 5
- Potential to fix memory ordering for value HOT 2
- a broadcast that only gets executed on idle threads
- `in_place_scope` documentation is confusing/unclear HOT 14
- use_current_thread and the global thread pool HOT 2
- Why does ThreadPool block until a operation is finished? HOT 1
- A way to hook before/after job execution (and/or a way to see a number of tasks in the processing queue)
- Function dependencies with Rayon without blocking the thread / Rayon synchronization primitives?
- Add synchronization guarantees for `ParallelIterator::for_each*` HOT 1
- Par bridge with optional buffering IO handling HOT 2
- Thread pool without work stealing HOT 6
- Question about return types from .map() HOT 4
- Terribly inefficient design and possible solution HOT 5
- [Discussion] Is it possible to impl `IntoIterator` for `ParallelIterator`? HOT 3
- Yield locks HOT 1
- the method `par_iter` exists for struct `ChunkByMut<'_, BinF32, {[email protected]:207:78}>`, but its trait bounds were not satisfied HOT 5
- Unupdated crossbeam dependency makes MIRI angry HOT 1
- help: How to implement memoized chunks for parallel iterator? HOT 2
- wrong repo, issue deleted HOT 3
- Comparison with Spice HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rayon.