Comments (15)
from rayon.
You could put the map under a mutex
ah, thanks!
from rayon.
@flying-sheep I personally would not do this because of performance issues.
from rayon.
Collecting is likely very fast, mutez maybe not, but it depends on how many items and the relative cost of inserting them vs creating them. You could also achieve the async pattern by making a port and having the parallel iterator send msgs to the port (and starting a thread too). I actually do intend to add some kind of combinators or helpers like this but am not sure just how they should look.
On Feb 18, 2016 2:47 PM, Lilian Anatolie Moraru [email protected] wrote:@flying-sheep I personally would not do this because of performance issues.
—Reply to this email directly or view it on GitHub.
from rayon.
ok, then i’ll reopen this.
i don’t really like the idea of having to collect things just to go over them and building a map from them 😐
from rayon.
I think the right solution is to just pass the buck to a ConcurrentHashMap
. I don't know of any good implementations as of now, though.
from rayon.
I've got a somewhat related problem:
I'm using Rayon to count duplicate files within a directory. To keep a progress bar updated, I would like to send the current progress of the operation to a different thread that displays the progress bar (since updating the progress bar from within the thread that does the work is very slow). I've reduced the example to the following code:
https://gist.github.com/GyrosOfWar/059b70688904c6b5b433
The issue I'm having is that I can't share the Sender in the for_each closure, since it's not Sync
. The usual way would be to clone the Sender, but in this case, it doesn't seem to help. I'm pretty sure the problem is just me not understanding Rust's concurrency quite right, but I can't figure this out and Google is not helping (since cloning the Sender doesn't help)
from rayon.
@GyrosOfWar interesting. So this is actually just a bit tricky. (As an aside, I wonder if there is a good reason that the channel is not Sync
, I should check.) What you have to do is to clone the sender once for each directory, so something like:
let senders: Vec<_> = nums.iter().map(|_| sender.clone()).collect();
nums.into_par_iter().zip(senders).for_each(move |i, sender| {
let d = rx.clone();
d.send(i).unwrap();
});
but this seems sub-optimal. If nothing else, it might be nice to have a kind of "repeater" iterator for cloning things that are not Send only as needed. In other words, it would work something like
nums.into_par_iter().zip(repeat(sender)).for_each(|(num, sender_ref)| ...)
where sender_ref
has type &Sender
. The idea would be that it takes ownership of the sender and clones it for each thread split, but then lends it out for sequential processing (so as to do the minimal number of clones possible). That ought to be fairly easy to whip-up.
Of course the hard part is to come up with a good name: it seems like there are a number of similar "adapters" one might want for distinct ownership semantics (e.g., one that clones eagerly, since cloning is cheap, etc)
from rayon.
Channels lazily become Sync internally when cloned, as an optimization for the case when they don't need to be Sync. There was a rejected RFC recently about adding a Sync channel function. The problem as seen by the libs team with exposing a Sender that is Sync as its own type is that all existing code would be incompatible with it, as well as being utterly confusing with the existing inaptly-namer SyncSender that means a completely different kind of synchronous.
from rayon.
.zip(repeat(sender))
Hmm, zip
requires an IndexedParallelIterator
, but repeat
doesn't know the length. It could lie and say usize::MAX
, but I think it might be better to have zip+repeat
be its own special method, kind of like enumerate
is nearly zip+range
.
from rayon.
Of course, it'd probably be easier to just not use the channel, and instead use one of the queue's available from crossbeam, e.g. https://github.com/aturon/crossbeam/blob/master/src/sync/ms_queue.rs.
from rayon.
Thanks for the help! I'm using that lock-free queue plus a scoped thread from the crossbeam crate now and it seems to work well. Haven't tested it a lot yet, though. For anyone who's interested, the code is here (specifically, the find_duplicates
method in src/dupfinder.rs
)
from rayon.
@GyrosOfWar I guess you mean this code? It seems a bit surprising though because I think that the spawned thread will be joined by the end of-scope, and hence I don't think that thread is really doing anything.
If however you move the into_par_iter
code up inside the scope
closure, that would probably work.
FWIW this WIP API https://github.com/nikomatsakis/rayon/pull/71 would also help, but obviously that's not landed yet.
from rayon.
PR #131 adds for_each_atomic
, which basically fulfills the original feature request.
from rayon.
Sorry, I should have referred to https://github.com/nikomatsakis/rayon/pull/124, which specifically adds ability to collect into a hashmap.
from rayon.
Related Issues (20)
- Synchronous nature of join might result in suboptimal use of compute resources HOT 1
- par_bridge and IndexedParallelIterator HOT 2
- Feature Request: par_enumerate() HOT 3
- rayon_wasm : "Operation not supported on this platform" when setting number of global threads to be used HOT 1
- Extremely deep call stack on MacOS HOT 2
- calling `buffer.par_sort_unstable_by_key` from a task calls the task itself HOT 4
- How to implement ParallelIterator for a custom Range? HOT 13
- How to dynamically change the number of threads during runtime? HOT 2
- `ParallelExtend` for tuples of references HOT 3
- Using async iterator-like SQLX fetch with Rayon HOT 4
- Docs on "spawn" don't say what exactly this function does HOT 1
- Add SIMD SORT as an option HOT 1
- Handle/guard support for current thread pool HOT 1
- Optional parallelization
- Way to have assertion whether something is outside of a rayon task HOT 2
- how drop rayon whren it in a dylib and dylib should be droped? HOT 4
- Error reporting in scoped tasks
- cooperative yield in ThreadPool::install() causes unexpected behavior in nested pools HOT 4
- rayon-core tests fail to build. HOT 6
- Matrix multiplication with Rayon doesn't see perf improvements HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rayon.