Comments (4)
I removed inner parallelism instead, it was easier to implement and didn't have a significant impact on performance in my workload.
Thanks again for your help
from rayon.
Reproduced on an AMD Ryzen 9 7950X, since @progval was worried about their CPU being bugged. ;)
Add this to the top of the file for use with cargo play
:
//# stderrlog = '*'
//# thread_local = '*'
//# rayon = '*'
from rayon.
This is an instance of #592 -- the call to par_sort
may lead to work-stealing while it waits for sub-tasks to finish, and that may steal another piece of your task that wants to use that mutex, making you re-entrant.
As a workaround, maybe your TLS can have an Option<Vec<T>>
for the buffer that you take
out while you're working with it, rather than holding the Mutex
locked? Then store it back as Some
when you're done. When re-entrancy finds it None
, you can allocate a new one instead. Maybe you can even keep Vec<Vec<T>>
to save multiple allocations just in case, with pop
to use and then push
it back (or push
the new one).
(I'm assuming the buffer is to save allocation time, not needing re-used data between instances.)
from rayon.
This is an instance of #592
Ouch. I noticed that issue but assumed I wouldn't be affected because of the thread-specific mutexes.
may lead to work-stealing while it waits for sub-tasks to finish
Indeed, that matches experience of the deadlocks often occurring toward the end of the execution.
As a workaround, maybe your TLS can have an Option<Vec> for the buffer that you take out while you're working with it, rather than holding the Mutex locked? Then store it back as Some when you're done. When re-entrancy finds it None, you can allocate a new one instead.
Makes sense. This should be pretty rare so the overhead won't be too much. (I'll need to keep the mutexes instead of UnsafeCells I initially used, though.)
Thanks!
(I'm assuming the buffer is to save allocation time, not needing re-used data between instances.)
Correct, my actual code .clear()
s it after sorting.
from rayon.
Related Issues (20)
- Potential to fix memory ordering for value HOT 2
- a broadcast that only gets executed on idle threads
- `in_place_scope` documentation is confusing/unclear HOT 14
- use_current_thread and the global thread pool HOT 2
- Why does ThreadPool block until a operation is finished? HOT 1
- A way to hook before/after job execution (and/or a way to see a number of tasks in the processing queue)
- Function dependencies with Rayon without blocking the thread / Rayon synchronization primitives?
- Add synchronization guarantees for `ParallelIterator::for_each*` HOT 1
- Par bridge with optional buffering IO handling HOT 2
- Thread pool without work stealing HOT 6
- Question about return types from .map() HOT 4
- Terribly inefficient design and possible solution HOT 5
- [Discussion] Is it possible to impl `IntoIterator` for `ParallelIterator`? HOT 3
- Yield locks HOT 1
- the method `par_iter` exists for struct `ChunkByMut<'_, BinF32, {[email protected]:207:78}>`, but its trait bounds were not satisfied HOT 5
- Unupdated crossbeam dependency makes MIRI angry HOT 1
- help: How to implement memoized chunks for parallel iterator? HOT 2
- wrong repo, issue deleted HOT 3
- Comparison with Spice HOT 1
- Switching par_iter makes memory usage explode (40x) HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rayon.