Comments (13)
It is not currently possible, but one of my big To Do items for rayon is to factor out the backend into a distinct crate.
from rayon.
FWIW I'm running into this I believe with the wasm32-unknown-unknown
target where std::thread::spawn
doesn't work but we're able, with wasm-bindgen
, to get something that looks similar-ish to thread spawning. In that sense Rayon can't spawn any threads because it won't work, but I can either empower it to spawn threads or give it a pool of threads to draw from.
(just wanted to chime in with another use case!)
from rayon.
The install
closure has no arguments, but you can capture anything you like. For instance:
let hash_map = get_map();
my_pool.install(|| {
// this implicitly captures a reference to `hash_map`
hash_map.par_iter().for_each(|(k, v)| {
// do stuff ...
})
});
from rayon.
@alexcrichton Ultimately, wasm should just implement std::thread
normally, right? In this case, it feels like a custom spawn is just a stopgap measure while wasm works out its thread story.
The default function would just be
|_index, body| std::thread::spawn(body)
, something like that.
Our current spawn loop looks like this:
for (index, worker) in workers.into_iter().enumerate() {
let registry = registry.clone();
let mut b = thread::Builder::new();
if let Some(name) = builder.get_thread_name(index) {
b = b.name(name);
}
if let Some(stack_size) = builder.get_stack_size() {
b = b.stack_size(stack_size);
}
if let Err(e) = b.spawn(move || unsafe { main_loop(worker, registry, index, breadth_first) }) {
return Err(ThreadPoolBuildError::new(ErrorKind::IOError(e)))
}
}
If we did this with a user's spawn function, we'd need arguments for the name
, the stack_size
, and the spawn
closure, but the last is an FnOnce
which makes callbacks awkward. I guess we can also pass the index
as you suggest, but I'm not sure what they would use it for.
We could instead define a trait like ThreadBuilder
, with methods matching std::thread::Builder
as needed. Only the spawn
function is really required, and the name
and stack_size
could probably just be defaulted as no-ops.
from rayon.
From looking at the source, I wonder if as an alternative (or in addition/ temporal workaround), it would be possible to have a finer control of the available number of threads for execution at a given time (ie. setting it up before running a parallel iterator chain) during runtime though a function call (that would in turn modify the Registry).
This would be really helpful in the context of load balancing in an application where you have several live threads competing over resources with different pools (other than rayon), for example. Is a bit hackish, but still something.
(edit: NVM, cleared up by cuviper The ideal solution would be IMO to have the same functionality provided by the global thread pool provided by instantiated ThreadPool types so we would have total control over how we split resources, a good steep would be to have something more akin https://github.com/frewsxcv/rust-threadpool incorporated to rayon).
from rayon.
The ideal solution would be IMO to have the same functionality provided by the global thread pool provided by instantiated ThreadPool types so we would have total control over how we split resources
What do you mean by this? Rayon's global thread pool is literally just a global ThreadPool
instance. You can ThreadPool::install
your parallel iterator into whatever pool you like.
from rayon.
@cuviper Didn't see how to do it in the documentation (maybe is just that, a lack of documentation/example?), but for example:
hash_map
.par_iter()
.for_each(|(k, v)| // do stuff ... );
works but when if I use the install
method I can't (install
closure signature is FnOnce() -> R + Send so does not expect arguments), that's what i was thinking when saying 'have the same functionality'.
Probably missing something obvious.
from rayon.
https://github.com/nikomatsakis/rayon/pull/353 makes it possible to have a custom backend for parallel iterators, at least.
from rayon.
The rustc-rayon fork also adds a "custom main function" -- seems like if you could specify the "spawn thread" function, which @alexcrichton would like, then you could also control the "main". The main difference is that rustc-rayon would also like to know the thread index.
from rayon.
Oh that could work! (I think?)
In wasm we'll for sure have a way to get the thread index
from rayon.
What I mean is:
I was envisioning that we could give you (on the thread-pool) the ability to specify the spawn function. When spawning threads, we would call the function with an integer (thread index) plus a closure that you are supposed to invoke. The default function would just be |_index, body| std::thread::spawn(body)
, something like that.
from rayon.
We could get that to work!
from rayon.
@cuviper to me at least it's not actually clear whether std::thread
will ever be implementable with wasm. I suspect in the limit of time it will likely happen, but likely for the next few years it will remain unimplemented. The current proposal is too minimal to implement std::thread
as-is but there's other future proposals/ideas which may empower it.
I think the ideal interface for wasm would be something along the lines of "rayon, you can control this thread for some time" where a thread sort of opts-in to being a rayon worker thread. Requiring that rayon is still the one to spawn the thread may be too restrictive still, but I don't mind testing out to see if it's the case!
from rayon.
Related Issues (20)
- Synchronous nature of join might result in suboptimal use of compute resources HOT 1
- par_bridge and IndexedParallelIterator HOT 2
- Feature Request: par_enumerate() HOT 3
- rayon_wasm : "Operation not supported on this platform" when setting number of global threads to be used HOT 1
- Extremely deep call stack on MacOS HOT 2
- calling `buffer.par_sort_unstable_by_key` from a task calls the task itself HOT 4
- How to implement ParallelIterator for a custom Range? HOT 13
- How to dynamically change the number of threads during runtime? HOT 2
- `ParallelExtend` for tuples of references HOT 3
- Using async iterator-like SQLX fetch with Rayon HOT 4
- Docs on "spawn" don't say what exactly this function does HOT 1
- Add SIMD SORT as an option HOT 1
- Handle/guard support for current thread pool HOT 1
- Optional parallelization
- Way to have assertion whether something is outside of a rayon task HOT 2
- how drop rayon whren it in a dylib and dylib should be droped? HOT 4
- Error reporting in scoped tasks
- cooperative yield in ThreadPool::install() causes unexpected behavior in nested pools HOT 4
- rayon-core tests fail to build. HOT 6
- Matrix multiplication with Rayon doesn't see perf improvements HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rayon.