Comments (6)
@cardin but in both cases it does use O(n) memory, and at some point that becomes too much.
Processing an array with "workers" uses only O(nWorkers) memory.
V8 is pretty efficient, but if small changes (amount of code is similar) can use less memory, that's a good thing, right? Less memory trashing => less GC => faster, even apart from host constraints
from async-sema.
What I have been doing is something like this:
await Promise.all(files.map(async (file) => {
await sema.acquire();
try {
await upload(file);
} finally {
sema.release();
}
}));
from async-sema.
@wmertens Actually, for your particular problem, you should look at this module instead: https://github.com/sindresorhus/p-queue
p-queue has some queue-based operations, event-emitters, and start/pause functionality.
async-sema has its uses of course, for e.g. if we are concerned about rate limiting.
from async-sema.
@OlliV That's nice too, with the difference that you create all the promises at once. So for <1000 items that's easier to reason about, but beyond that I'm worried about memory consumption.
@cardin hmmm p-queue seems to only work by adding functions to a queue, not by passing an array and having it be processed. So it doesn't help in this case, or am I mistaken?
from async-sema.
@wmertens The example code that OlliV gave shouldn't run into memory issues, since even 10,000 pending promises shouldn't be too much of an issue if they're not allowed to start.
p-queue
has an .addAll() that accepts an array of functions. Like queue.add(() => uploadOne(upload))
. As for why functions instead of promises, the author's rationale is to have more control over when the queue begins -> functions only execute the async operation when called, unlike promises which is uncontrollable.
from async-sema.
@wmertens oh right. I have seen that happening (OOM). I have used array-split for splitting a long array and processing it in chunks.
Maybe I could add it to the examples.
from async-sema.
Related Issues (17)
- Possible p() & v() confusion HOT 2
- broken typescript definition HOT 3
- Action required: Greenkeeper could not be activated 🚨
- Add examples to README
- Make it OSS
- Intuition surrounding `nr` values
- Rate limit and throttle: Last item throttled HOT 2
- Way to cancel pending tokens HOT 3
- .release() does not check actual token count HOT 3
- IE 11 Support HOT 3
- Args object should be optional
- Add Sema.run utility for easy task processing
- RateLimit does not work with number less than 1. HOT 2
- Uncaught TypeError: Class extends value #<Object> is not a constructor or null HOT 1
- Is it possible to make 1 request every 2 seconds? HOT 1
- What is the `nr` variable? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from async-sema.