Comments (11)
by the "leakiness" and general friction seen here, it does seem a primitive is missing. I think it would be useful to have something like a semaphore in the standards library
from deno_std.
+1 for a semaphore
I just got a case where I have nested arrays of promises, all ending up in a remote API call, overloading the remote (or risk getting blocked/banned for too many open connections), so I need a way to limit the number of "active" API calls just where they are happening.
from deno_std.
Doesn't pooledMap()
achieve essentially the same thing with its poolLimit
parameter?
from deno_std.
Doesn't
pooledMap()
achieve essentially the same thing with itspoolLimit
parameter?
Not easily I think, this scenario is for when we don't know the iterable up front. So we still have to create and manage an iterable here (and probably prefer to use some lib for it). How would you write the above code with pooledMap?
from deno_std.
With the above code, how do you imagine the iterable would be provided to it?
from deno_std.
I'm not sure pLimit
is common abstraction of that situation/requirement.
Could Semaphore
class in this package ( https://jsr.io/@lambdalisue/async ) achieve that same thing?
This issue ( #4536 ) might be related to this topic.
from deno_std.
Would that be:
import { Semaphore } from "@std/async";
const limiter = new Semaphore(10);
// only 10 concurrent requests
export const geocodeAddress = limiter.lock(async (address: string) => {...})
If it's about that, np.
Btw, how would you rate limit invocations? (max calls per duration)
from deno_std.
I imagine you could do something like this to accomplish the same thing you're after.
import { pooledMap } from '@std/async'
function pLimit(poolLimit: number) {
return function <T, R>(iteratorFn: (data: T) => Promise<R>) {
return function <T>(array: Iterable<T> | AsyncIterable<T>) {
return pooledMap(poolLimit, array, iteratorFn)
}
}
}
const limit = pLimit(10)
const geocodeAddress = limit(async (address: string) => {...})
geocodeAddress(['an array of addresses'])
from deno_std.
I imagine you could do something like this to accomplish the same thing you're after.
import { pooledMap } from '@std/async' function pLimit(poolLimit: number) { return function <T, R>(iteratorFn: (data: T) => Promise<R>) { return function <T>(array: Iterable<T> | AsyncIterable<T>) { return pooledMap(poolLimit, array, iteratorFn) } } } const limit = pLimit(10) const geocodeAddress = limit(async (address: string) => {...}) geocodeAddress(['an array of addresses']) ``
your code is not equivalent. it requires all addresses to be batched in the same lexical place. In practice, to be equivalent you need to introduce state.
from deno_std.
your code is not equivalent. it requires all addresses to be batched in the same lexical place. In practice, to be equivalent you need to introduce state.
const { readable, writable } = new TranformStream<string, string>()
geocodeAddress(readable)
const writer = writable.getWriter()
// write an address in different places
writer.write('address')
// call at the very end when you know there is no more addresses
writer.close()
from deno_std.
import { pooledMap } from '@std/async'
const geocodeAddress = function() {
const { readable, writable } = new TransformStream<string, string>()
pooledMap(10, readable, async (address: string) => {
...
})
return writable.getWriter()
}()
geocodeAddress.write('address')
geocodeAddress.close()
from deno_std.
Related Issues (20)
- stabilize `@std/testing`
- stabilize `@std/jsonc`
- stabilize `@std/csv`
- stabilize `@std/semver`
- stabilize `@std/expect`
- stabilize `@std/yaml`
- stabilize `@std/front-matter` HOT 1
- stabilize `@std/ini`
- stabilize `@std/dotenv`
- msgpack: non-string/non-number key supprt HOT 1
- `describe` should throw if passed a function that returns a promise HOT 2
- Add cli for dotenv like npm dotenv-cli HOT 1
- `@std/cli`: Support multiple spinners at the same time HOT 9
- Support Cookies Having Independent Partitioned State (CHIPS)/Partioned cookies in http/cookie.ts HOT 1
- deleteCookie doesn't allow deleting cookies with the SameSite, Secure or unparsed attributes set
- proposal: drop `std/io` from the first stabilization targets HOT 4
- bug: packages appear twice in dependency graph
- to-do: remove access modifiers across the codebase HOT 1
- feature request(streams): lines(readable): AsyncIterable<string> HOT 6
- suggestion: internal `assertStrictThrows()` and `assertStrictRejects()` HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deno_std.