mcollina / hwp

hwp

Consume Async Interators with a highwatermark, i.e. in batches. This allow for multiple processing happening in parallel instead of one at a time.

The default highwatermark is 16.

Install

npm i hwp

Usage

import { forEach, map, mapIterator, mapper } from 'hwp'
import { pipeline } from 'stream/promises'

const expected = ['a', 'b', 'c']

async function * something () {
  const toSend = [...expected]
  yield * toSend
}

await forEach(something(), async function (item, { signal }) {
  return someAsyncFunction(item, { signal })
}, 16)

const res = mapIterator(something(), async function (item, { signal }) {
  return someAsyncFunction(item, { signal })
}, 16)

for await (const item of res) {
  console.log(item)
}

console.log(await map(something(), async function (item, { signal }) {
  return someAsyncFunction(item, { signal })
}), 16)

await pipeline(
  something(),
  mapper((item, { signal }) => {
    return someAsyncFunction(item, { signal })
  }, 16),
  async function (source) {
    for await (const item of source) {
      console.log(item)
    }
  }
)

License

MIT

hwp's Issues

mapIterator broken?

It looks to me like map iterator will never yield unless the concurrency limit has been reached. So basically if I have a conccurency of e.g. 128, I will never get any values until very much later (if at all).

fp version for pipeline compatibility?

Would be nice to have a fp version of e.g. mapIterator so we can do e.g.

pipeline(
  src,
  mapIterator(transform, 16), // (fn, concurrency) => src => mapIterator(src, fn, concurrency),
  dst
)

mapIterator needs abort signal

Same problam as in nodejs/node#39067