achingbrain / it

A collection of utilities for making working with iterables more bearable

it

A collection of utilities for making working with iterables more bearable

Packages

• /packages/blob-to-it Turns a blob into an async iterator
• /packages/browser-readablestream-to-it Turns a browser readble stream into an async iterator
• /packages/it-all Collects all values from an (async) iterable and returns them as an array
• /packages/it-batch Takes an async iterator that emits things and emits them as fixed size batches
• /packages/it-batched-bytes Takes an async iterator that emits byte arrays and emits them as fixed size batches
• /packages/it-buffer-stream An async iterator that emits buffers containing bytes up to a certain length
• /packages/it-byte-stream Read and write arbitrary bytes over a duplex stream
• /packages/it-drain Empties an async iterator
• /packages/it-filter Filters the passed iterable by using the filter function
• /packages/it-first Returns the first result from an async iterator
• /packages/it-flat-batch Takes an async iterator that emits variable length arrays and emits them as fixed size batches
• /packages/it-foreach Invokes the passed function for each item in an iterable
• /packages/it-glob Async iterable filename pattern matcher
• /packages/it-last Returns the last result from an async iterator
• /packages/it-length Counts the number of items in an async iterable
• /packages/it-length-prefixed-stream Read and write length-prefixed byte arrays over a duplex stream
• /packages/it-map Maps the values yielded by an async iterator
• /packages/it-merge Treat one or more iterables as a single iterable
• /packages/it-multipart Async iterable http multipart message parser
• /packages/it-ndjson Parse iterators as ndjson and transform iterators to ndjson
• /packages/it-parallel Process incoming async(iterable) functions in parallel
• /packages/it-parallel-batch Process (async)iterable values as functions with concurrency control
• /packages/it-peekable Allows peeking/pushing an iterable
• /packages/it-protobuf-stream Read and write protobuf messages over a duplex stream
• /packages/it-reduce Reduces the values yielded from an async iterator
• /packages/it-skip Skip items from an iterable
• /packages/it-sort Collects all values from an async iterator, sorts them using the passed function and yields them
• /packages/it-split Splits Uint8Arrays emitted by an (async) iterable by a delimiter
• /packages/it-take Stop iteration after n items have been received
• /packages/it-to-browser-readablestream Takes an async iterator and turns it into a browser readable stream
• /packages/it-to-buffer Takes an async iterator that yields buffers and concatenates them all together

API Docs

• https://achingbrain.github.io/it

License

Licensed under either of

• Apache 2.0, (LICENSE-APACHE / http://www.apache.org/licenses/LICENSE-2.0)
• MIT (LICENSE-MIT / http://opensource.org/licenses/MIT)

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

it's Issues

Upgrade it-* dependencies in all packages

It seems it-merge still uses an old version of it-pushable https://github.com/achingbrain/it/blob/master/packages/it-merge/package.json#L22

it-merge is a dependency of it-pipe (and likely others)

There are likely other packages here too that also need upgrading to push the upgraded it-* dependencies through

fix: does not accept asyncIterable<Uint8Array>

but documentation says it should accept any asyncIterable:

Turns an (async)iterable into a W3C ReadbleStream.
// This can also be an iterator, async iterator, generator, etc

Functions with union Promised return types "Promise<void> | void" will fail eslint checks.

ESlint is not able to handle conditional returns like this in terms of awaiting.

Looks like folks over at typescript-eslint suggest to always mark functions that can return a promise with async, even if it returns immediately.

typescript-eslint/typescript-eslint#6421

Ran into this problem when awaiting drain, eslint threw back this error in response.

Unexpected `await` of a non-Promise (non-"Thenable") value. [Error/@typescript-eslint/await-thenable]
Placing a void expression inside another expression is forbidden. Move it to its own statement instead. [Error/@typescript-eslint/no-confusing-void-expression]

When publishing these packages, publish built / transpiled JavaScript

Cool project!

@achingbrain When publishing these packages to npm, etc., would you consider publishing built / transpiled JavaScript using a tool like rollup? Otherwise it makes it hard to use these libs in a project with a different Babel config.

This would mean adding a build script for each child package so that they're all run with the lerna command from the root dir.

Thanks!

it-latest: drop messages w/ backpressure

This would be useful for rate limiting.

This would be a stage in a pipe() which keeps & emits the most recently received value (from another iterable) and drops messages during backpressure (keeping the latest message only).

formidable v1 is deprecated

Can you pls update formidable? Used in it-multipart at least.

https://github.com/node-formidable/formidable/blob/master/VERSION_NOTES.md

browser-readablestream-to-it should release reader lock and cancel stream on break

Implementation here doesn't release reader lock

it/packages/browser-readablestream-to-it/index.js

Lines 3 to 15 in 1b928cb

	async function * browserReadableStreamToIt (stream) {
	const reader = stream.getReader()
	while (true) {
	const result = await reader.read()
	if (result.done) {
	return
	}
	yield result.value
	}
	}

According to the ReadableStreams specification 4.2.5. Asynchronous iteration

for await (const chunk of stream) { ... }

for await (const chunk of stream.values({ preventCancel: true })) { ... }

Asynchronously iterates over the chunks in the stream’s internal queue.

Asynchronously iterating over the stream will lock it, preventing any other consumer from acquiring a reader. The lock will be released if the async iterator’s return() method is called, e.g. by breaking out of the loop.

By default, calling the async iterator’s return() method will also cancel the stream. To prevent this, use the stream’s values() method, passing true for the preventCancel option.

Would be nice to also have { preventCancel: true } option as in spec

it-multipart: silently skips part of the file with diacritics

When uploaded a file with diacritics, such as the aáa.txt, a multipart iterator silently skips it.

TypeScript doesn't allow use with AsyncIterator<T>

it-take has a comment like this in its usage example:

// This can also be an iterator, async iterator, generator, etc
const arr = await all(take([0, 1, 2, 3, 4], 2))

So, I tried to use it with an async iterator. But the released packages include TypeScript definitions that look like this:

cd `mktemp -d`
npm pack it-take
tar zxf it-take-*.tgz
cat package/dist/index.d.ts

/**
 * Stop iteration after n items have been received.
 *
 * @template T
 * @param {AsyncIterable<T>|Iterable<T>} source
 * @param {number} limit
 * @returns {AsyncIterable<T>}
 */
declare function take<T>(source: AsyncIterable<T> | Iterable<T>, limit: number): AsyncIterable<T>;

Specifically, the source parameter is typed as AsyncIterable<T> | Iterable<T>

Unless I'm missing something, this does not seem to allow an AsyncIterator<T> to be passed?AsyncIterator<T> doesn't meet the type constraint, so trying to use it with an AsyncIterator<number> results in:

error TS2345: Argument of type 'AsyncIterator<number, any, undefined>' is not assignable to parameter of type 'AsyncIterable | Iterable'.
Property '[Symbol.iterator]' is missing in type 'AsyncIterator<number, any, undefined>' but required in type 'Iterable'.

---

Two things indicate to me this is only a typing error, and can be fixed in just the TS definitions:

• Converting the iterator to an actual iterable results in it working, e.g. using something like:

  const toIterable = <T>(iterator: AsyncIterator<T>): AsyncIterable<T> => {
      return {
          [Symbol.asyncIterator]() {
              return iterator
          },
      }
  }

• But, more importantly, type-casting the iterator to an iterable with no further changes also works as-is (i.e. take(asyncIteratorOfNumber as unknown as AsyncIterable<number>, 1) works fine)