i-mortals / carbites

🚗 🚙 🚕 Chunking for CAR files. Split a single CAR into multiple CARs.

carbites

Chunking for CAR files. Split a single CAR into multiple CARs.

Install

npm install carbites

Usage

Carbites supports 3 different strategies:

1. Simple (default) - fast but naive, only the first CAR output has a root CID, subsequent CARs have a placeholder "empty" CID.
2. Rooted - like simple, but creates a custom root node to ensure all blocks in a CAR are referenced.
3. Treewalk - walks the DAG to pack sub-graphs into each CAR file that is output. Every CAR has the same root CID, but contains a different portion of the DAG.

Simple

import { CarSplitter } from 'carbites'
import { CarReader } from '@ipld/car'
import fs from 'fs'

const bigCar = await CarReader.fromIterable(fs.createReadStream('/path/to/big.car'))
const targetSize = 1024 * 1024 * 100 // chunk to ~100MB CARs
const splitter = new CarSplitter(bigCar, targetSize) // (simple strategy)

for await (const car of splitter.cars()) {
  // Each `car` is an AsyncIterable<Uint8Array>
}

⚠️ Note: The first CAR output has roots in the header, subsequent CARs have an empty root CID bafkqaaa as recommended.

Rooted

Instead of an empty CID, carbites can generate a special root node for each split CAR that references all the blocks and the original roots (only in the first CAR). To do this, use the RootedCarSplitter constructor. When reading/extracting data from the CARs, the root node should be discarded.

Example

import { RootedCarSplitter } from 'carbites/rooted'
import { CarReader } from '@ipld/car/reader'
import * as dagCbor from '@ipld/dag-cbor'
import fs from 'fs'

const bigCar = await CarReader.fromIterable(fs.createReadStream('/path/to/big.car'))
const targetSize = 1024 * 1024 * 100 // chunk to ~100MB CARs
const splitter = new RootedCarSplitter(bigCar, targetSize)

const cars = splitter.cars()

// Every CAR has a single root - a CBOR node that is an tuple of `/carbites/1`,
// an array of root CIDs and an array of block CIDs.
// e.g. ['/carbites/1', ['bafkroot'], ['bafy1', 'bafy2']]

const { done, value: car } = await cars.next()
const reader = await CarReader.fromIterable(car)
const rootCids = await reader.getRoots()
const rootNode = dagCbor.decode(await reader.get(rootCids[0]))

console.log(rootNode[0]) // /carbites/1
console.log(rootNode[1]) // Root CIDs (only in first CAR)
/*
[
  CID(bafybeictvyf6polqzgop3jt32owubfmsg3kl226omqrfte4eyidubc4rpq)
]
*/
console.log(rootNode[2]) // Block CIDs (all blocks in this CAR)
/*
[
  CID(bafybeictvyf6polqzgop3jt32owubfmsg3kl226omqrfte4eyidubc4rpq),
  CID(bafyreihcsxqhd6agqpboc3wrlvpy5bwuxctv5upicdnt3u2wojv4exxl24),
  CID(bafyreiasq7d2ihbqm5xvhjjzlmzsensuadrpmpt2tkjsuwq42xpa34qevu)
]
*/

The root node is limited to 4MB in size (the largest message IPFS will bitswap). Depending on the settings used to construct the DAG in the CAR, this may mean a split CAR size limit of around 30GiB.

Treewalk

Every CAR file has the same root CID but a different portion of the DAG. The DAG is traversed from the root node and each block is decoded and links extracted in order to determine which sub-graph to include in each CAR.

Example

import { TreewalkCarSplitter } from 'carbites/treewalk'
import { CarReader } from '@ipld/car/reader'
import * as dagCbor from '@ipld/dag-cbor'
import fs from 'fs'

const bigCar = await CarReader.fromIterable(fs.createReadStream('/path/to/big.car'))
const [rootCid] = await bigCar.getRoots()
const targetSize = 1024 * 1024 * 100 // chunk to ~100MB CARs
const splitter = new TreewalkCarSplitter(bigCar, targetSize)

for await (const car of splitter.cars()) {
  // Each `car` is an AsyncIterable<Uint8Array>
  const reader = await CarReader.fromIterable(car)
  const [splitCarRootCid] = await reader.getRoots()
  console.assert(rootCid.equals(splitCarRootCid)) // all cars will have the same root
}

CLI

Install the CLI tool to use Carbites from the comfort of your terminal:

npm i -g carbites-cli

# Split a big CAR into many smaller CARs
carbites split big.car --size 100MB --strategy simple # (default size & strategy)

# Join many split CARs back into a single CAR.
carbites join big-0.car big-1.car ...
# Note: not a tool for joining arbitrary CARs together! The split CARs MUST
# belong to the same CAR and big-0.car should be the first argument.

API

• class CarSplitter
  • constructor(car: AsyncIterable<Uint8Array>, targetSize: number)
  • cars(): AsyncGenerator<AsyncIterable<Uint8Array> & RootsReader>
  • static async fromBlob(blob: Blob, targetSize: number): CarSplitter
  • static async fromIterable(iterable: AsyncIterable<Uint8Array>, targetSize: number): CarSplitter
• class CarJoiner
  • constructor(cars: Iterable<AsyncIterable<Uint8Array>>)
  • car(): AsyncGenerator<Uint8Array>
• class RootedCarSplitter
• class RootedCarJoiner
• class TreewalkCarSplitter
• class TreewalkCarJoiner

class CarSplitter

Split a CAR file into several smaller CAR files.

Import in the browser:

import { CarSplitter } from 'https://cdn.skypack.dev/carbites'

Import in Node.js:

import { CarSplitter } from 'carbites'

Note: This is an alias of SimpleCarSplitter - the default strategy for splitting CARs.

constructor(car: CarReader, targetSize: number)

Create a new CarSplitter for the passed CAR file, aiming to generate CARs of around targetSize bytes in size.

cars(): AsyncGenerator<AsyncIterable<Uint8Array> & RootsReader>

Split the CAR file and create multiple smaller CAR files. Returns an AsyncGenerator that yields the split CAR files (of type AsyncIterable<Uint8Array>).

The CAR files output also implement the RootsReader interface from @ipld/car which means you can call getRoots(): Promise<CID[]> to obtain the root CIDs.

static async fromBlob(blob: Blob, targetSize: number): CarSplitter

Convenience function to create a new CarSplitter from a blob of CAR file data.

static async fromIterable(iterable: AsyncIterable<Uint8Array>, targetSize: number): CarSplitter

Convenience function to create a new CarSplitter from an AsyncIterable<Uint8Array> of CAR file data.

class CarJoiner

Join together split CAR files into a single big CAR.

Import in the browser:

import { CarJoiner } from 'https://cdn.skypack.dev/carbites'

Import in Node.js:

import { CarJoiner } from 'carbites'

Note: This is an alias of SimpleCarJoiner - a joiner for the the default CAR splitting strategy.

constructor(cars: Iterable<CarReader>)

Create a new CarJoiner for joining the passed CAR files together.

car(): AsyncGenerator<Uint8Array>

Join the CAR files together and return the joined CAR.

class RootedCarSplitter

Split a CAR file into several smaller CAR files ensuring every CAR file contains a single root node that references all the blocks and the original roots (only in the first CAR). When reading/extracting data from the CARs, the root node should be discarded.

Import in the browser:

import { RootedCarSplitter } from 'https://cdn.skypack.dev/carbites/rooted'

Import in Node.js:

import { RootedCarSplitter } from 'carbites/rooted'

The API is the same as for CarSplitter.

Root Node Format

The root node is a dag-cbor node that is a tuple of the string /carbites/1, an array of root CIDs (only seen in first CAR) and an array of block CIDs (all the blocks in the CAR). e.g. ['/carbites/1', ['bafkroot'], ['bafy1', 'bafy2']].

Note: The root node is limited to 4MB in size (the largest message IPFS will bitswap). Depending on the settings used to construct the DAG in the CAR, this may mean a split CAR size limit of around 30GiB.

class RootedCarJoiner

Join together CAR files that were split using RootedCarSplitter.

The API is the same as for CarJoiner.

class TreewalkCarSplitter

Split a CAR file into several smaller CAR files. Every CAR file has the same root CID but a different portion of the DAG. The DAG is traversed from the root node and each block is decoded and links extracted in order to determine which sub-graph to include in each CAR.

Import in the browser:

import { TreewalkCarSplitter } from 'https://cdn.skypack.dev/carbites/treewalk'

Import in Node.js:

import { TreewalkCarSplitter } from 'carbites/treewalk'

The API is the same as for CarSplitter.

class TreewalkCarJoiner

Join together CAR files that were split using TreewalkCarSplitter.

The API is the same as for CarJoiner.

Releasing

You can publish by either running npm publish in the dist directory or using npx ipjs publish.

Related

• Carbites in Golang

Contribute

Feel free to dive in! Open an issue or submit PRs.

License

Dual-licensed under MIT + Apache 2.0