alanclarke / batch-stream-csv

Process large CSV files in batches without backpressure

batch-stream-csv

Process large CSV files in batches without backpressure

The CSV is streamed in batches and the stream is paused while you perform async operations on each batch to prevent backpressure building up

Uses csv-parser under the hood

Install

$ npm install batch-stream-csv

Usage:

const batch = require('batch-stream-csv');

batch('./file.csv', batchHandler, options).then(() => console.log('All done!'))

async function batchHandler (batch, progress) => {
  console.log(batch) // batch of rows parsed as JSON, e.g. [{ a: 1, b: 2 }, {  a: 2, b: 3 }]

  await db.batchInsert('table', batch) // the stream is paused while you do your inserts

  console.log(`${progress * 100}% complete`) // progress value can be fed into a progress bar
}

API:

Options:

• batchSize: number of rows to process per batch (default = 5000)
• ...see https://github.com/mafintosh/ for other options