Comments (3)
https://observablehq.com/@mbostock/streaming-csv
from d3-dsv.
This is fine for Node: https://github.com/mafintosh/csv-parser
Would still be nice to have streaming CSV parsing in the browser.
from d3-dsv.
you probably mean something else, but I wrote a little class that "streams" csv rows as they are loaded through the progress event:
import { request, csvParse, csvParseRows } from 'd3'
import EventEmitter from 'eventemitter3'
export default class CsvLoader extends EventEmitter {
constructor (url) {
super()
this.loadedChars = 0
this.header = undefined
this.data = []
this.xhr = undefined
this.request = request(url)
.mimeType('text')
.on('progress', this.onProgress)
.on('load', () => this.emit('loaded'))
return this
}
get () {
this.request.get()
return this
}
onProgress = (p) => {
const progress = p.total / p.loaded
const responseText = p.target.responseText
// console.log(responseText.length)
const lineBreak = responseText.lastIndexOf('\n') + 1
const char = responseText.slice(this.loadedChars, lineBreak)
let parsed = csvParseRows(char)
if (!this.header) {
this.header = parsed.shift()
}
const obj = parsed.map(p =>
this.header.reduce((a, b, i) => {
a[b] = p[i]
return a
}, {})
)
this.loadedChars = lineBreak
this.emit('row', obj)
this.emit('progress', progress)
}
}
from d3-dsv.
Related Issues (20)
- csv2json cli throws error when using --auto-type
- Add shoutout to d3.autoType in d3.parse documentation
- TypeError: Cannot read property 'trim' of undefined HOT 3
- dsv.parse's returned columns property doesn't reflect row conversion function HOT 4
- Use Object.fromEntries to convert arrays into objects? HOT 1
- date tests fail HOT 2
- Export individual "formatRow" and "formatValue" functions HOT 5
- Avoid "Function" constructor HOT 1
- Add documentation about CSP incompatibility HOT 1
- Could we possibly add a ssv() or wssv()? HOT 4
- [RFC] Split out the CLI to a separate package HOT 3
- Repeated columns names erase each other for xParse HOT 6
- wrong parsing d3.tsvParse with only number header HOT 4
- d3.autotype: keep leading 0 HOT 1
- Option to quote empty strings? HOT 1
- Error: Must use import to load ES Module HOT 4
- Potential performance issue - Columns property added to returned array from dsv.parse could lose browser engine optimization on array operations HOT 1
- Column data will be lost if the column name is the same HOT 1
- `objectConverter` can work without `Function`. HOT 1
- https://observablehq.com/@mbostock/streaming-csv HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from d3-dsv.