Comments (4)
Some preliminary testing shows that this is not too difficult. Fortunately both the FileReader API and AJAX requests are asynchronous, so the groundwork is already laid to parse chunks of a CSV file asynchronously. We just need to build in support for using something like this:
$.ajax("/plu_codes.csv", {
type: "GET",
headers: {
"Range": "bytes=0-1024"
}
}).done(function(data)
{
// ... treat data just as if it was a chunk read from the FileReader
});
from papaparse.
Thinking out loud here...
Already in the current version of Papa Parse, downloading a file and parsing it (assuming it is not too huge and can fit in memory well -- say, under 100 MB) works as easy as this:
$.get("some_file.csv", function(data) {
var results = $.parse(data);
});
But that doesn't "stream" the file: if that file is too big for the browser tab to handle, say even 1 GB, then this would just cause the browser the lock up. In order to download and parse huge files, while keeping Papa easy to use, what about invoking Papa so that it uses the Range header as described above like this:
$.parse("some_file.csv", {
ajax: true,
step: function(data, jqxhr) {
console.log(data.results);
}
});
So you specify ajax: true
in the config object in order to tell Papa that the string you gave it is a path to a CSV file to download, so it uses a GET request to download the file. If you also specify the step
function, as we have here, it uses the Range header to stream the file chunks at a time.
Two things to work out still:
- When doing AJAX parsing, the call to
$.parse
is asynchronous, meaning it needs a callback function. This is similar to how files are already parsed (you supply acomplete
) callback. How should this work? - The underlying AJAX requests aren't customizable. I'm worried that letting users pass in a config object for
$.ajax
would make it easy for users to break Papa unintentionally. In other words, the target file better be accessible with a simple GET request. It's a tradeoff I think I'm willing to make, but will accept feedback if anyone has it.
Since this is for 3.0, I'm willing to make big breaking changes to keep Papa easy to use.
from papaparse.
Okay, I think I've resolved both those things.
$.get.parse("files/asdf.csv", {
config: {
step: function(data, handle) {
console.log(data, handle);
// handle gives access to pause(), resume(), jqxhr, etc.
}
},
complete: function(data) {
console.log("Done!");
}
});
Calling $.get.parse
indicates to Papa that the string given it should be downloaded via a GET request to then be parsed. The second argument has basically the same object structure as when you parse a file, thus resolving number (1) from above.
Number (2) above is resolved because I've decided that the AJAX request done by Papa will be a simple GET request. However, the internal functions that perform the network requests, file reading, and do the parsing will be exposed so the user can utilize them at a lower level if desired.
from papaparse.
Still have some tweaking and optimizing to do, but this is now done.
from papaparse.
Related Issues (20)
- I found an example of a local file without jQuery HOT 2
- Add npm import docs HOT 2
- Header columns in double quotes ends up with an InvalidQuotes error when I dont think it should HOT 1
- Delimiter detection issue with decimal values HOT 1
- transformHeader returns function instead of string: trim is not a function HOT 8
- Incorrect header detection in CSV parsing 5.4.1 HOT 1
- Error in dynamic import in web component HOT 5
- PapaParse with stimulus.js HOT 1
- characters
- Feature Request: progress callback + complete callback with the bulk of the parsed entries
- Papa Unparse Delimiter and QuoteChar not working together HOT 1
- Preview of first row fails with "UndetectableDelimiter" error if file is too large HOT 1
- Parsing data with duplicate headers in streaming mode results in empty result HOT 1
- Extra columns when parsing CSV rows with JSON object HOT 1
- For CSV comma-delimited, when a row ends with an empty cell, papaparse mangles the next row's first cell HOT 1
- Autodetect of date formats
- renamedHeaders not showing in meta HOT 4
- Parse string with \n chars
- transformHeader called twice HOT 7
- nrows / skiprows HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from papaparse.