kbajalc / parquets Goto Github PK
View Code? Open in Web Editor NEWLicense: MIT License
License: MIT License
Is there anyway to upload parquet file directly to cloud without creating a local file?
WARNING: There are compatibility issues with the reference implementation when using 'optional' columns!
I'm assuming I should avoid optional columns altogether, but not sure I can.
node-int64 dependency has published 8 years ago and it's GitHub repository is in archived stated now.
It has Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
issue unresolved.
Are there any plans to migrate to BigInt
or some other alternative?
I am trying to read a parquet file buffer from a React application, using this code:
const fileDataBuffer = Buffer.from(fileData);
let reader = await ParquetReader.openBuffer(fileDataBuffer);
// create a new cursor
let cursor = reader.getCursor();
// read all records from the file and print them
let record = null;
while (record = await cursor.next()) {
console.log(record);
}
However, when the code is executing this: record = await cursor.next()
, I am getting the following error:
TypeError: cursor.buffer.readUInt32LE is not a function
decodeValues_BYTE_ARRAY plain.ts:191
decodeValues plain.ts:43
decodeValues reader.ts:354
decodeDataPage reader.ts:496
decodeDataPages reader.ts:385
readColumnChunk reader.ts:321
readRowGroup reader.ts:295
next reader.ts:63
processFile index.tsx:169
onSuccess index.tsx:215
NotifyManager notifyManager.js:62
notifyFn notifyManager.js:10
flush notifyManager.js:77
flush notifyManager.js:76
batchedUpdates$1 React
flush notifyManager.js:75
promise callback*scheduleMicrotask utils.js:322
flush notifyManager.js:74
batch notifyManager.js:30
dispatch query.js:390
setData query.js:83
onSuccess query.js:334
resolve retryer.js:58
promise callback*run retryer.js:116
Retryer retryer.js:156
fetch query.js:330
executeFetch queryObserver.js:199
setOptions queryObserver.js:85
useBaseQuery useBaseQuery.js:78
React 5
unstable_runWithPriority scheduler.development.js:468
React 4
unstable_runWithPriority scheduler.development.js:468
React 5
onSuccess useClientInputDatasets.tsx:55
notify queryObserver.js:497
batch notifyManager.js:25
notify queryObserver.js:494
updateResult queryObserver.js:454
onQueryUpdate queryObserver.js:484
Could someone help me understand the issue here and how can I fix it? Appreciate it.
While you can use openStream to write to any stream. Utils assumes it's a file stream and calls close in osclose.
Is this project maintained?
I guess it started as a fork of parquetjs which was not really well maintained (and is still not).
But now it seems parquets is not really maintained too. Is there any plan to add contributors so issues and pull requests can receive some answers?
Maybe we can considere a fork if no answer is provided, anyone?
Hello,
When I try to upload the parquet file to S3 Bucket, the file is getting created with the set name but it's creating without any content in it.
Is there any fix?
When writing out parquet the min and min_value metadata can be used by a reader to skip row groups and even entire files. However, this metadata is not currently calculated by this version of the library.
It would be great to port over this feature from https://github.com/ZJONSSON/parquetjs
Would you accept a PR with this?
It would be nice to port over the S3 support from https://github.com/ZJONSSON/parquetjs
Would you accept a PR for this?
I'm trying to read a file which contains ENUM types, however (e.g. parquet-tools schema
shows something like required binary entryMethod (ENUM);
) but parquet.ParquetReader.openFile()
just throws an error:
Error: invalid parquet type: ENUM
... which means I can't use this package to read my files.
What are my options ?
Are there any plans to support this type ?
Thanks,
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.