kbajalc / parquets Goto Github PK

Is there anyway to upload parquet file directly to cloud without creating a local file?

WARNING: There are compatibility issues with the reference implementation when using 'optional' columns!

I'm assuming I should avoid optional columns altogether, but not sure I can.

node-int64 dependency has published 8 years ago and it's GitHub repository is in archived stated now.

It has Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead. issue unresolved.

Are there any plans to migrate to BigInt or some other alternative?

I am trying to read a parquet file buffer from a React application, using this code:

      const fileDataBuffer = Buffer.from(fileData);
      let reader = await ParquetReader.openBuffer(fileDataBuffer);
      
      // create a new cursor
      let cursor = reader.getCursor();
      
      // read all records from the file and print them
      let record = null;
      while (record = await cursor.next()) {
        console.log(record);
      }

However, when the code is executing this: record = await cursor.next(), I am getting the following error:

TypeError: cursor.buffer.readUInt32LE is not a function
    decodeValues_BYTE_ARRAY plain.ts:191
    decodeValues plain.ts:43
    decodeValues reader.ts:354
    decodeDataPage reader.ts:496
    decodeDataPages reader.ts:385
    readColumnChunk reader.ts:321
    readRowGroup reader.ts:295
    next reader.ts:63
    processFile index.tsx:169
    onSuccess index.tsx:215
    NotifyManager notifyManager.js:62
    notifyFn notifyManager.js:10
    flush notifyManager.js:77
    flush notifyManager.js:76
    batchedUpdates$1 React
    flush notifyManager.js:75
    promise callback*scheduleMicrotask utils.js:322
    flush notifyManager.js:74
    batch notifyManager.js:30
    dispatch query.js:390
    setData query.js:83
    onSuccess query.js:334
    resolve retryer.js:58
    promise callback*run retryer.js:116
    Retryer retryer.js:156
    fetch query.js:330
    executeFetch queryObserver.js:199
    setOptions queryObserver.js:85
    useBaseQuery useBaseQuery.js:78
    React 5
    unstable_runWithPriority scheduler.development.js:468
    React 4
    unstable_runWithPriority scheduler.development.js:468
    React 5
    onSuccess useClientInputDatasets.tsx:55
    notify queryObserver.js:497
    batch notifyManager.js:25
    notify queryObserver.js:494
    updateResult queryObserver.js:454
    onQueryUpdate queryObserver.js:484

Could someone help me understand the issue here and how can I fix it? Appreciate it.

While you can use openStream to write to any stream. Utils assumes it's a file stream and calls close in osclose.

Is this project maintained?

I guess it started as a fork of parquetjs which was not really well maintained (and is still not).

But now it seems parquets is not really maintained too. Is there any plan to add contributors so issues and pull requests can receive some answers?

Maybe we can considere a fork if no answer is provided, anyone?

Hello,

When I try to upload the parquet file to S3 Bucket, the file is getting created with the set name but it's creating without any content in it.

Is there any fix?

When writing out parquet the min and min_value metadata can be used by a reader to skip row groups and even entire files. However, this metadata is not currently calculated by this version of the library.

It would be great to port over this feature from https://github.com/ZJONSSON/parquetjs

Would you accept a PR with this?

It would be nice to port over the S3 support from https://github.com/ZJONSSON/parquetjs

Would you accept a PR for this?

I'm trying to read a file which contains ENUM types, however (e.g. parquet-tools schema shows something like required binary entryMethod (ENUM);) but parquet.ParquetReader.openFile() just throws an error:

Error: invalid parquet type: ENUM

... which means I can't use this package to read my files.

What are my options ?
Are there any plans to support this type ?

Thanks,