fastparquet encoding issue. about fastparquet

[martindurant]

What is the data you are working with, please, how was it made?

[venkatsura]

we received it from other vendor. not sure how they are generating.

[venkatsura]

it's supporting for view of the online viewers and some of them were not able to open. i was able to load all the 25 files with ADF. if i assumed files are corrupted. it's should be load from any of the viewer or ETL.

[martindurant]

The indicated line in the code is reading a Parquet V1 data page, but encoding 5, DELTA_BINARY_PACKED, is a feature of Parquet V2. There's no technical reason V1 pages couldn't be decoded, but the code path does not exist.

[venkatsura]

I appologies for that. I didn't get you. you mean my code is refering v1. current i am using 0.8.3 fastparquet version.

[martindurant]

No, V1 of the parquet spec, which (roughly) corresponds to V1 type data pages, which is what your file has

[venkatsura]

Ok. As per my understanding, some of the files has been generated with V1 type and some of them are genereated with V2 type? Could you please suggest how can i verfiy? if that is the case which is the better approch to load V1 and V2 type files with fastparquet lib.

[venkatsura]

The NotImplementedError you're encountering indicates that fastparquet encountered an encoding that it does not support. The error message specifically mentions "Encoding 5," which is likely a Parquet encoding method or type that fastparquet does not currently handle.

[martindurant]

some of the files has been generated with V1 type and some of them are genereated with V2 type

I suppose so? fastparquet does not expose this information, although I can see it would be useful.

To be sure, it is the combination of v1 and encoding 5 that is the problem. encoding 5 works with v2, and other encodings work with v1.

[venkatsura]

Thank you so much.

[martindurant]

Given that the decoding algorithm already exists, I'll see if I can carve out a little time to implement it for v1 data pages - but I would need a file to test against.

[venkatsura]

I didn't find the file attachment. can you share me your email id?

[martindurant]

You can put binary files directly into a comment box here, but maybe it needs to be inside a .zip . Alternatively, you can start a private Gist and invite me if the file is sensitive.

I can't promise when I might get a chance to look into the matter.

[venkatsura]

TestFile_Failure.zip

[martindurant]

I am seeing BookingTypeIDs incrementing from 1 to 12. Is this right?

Can this test file be included in the repo?

[venkatsura]

yes. Thats the file only. please do not include in repo.

[martindurant]

@venkatsura , can you build and test this on your data, please?

[venkatsura]

I am using astparquet-2023.8.0. build my code and tested. it's giving the same error.

[venkatsura]

Thank you. It's working. May i know when it can be released to public. we are going to be deploy it to Azure. for azure lib's will be installed by pip online.

[venkatsura]

FYI.. we are doing some testing. we identified integer values are loading with negative (-) sign. I will post the details.