Is README accurate? about orc

[scritchley]

Closing this for now. I will be making writer support official in the upcoming couple of weeks.

[scritchley]

Yes, there is a writer implementation that works, although, there are still cases where its behaviour might not or does not conform to the ORC spec. Namely:

• Row indexes - It's quite hard to decipher exactly what the spec intends: https://orc.apache.org/docs/spec-index.html
• Compression - Currently all of the compression writers do not compress, they simple write the file uncompressed.

It's this that's made me refrain from declaring it as a supported feature. I'm waiting on a project that requires the writer functionality to more thoroughly test it.

[5k3105]

Hi,

I'm waiting on a project that requires the writer functionality to more thoroughly test it.

I have half a dozen projects that may benefit from a good compressed database format.

Don't really need acid (though nice), just need something compressed effectively.

sqlite is bloated and I hear lmdb gets large as well.

[binary132]

Hello @scritchley, we're looking at using your writer. Do you think it's working reasonably well? I see you have zlib compression now.

How bad is the row index issue?

[scritchley]

The issue (if there is one) specifically relates to the 'positions' field in the row index detailed here: https://orc.apache.org/docs/spec-index.html

The writer as it stands does write the indexes, however, the code that writes the positions may not behave as intended by the spec. I suppose this could cause issues in readers that utilise the indexes to prune rows. I've had limited experience of this in practice though.

I would need to compare the row indexes created by this writer with those created by one of the official tools.