Memory Consumption Too High about simple_xlsx_reader

[levilansing]

For kicks I tried to read a 125MB xlsx file (1 sheet, plain data, no formatting). after about a minute and 18GB of ram allocated, I got a OOM error from Nokogiri.

[woahdae]

This has been addressed in https://github.com/woahdae/simple_xlsx_reader/tree/2.0.0-pre, I'd love feedback on the changes if anyone here still uses this gem.

[woahdae]

Fixed in 2.0, just released.

[EtienneDepaulis]

@chbach there is an other gem which does that: https://github.com/weshatheleopard/rubyX
It has the same dependencies however we found their API hard to use for our simple usage (importing 100+lines Excel files)

[chbach]

Thanks, I’ll give it a shot. However, I like the simplicity and API of this gem.

[EtienneDepaulis]

This is exactly why we switched to this gem this morning :)

[woahdae]

I'd love to make it more memory efficient (or accept a pull request to that
effect!) but there are some inherent tradeoffs WRT excel vs say CSV. I'll
check out the linked gem and see how it works, but to address a couple
issues:

1. xlsx is an archive format using cross-referenced xml files, and ruby
   strings aren't small. Once unarchived and represented as nokogiri nodes,
   and given ruby's memory allocation strategy that tends to double itself
   when more memory is needed, I'm not shocked a 2mb file can represent big
   in-memory data.

2. xlsx cross-references a few internal files to represent a sheet, most
   notably the "shared strings," which makes a streaming parser non-obvious in
   terms of implementation.

One win could be to switch to ox in
general, and especially to use its SAX callback api for the main sheet. I
think we'd still have to load shared strings into memory though.

On Thursday, April 28, 2016, Etienne Depaulis [email protected]
wrote:

This is exactly why we switched to this gem this morning :)

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#25 (comment)

Sent from my phone

[darkseid]

Guys, I'm having the same issue: A 96MB xlsx file (only plain data) is allocating about 20GB of RAM and then I got the same OOM error from Nokogiri: "Nokogiri::XML::XPath::SyntaxError: Memory allocation failed : growing nodeset hit limit: growing nodeset hit limit" Any advice ? I'm considering move to the good and old CSV format...