Support zipped csv files in `csv_collection` about maintenance_tasks

[adrianna-chang-shopify]

👋 Hey @samy-at-shopify ! Supporting compressed CSVs sounds like a simple solution to the large file upload problem, which we've definitely encountered before 🤔 (e.g. #600). I think we may want to look more into direct file uploads at one point, but I still think supporting compressed file formats makes sense.

cc @etiennebarrie -- any opinion here?
@samy-at-shopify are you interested in trying to put together a PR for this?

[samy-at-shopify]

Hey @adrianna-chang-shopify 👋
Happy to take this on! I'm not sure how much time I'll be able to dedicate to this though, so don't worry about hot-potatoing this to someone else if the need to support compressed files becomes urgent 👍

[samy-at-shopify]

Thought about this for a little more, and this feature might not be as easy to implement as I first thought.

These lines handle reading the contents of the .csv input file. They do so by reading the entire file and representing its contents as a string in memory.

This clashes a bit with the intention for supporting zipped .csv files (allowing someone to use a very large .csv file as input; e.g., > 1GB): the gem currently loads the entire file rather than reading and processing lines one-by-one.

[samy-at-shopify]

Ruby does have the IO.foreach method that would allow the gem to process a .csv file line-by-line without putting it all in memory. I'll see if I can get a version of the gem that works using this lazy approach

[etiennebarrie]

No strong opinions on that, but I'd rather avoid the additional dependency it will require (rubyzip I assume). So maybe we should make the feature optional on the dependency being present, and otherwise not allow zip files to be uploaded?

I wish we could let the browser/frontend web server do the work of compressing/uncompressing, but quickly looking I couldn't find anything to do that.

Regarding the fact that we download everything at once, it's because while ActionStorage lets you download in chunk and StringIO lets you turn a String into an IO, there's nothing in Ruby that would combine the two, an IO that would go yield to a block when it needs more data (which could be used to download the next chunk with ActiveStorage). It would be neat to have that, but that wouldn't solve the large file upload issue.

[github-actions]

This issue has been marked as stale because it has not been commented on in two months.
Please reply in order to keep the issue open. Otherwise, it will close in 14 days.
Thank you for contributing!