R for Data Science (2e) - 24 Arrow about slfhelper

[Moohan]

https://ursalabs.org/blog/2019-10-columnar-perf/

[Moohan]

I looked into this today and it's definitely promising. The write times (on PWB) were super slow to make a parquet file, I partitioned it by hscp2018.

Rough results

• The total space usage is about 50% greater than fst but I think that's fine. - Possibly this could be improved with better compression settings.
• The read speed of a full file is marginally slower than fst, as is the read speed for variable selections.
• The memory usage is absolutely tiny compared to fst on everything I checked.
• The huge advantage comes when filtering on things which we filtered, so reading in a single partnerships data was much faster than with fst. The trick here will be to pick the partitions well so that we capture most use cases!
• The other big advantage is we could return an apache dataset, where the user could do more manipulation using dplyr before finally calling collect(), we could make this optional and if the user doesn't want it, just do the collect() behind the scenes.

[Moohan]

Public-Health-Scotland/R-Resources/blob/master/posit_workbench_writing_and_reading_data.md Also relevant - apparently the parquet files should read about as fast as fst (ignoring the other benefits) but will write much faster as for some reason writing fst on PWB is super slow.