Arrow.DictEncoded conversion in, e.g. DataFrame about arrow-julia

[bkamins]

I would add dependency on PooledArrays.jl as it is lightweight and do as @dmbates suggests. Then, in particular this benefits DataFrames.jl which has a fast path for PooledVector.

[quinnj]

Hmmm, yes, we should have better copy definitions for Arrow column types. But we should also wrap Arrow.Table in Tables.CopiedColumns so DataFrames doesn't make a copy by default; the point of Arrow.jl is to very efficiently load data and be able to do DataFrames stuff on them, so we want to ensure the default avoids the copying. We should also highlight in the README/docs that the arrow format is very much immutable; you can run great analytics/queries over the data, but for mutating, it's an explicit step to convert away from the arrow formatted columns.

[bkamins]

We should have better copy definitions for Arrow column types

agreed

But we should also wrap Arrow.Table in Tables.CopiedColumns so DataFrames doesn't make a copy by default

This would be OK given the objective of the package, but as you note - we then should be very clear that this is the case. Note that in old CSV.jl 90% of issues/questions raised in DataFrames.jl were about immutability of returned columns (in the time they were immutable). And we then should add an instruction how to perform a copy somewhere in the docs.

[quinnj]

Ok, fix is up: #21