Some tricky data for Arrow about arrow-julia

[quinnj]

It's helpful to post backtraces as that can often quickly show where the error is

[xiaodaigh]

It prints out alot of stuff. It's like 11pm. I will try to get a MWE together.

[quinnj]

Alright, so the error here is because it doesn't know how to map Symbol or Char columns to arrow format. I guess in both cases, we should just map them to strings.

[quinnj]

I guess the biggest question is if we're ok not having round-tripping in these cases; i.e. we could allow writing columns of Symbol/Char, but they'll be read back in as strings. @bkamins @nalimilan , any thoughts/preferences here?

[nalimilan]

Not sure. Maybe see what Feather.jl does? Also of interest is what RCall should do with CategoricalArray{T} when !(T<:AbstractString), as R only supports string factors (JuliaInterop/RCall.jl#331).

[bkamins]

I think the priority is to allow saving things with as little problems as possible. Symbol and Char are normal in Julia.

Given your comment I understand that we have a choice:

1. either error on them
2. or convert to String when saving

I would prefer the option 2. maybe with a @warn about conversion

[quinnj]

I mean, Feather.jl will just be a wrapper around Arrow.jl pretty soon, so they'll do whatever we do. I also think I like the idea of option 2 w/ a warning. I think people should understand that Arrow.jl isn't meant to be a serialization package, even though it can handle a wide range of types of objects. But even still, I think doing some automatic conversions will be convenient.

[bkamins]

Agreed

[xiaodaigh]

There is a way to store arbitrary metadata right? We should remember them as Julia:Symbol and Julia:Char types. So in Python and r, they get read as String but in Julia we can use the metadata to restore them as Symbol and Char?

[quinnj]

Ah, I forgot about that; yes, that's a pretty good idea. I'll have to see how we can work the plumbing in for that.

[quinnj]

Ok, took a bit of work, but I think I've got it all working for Symbol/Char. #11