Arrow, writing nested types. about tech.ml.dataset

[cnuernber]

Definitely interested in contributions here - we have read-only support for a subset of this - lists I believe. Let me know if you want to discuss details of the design or anything else. If not and you are fine just researching and implementing that works for us too.

[cnuernber]

@archaic - is this still an issue for you? I would rather see this contribution in tmducken but arrow also makes sense.

[archaic]

I wasn't able to come up with a good solution. I would love to have this functionality, as most of the datasets I work with are annoying enough to have small segments of nested data that are difficult to wrangle into columns. I think it is a difficult problem to handle generically - does the schema get inferred, that is hard in itself?

I ended up falling back to using metosin/malli (for columns with complex schemas) to define schema's for columns, then the raw java arrow library to convert the malli schema's to arrow datatypes and column writers, however this seems like a regression to just be able to use a dataset and arrow/write! automatically.