`ds/rows` produces something vector-of-maps-like that transit cannot handle about tech.ml.dataset

[cnuernber]

Yes I have thought that on and off in the past - its time...

[harold]

Interestingly ds/mapseq-reader is in the same spot, presently:

user> (let [baos (java.io.ByteArrayOutputStream.)]
        (transit/write (transit/writer baos :json) (ds/mapseq-reader ds))
        (.toString baos))
Execution error (NullPointerException) at com.cognitect.transit.impl.AbstractEmitter/marshalTop (AbstractEmitter.java:203).
null

[cnuernber]

I really don't think this is worth fixing honestly. We have better ways of serializing the dataset; it has transit bindings specifically.

[harold]

Oh, is there a better way? What does it look like?

[cnuernber]

I think this file is in the wrong place but this is the actual transit bindings for tmd.

[harold]

Trippy!

user> (require '[tech.v3.libs.transit :as ds-transit])
nil
user> (require '[tech.v3.dataset :as ds])
nil
user> (def ds (ds/->dataset {:a [1 2 3]}))
#'user/ds
user> ds
_unnamed [3 1]:

| :a |
|---:|
|  1 |
|  2 |
|  3 |
user> (ds-transit/dataset->transit-str ds)
"[\"~#tech.v3.dataset\",[\"^ \",\"~:metadata\",[\"^ \",\"~:name\",\"_unnamed\"],\"~:flavor\",\"~:transit\",\"~:version\",1,\"~:columns\",[[\"^ \",\"^1\",[\"^ \",\"^2\",\"~:a\",\"~:datatype\",\"~:int64\",\"~:n-elems\",3],\"~:missing\",[],\"~:data\",\"AQAAAAAAAAACAAAAAAAAAAMAAAAAAAAA\"]]]]"
user> (charred.api/read-json *1)
["~#tech.v3.dataset"
 ["^ "
  "~:metadata"
  ["^ " "~:name" "_unnamed"]
  "~:flavor"
  "~:transit"
  "~:version"
  1
  "~:columns"
  [["^ "
    "^1"
    ["^ " "^2" "~:a" "~:datatype" "~:int64" "~:n-elems" 3]
    "~:missing"
    []
    "~:data"
    "AQAAAAAAAAACAAAAAAAAAAMAAAAAAAAA"]]]]
user> (def t (ds-transit/dataset->transit-str ds))
#'user/t
user> (ds-transit/transit-str->dataset t)
_unnamed [3 1]:

| :a |
|---:|
|  1 |
|  2 |
|  3 |

Maybe tech.v3.libs.transit should be promoted to be in the tech.ml.dataset jar?

What do you think?

[harold]

He's 🔥 on fire!