Support serial type (rowid) in sink about risingwave

[xiangjinwu]

One benefit of fixed-length 0x00012345 over 12345 is that consumer can order by it without casting it back to bigints, which can be easily forgotten. This is treating serial like an opaque type.

If that is not a concern, or we already display serial in decimal in some other places, using a consistent decimal representation could make crossing referencing much easier.

[shanicky]

I submitted a PR trying to use padded hexadecimal strings in the JSON. It looks ok.

{"_row_id":"0x05cd07777f434000","v":1}
{"_row_id":"0x05cd07758bc9c000","v":2}

[fuyufjh]

Any progress? @xxchan @shanicky

[shanicky]

Currently, we have supported int64 in protobuf format directly as a native number, but we haven't supported it in JSON and Avro formats. Because we need to use serial as a key and JSON's handling of large numbers cannot guarantee precision, we need to convert it to a string type. cc @fuyufjh Is this approach okay? Should we convert it directly to a string as "12345", or convert it to hexadecimal as "0x12345", or perhaps add a prefix like "rw-serial:12345"? Which option would have the lowest overhead for us in terms of interpretation?

Avro doesn't have this issue, so we can use native int64.

[fuyufjh]

Agree that the serial value needs to be converted to string. This because our row_id is almost 100% out of the MAX_SAFE_INTEGER so that a normal downstream JSON parser may lose precision.

Either plain string "12345" or hexadecimal "0x12345" looks good to me.