Vlen sequence of variable-length UTF-8 strings cannot be written about hsds

[mattjala]

From further testing, this error doesn't have anything to do with UTF-8 encoding. It happens when attempting to write a sequence of variable-length strings of any type, because HSDS's current schema for sequences (4 byte length following by the rest of the data) has no way to delineate non-fixed length strings within that sequence.

[jreadey]

Doesn't testPutVLenCompoundBinary in vlen_test.py have vlen and fixed types in the same compound type?

[mattjala]

Doesn't testPutVLenCompoundBinary in vlen_test.py have vlen and fixed types in the same compound type?

Yes, but that test doesn't cover a variable-length sequence of variable length strings.

That said, one of the fields in that test is an H5T_ARRAY that contains two variable-length strings, which seems like it would have run into this same issue - how does it know where the first vlen string stops and the second vlen string starts? I'll take a look at that and see if whatever is done there could apply to this case.

[jreadey]

But currently vlen data isn't supported with H5T_ARRAY - see https://github.com/HDFGroup/hsds/blob/master/hsds/util/hdf5dtype.py, line 663

[jreadey]

It might be easiest to use msgpack going forward.
I noodled around with it a bit today. See: https://gist.github.com/jreadey/acafbc048c13ed245beae22619c3ac5c

[mattjala]

But currently vlen data isn't supported with H5T_ARRAY - see https://github.com/HDFGroup/hsds/blob/master/hsds/util/hdf5dtype.py, line 663

H5T_ARRAY doesn't allow its base type to be a variable length sequence (type of class H5T_VLEN), but it is allowed to be a variable length string (string with length H5T_VARIABLE) - see

hsds/tests/integ/vlen_test.py

Line 606 in ccd63c2

"length": "H5T_VARIABLE",

[mattjala]

This behavior is now properly tested as an expected failure (since vlen sequences of vlen data aren't supported) by testPutVlenVlenError