ValueError raised when attribute has type `h5py.Reference` about h5netcdf

[kmuehlbauer]

@fsvenson Thanks for raising this. Could you share the full error traceback please?

[fsvenson]

It happened when reading via xarray:

Traceback (most recent call last):
  File "/home/efes/dev/issues/DATA-375-3d-vis/test.py", line 38, in <module>
    bm = xr.open_dataset(
  File "/home/efes/.cache/pypoetry/virtualenvs/meta-world-v48m9Vhg-py3.10/lib/python3.10/site-packages/xarray/backends/api.py", line 540, in open_dataset
    backend_ds = backend.open_dataset(
  File "/home/efes/.cache/pypoetry/virtualenvs/meta-world-v48m9Vhg-py3.10/lib/python3.10/site-packages/xarray/backends/h5netcdf_.py", line 419, in open_dataset
    ds = store_entrypoint.open_dataset(
  File "/home/efes/.cache/pypoetry/virtualenvs/meta-world-v48m9Vhg-py3.10/lib/python3.10/site-packages/xarray/backends/store.py", line 28, in open_dataset
    vars, attrs = store.load()
  File "/home/efes/.cache/pypoetry/virtualenvs/meta-world-v48m9Vhg-py3.10/lib/python3.10/site-packages/xarray/backends/common.py", line 128, in load
    (_decode_variable_name(k), v) for k, v in self.get_variables().items()
  File "/home/efes/.cache/pypoetry/virtualenvs/meta-world-v48m9Vhg-py3.10/lib/python3.10/site-packages/xarray/backends/h5netcdf_.py", line 221, in get_variables
    return FrozenDict(
  File "/home/efes/.cache/pypoetry/virtualenvs/meta-world-v48m9Vhg-py3.10/lib/python3.10/site-packages/xarray/core/utils.py", line 469, in FrozenDict
    return Frozen(dict(*args, **kwargs))
  File "/home/efes/.cache/pypoetry/virtualenvs/meta-world-v48m9Vhg-py3.10/lib/python3.10/site-packages/xarray/backends/h5netcdf_.py", line 222, in <genexpr>
    (k, self.open_store_variable(k, v)) for k, v in self.ds.variables.items()
  File "/home/efes/.cache/pypoetry/virtualenvs/meta-world-v48m9Vhg-py3.10/lib/python3.10/site-packages/xarray/backends/h5netcdf_.py", line 188, in open_store_variable
    attrs = _read_attributes(var)
  File "/home/efes/.cache/pypoetry/virtualenvs/meta-world-v48m9Vhg-py3.10/lib/python3.10/site-packages/xarray/backends/h5netcdf_.py", line 65, in _read_attributes
    for k, v in h5netcdf_var.attrs.items():
  File "/usr/lib/python3.10/_collections_abc.py", line 906, in __iter__
    yield (key, self._mapping[key])
  File "/home/efes/.cache/pypoetry/virtualenvs/meta-world-v48m9Vhg-py3.10/lib/python3.10/site-packages/h5netcdf/attrs.py", line 72, in __getitem__
    if not np.isscalar(output) and len(output) == 1:
TypeError: object of type 'h5py.h5r.Reference' has no len()

[kmuehlbauer]

Thanks @fsvenson, that's helpful. So if you just want to get back the reference we could do this as suggested. I've no idea about the implications this might have downstream. What does netcdf4 engine return in this situation?

[kmuehlbauer]

I looks like netcdf4-python is just ignoring attributes with type reference.

In the past h5netcdf added these features which are unsupported in netcdf4/netcdf-c as invalid_netcdf.

Looking at

h5netcdf/h5netcdf/core.py

Lines 1127 to 1148 in 18c86f4

	def _check_valid_netcdf_dtype(self, dtype):
	dtype = np.dtype(dtype)
	if dtype == bool:
	description = "boolean"
	elif dtype == complex:
	description = "complex"
	elif h5py.check_dtype(enum=dtype) is not None:
	description = "enum"
	elif h5py.check_dtype(ref=dtype) is not None:
	description = "reference"
	elif h5py.check_dtype(vlen=dtype) not in {None, str, bytes}:
	description = "non-string variable length"
	else:
	description = None
	if description is not None:
	_invalid_netcdf_feature(
	"{} dtypes".format(description),
	self.invalid_netcdf,
	)

dtype="reference" is already checked. So we can already write references with kwarg invalid_netcdf=True. This is most likely a regression which need to be taken care of in __getitem__ as you already suggested.

We need to check if that collides with the HIDDEN attributes somehow.

[kmuehlbauer]

@fsvenson Sorry, this fell through the cracks. Are you still in the works with this? Would you be able to create a MCVE producing such file (eg via h5py) or provide a small file to play with? Thanks!