sscdf,jim22k

sscdf's People

Contributors

Stargazers

Watchers

sscdf's Issues

zarr support

Hey @jim22k!

This is more of a comment than an actual issue or request, but just letting you know you almost have zarr support already.

The netcdf4 C library has a zarr implementation (which they are interested in splitting out). This library can write to that trivially. It reads fine as a zarr store, but I get a segmentation fault if I try to read it with netcdf4...

I did have to remove the zlib kwarg from this line to write it:

sscdf/sscdf.py

Line 218 in 14167c8

var = ds.createVariable(key, dtype, (key,), zlib=True)

Example:

import sscdf
import netCDF4 as nc
import graphblas as gb
import zarr
from scipy import sparse

X = gb.io.from_scipy_sparse(sparse.random(1000, 1000, density=0.1, format='csr'))
ds = nc.Dataset("file:///Users/isaac/tmp/test_zarr.nc#mode=nczarr,file", "w")

sscdf.Writer._save_tensor(ds, X)

z_store = zarr.open("/Users/isaac/tmp/test_zarr.nc")

z_store.tree()

/
 ├── col_indices (100000,) uint64
 ├── indptr (1001,) uint64
 ├── ncols (1,) uint64
 ├── nrows (1,) uint64
 └── values (100000,) float64

dict(z_store.attrs)

{'_NCProperties': 'version=2,netcdf=4.8.1,nczarr=2.0.0',
 'format': 'csr',
 'datatype': 'fp64',
 '_NCZARR_ATTR': {'types': {'_NCProperties': '<U1',
   'format': '<U1',
   'datatype': '<U1'}}}

But it segfaults if I try to read it with netcdf. I have not tried to debug this.

sscdf.read("file:///Users/isaac/tmp/test_zarr.nc#mode=nczarr,file")

[1]    33167 segmentation fault  ipython
/Users/isaac/miniconda3/envs/sscdf/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown                         
  warnings.warn('resource_tracker: There appear to be %d '

Recommend Projects