md5 checksum mismatch for identical files/data about h5netcdf

[kmuehlbauer]

Thanks @abunimeh for raising this.

Can you, by chance, create a MCVE with h5netcdf only?

[abunimeh]

MCVE

import h5netcdf
from time import sleep

with h5netcdf.File("sample1.nc", "w") as f1:
    f1.dimensions = {"x": 5}
    v = f1.create_variable("hello", ("x",), float)
    v[:] = [1, 1, 1, 1, 1]

# comment out this line and the md5 checksum will match
sleep(5)

with h5netcdf.File("sample2.nc", "w") as f2:
    f2.dimensions = {"x": 5}
    v = f2.create_variable("hello", ("x",), float)
    v[:] = [1, 1, 1, 1, 1]

without sleep() md5 checksum matches

md5 sample1.nc sample2.nc                                             
MD5 (sample1.nc) = 5a8ece3bce095ae7d651936306e21351
MD5 (sample2.nc) = 5a8ece3bce095ae7d651936306e21351

it looks like data/time of creation is embedded in the file which is causing the file to have different checksums. It would be nice to prevent embedding such info inside the file that way the file checksum is always the same regardless of when it was created.

with sleep() md5 checksum doesn't match

md5 sample1.nc sample2.nc 
MD5 (sample1.nc) = eeeb814c19f9efbaeb1f062ac89bab0f
MD5 (sample2.nc) = 9f1703c5ef4bf27d4c10b93e724b35db

[kmuehlbauer]

Thanks @abunimeh for providing that minimal example.

I'll have a look if this is something h5netcdf can control.

[kmuehlbauer]

One thing you can try is setting track_order=False when opening.

It might have to do how hdf5 is keeping track of creation order. Not sure if we can do anything about it.

[abunimeh]

track_order=False fixes it

import h5netcdf
from time import sleep
import hashlib


track_order = False

fname1 = "./sample1.nc"
fname2 = "./sample2.nc"

with h5netcdf.File(fname1, "w", track_order=False) as f1:
    f1.dimensions = {"x": 5}
    v = f1.create_variable("hello", ("x",), float)
    v[:] = [1, 1, 1, 1, 1]

# comment out this line and the md5 checksum will match
sleep(5)

with h5netcdf.File(fname2, "w", track_order=False) as f2:
    f2.dimensions = {"x": 5}
    v = f2.create_variable("hello", ("x",), float)
    v[:] = [1, 1, 1, 1, 1]

s1hash = hashlib.md5(open(fname1, "rb").read()).hexdigest()
s2hash = hashlib.md5(open(fname2, "rb").read()).hexdigest()

assert s1hash == s2hash

[kmuehlbauer]

Great this works. But that comes with a cost. If you want to use it in your original use case, to create netcdf-files, then this might create issues down the road. By design netcdf-c and netcdf4-python use track_order=True to keep track of object creation order.

[abunimeh]

Thanks!