Speed improvements to loading HDF5 trees about h5pyd

[jreadey]

I've added some caching logic to the group class. Try out this latest checkin: 1999417.

This is not a single operation recursive load, but I saw a speed up of about ~4x speed up walking the tree for the sample Nexus file. This is with using the hsls.py script in the app directory.

[jreadey]

@rayosborn - did you get a chance to try this out?

[rayosborn]

I have tested it, but I wasn't sure of the previous speeds because I forgot to do a proper timing before upgrading. I need to revert to the old version. However, I don't think I saw a factor four. It might have been a factor of two.

[jreadey]

There will be some variability based on the latency between client and server. My testing was with a server running on the same LAN. Also, the test driver is different.

Did the NexPy GUI need a lot of mods to work with h5serv? I could set it up in my environment.

[rayosborn]

I haven't made any changes to the NeXpy GUI yet. In the latest development version on my own clone of the nexusformat API, the nxremote branch has an added file, which subclasses the NXFile class for remote access. I was thinking of pushing this version to PyPI, since it is a test feature that only users with h5pyd would even be able to access. I'll let you know when I've done that.

[jreadey]

If you push the branch to github, I can just grab from there.

How would I use it to list the contents of a Nexus file?

[rayosborn]

The nxremote branch has been published on my Github. You can load a file by typing:

>>> a=nxloadremote(filepath, domain='exfac.org', server='some.server:5000')
>>> print(a.tree)

The file path is the path relative to the data directory. The module converts that to a domain name. The top domain is currently 'exfac.org' to match the test repository.

[jreadey]

@rayosborn - some updates on this old issue...
By default h5pyd.File(filepath) will return all the meta data for the domain in the request response. H5pyd caches this, so any attribute read or link access won't need to talk to the server. There's a limit on the number of objects fetched on the server of 500. This is so the GET request doesn't take an inordinate amount of time for domains with lots of attributes and/or links.

To compare the performance not using the prefetch, you can use: h5pyd.File(filepath, use_cache=False). This will return just information on the root group.

[rayosborn]

Thanks, @jreadey. I can't look into this for a couple of weeks, but I plan to soon.