Comments (6)
BTW, thanks to @ocefpaf for helping me figure this out!
from h5pyd.
What do you think that pythonic way would be to switch the dependent module?
E.g. something like this:
if "USE_H5PYD" in os.environ and os.environ["USE_H5PYD"]:
import h5pyd
else:
import h5py
Would that mess up packaging?
We could create an entirely new module (h5netcdfd?), but would then need to sync any changes from h5netcdf regularly.
from h5pyd.
The h5py module is actually only explicitly used a handful of times inside h5netcdf: https://github.com/shoyer/h5netcdf/blob/master/h5netcdf/core.py
Most of the time, we use method calls on existing h5py.File
objects.
What do you think that pythonic way would be to switch the dependent module?
Depending how much the interface needs to be changed to accommodate h5pyd, the cleanest way to do this is probably to add constructor arguments to the handful of h5netcdf functions/classes that open a file.
For example, we might change h5netcdf.File
like so:
class File(Group):
def __init__(self, path, mode='a', backend=h5py, **kwargs):
self._h5file = backend.File(path, mode, **kwargs)
Then using h5pyd behind the scenes as a user is as simple as h5netcdf.File(..., backend=h5pyd)
. If modules are not a complete drop-in equivalent, then at least we could accept string names like backend='h5pyd'
.
I certainly would be very happy to accept patches to add this flexibility in h5netcdf.
Generally checking environment variables for this sort of thing is discouraged, since it makes it hard to switch between options in user code (there are certainly legitimate cases for using both h5py and h5pyd at the same time).
from h5pyd.
@shoyer , this sounds great. I'd submit a PR right now except we should first wait for the shared dimensions to be working, #32, right @jreadey?
from h5pyd.
@ajelenak-thg and @jreadey,
Just to record this somewhere, here's what I did to get a custom environment for the ESIP Winter Meeting (Jan 9-11, 2017), with xarray
working with h5pyd
:
With this h5pyd_env.yml
:
name: h5pyd
channels:
- conda-forge
- defaults
dependencies:
- python=3.6
- h5py
- nb_conda_kernels
- pytz
- requests
- matplotlib
- pip:
- git+https://github.com/HDFGroup/h5pyd.git@master
I did:
conda env create -f h5pyd_env.yml
source activate h5pyd
conda install xarray
conda remove h5netcdf
pip install --no-deps --upgrade git+https://github.com/ajelenak-thg/h5netcdf.git@h5pyd
conda install --no-deps xarray
from h5pyd.
Xarray is now working nicely with HSDS: https://gist.github.com/rsignell-usgs/cc2d2d4fe1930bd949119e543b56bce1
Closing this issue, while Dask tasks remain: pangeo-data/pangeo#75 (comment)
from h5pyd.
Related Issues (20)
- hsload fails with datasets using scale offset filter HOT 1
- apply source compression filter in hsload HOT 1
- Show filters applied to any datasets in hsls HOT 1
- h5pyd not evaluating environment variables HOT 1
- An error related to hsrm HOT 12
- Error with 1D chunk sizes HOT 2
- `logging.info` call forces downstream package loggers to emit messages twice
- Recursive Domain Deletion Flag for `hsrm`
- git tags missing HOT 1
- hsload doesn't allow linkpath and fastlink options to be used together
- Update build process to use toml HOT 1
- CI Testing in Github HOT 1
- Support numpy-style broadcasting
- Support field selection from compound types HOT 1
- Support ordering links/attribute by creation index/name HOT 1
- Attributes in root group not displayed by hsls --showattrs HOT 1
- Support for H5Dread_multi/H5Dwrite_multi
- Missing v0.18.0 tag HOT 2
- PEP 632: Python 3.12 Incompatible with Distutils HOT 1
- Failures during vlen writes in test_dataset
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from h5pyd.