Comments (12)
Is this still active? I do get the same error:
Exception ignored in: <function CachingFileManager.__del__ at 0x7f8625ebcc80>
Traceback (most recent call last):
File "HOME/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/xarray/backends/file_manager.py", line 243, in __del__
File "HOME/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/xarray/backends/file_manager.py", line 221, in close
File "HOME/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/h5netcdf/core.py", line 701, in close
File "HOME/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/h5py/_hl/files.py", line 431, in close
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5f.pyx", line 267, in h5py.h5f.get_obj_ids
File "h5py/h5i.pyx", line 43, in h5py.h5i.wrap_identifier
ImportError: sys.meta_path is None, Python is likely shutting down
Using
lay_dat = xr.open_mfdataset(files.values(), engine='h5netcdf', concat_dim=Dim.it, combine='nested')
A within a function always raises the error, using result = xr.open_dataset(output, engine='h5netcdf')
this error only occurs if I do not manually invoke result.close()
.
I have to my knowledge the most recent versions installed:
INSTALLED VERSIONS
------------------
commit: None
python: 3.7.3 (default, Mar 27 2019, 22:11:17)
[GCC 7.3.0]
python-bits: 64
OS: Linux
OS-release: 3.16.0-10-amd64
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.4
libnetcdf: None
xarray: 0.14.1
pandas: 0.25.0
numpy: 1.16.4
scipy: 1.3.0
netCDF4: None
pydap: None
h5netcdf: 0.7.4
h5py: 2.10.0
Nio: None
zarr: None
cftime: None
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2.3.0
distributed: 2.3.0
matplotlib: 3.0.3
cartopy: None
seaborn: None
numbagg: None
setuptools: 41.0.1
pip: 19.1.1
conda: 4.7.11
pytest: 5.0.1
IPython: 7.5.0
sphinx: 2.2.1
from h5netcdf.
I don't have the original file anymore and I cannot reproduce either.
Thank you for closing this issue.
from h5netcdf.
xr.show_versions()
INSTALLED VERSIONS
commit: None
python: 3.6.7.final.0
python-bits: 64
OS: Linux
OS-release: 4.9.35-server-1.mga6
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: fr_FR.UTF-8
LOCALE: fr_FR.UTF-8
xarray: 0.11.0
pandas: 0.23.4
numpy: 1.15.4
scipy: 1.1.0
netCDF4: 1.4.2
h5netcdf: 0.6.2
h5py: 2.9.0
Nio: None
zarr: None
cftime: 1.0.2.1
PseudonetCDF: None
rasterio: None
iris: None
bottleneck: 1.2.1
cyordereddict: None
dask: 1.0.0
distributed: 1.25.0
matplotlib: 3.0.2
cartopy: 0.17.0
seaborn: 0.9.0
setuptools: 40.6.2
pip: 18.1
conda: None
pytest: 3.10.1
IPython: 7.2.0
sphinx: 1.8.2
from h5netcdf.
Could you share a fully reproducible example, including a copy of the netcdf file or code to create it?
from h5netcdf.
I would also suggest trying out xarray 0.11.2, which includes some fixes related to this functionality.
from h5netcdf.
I now think this is a probably a bug on the xarray side. We should be automatically cleaning-up unclosed files using atexit rather than only relying on __del__
: https://stackoverflow.com/questions/14986568/python-how-to-ensure-that-del-method-of-an-object-is-called-before-the-mod
If you could share a script that reproduces the bug that would be very helpful for testing. Please file an issue in the xarray tracker and ping me. Thanks!
from h5netcdf.
Thanks for the info, I will try to come up with a minimal script this weekend and open a pull request.
I see. Using __del__
is indeed no reasonable way to clean up. Before switching to xarray
I had a hacky solution to automatically close my files, which I happily appended and deferred the problem to xarray
. I patch something together using weakref.finalize
, which sounded to me like the right solution. weakref.finalize
is called when the object is garbage collected or atexit
.
I used a wrapper around the file object, having the following method:
def _autoclean_load(self, *args, **kwds):
data = np.load(*args, **kwds)
finalize(self, data.close)
return data
If the wrapper object is collected or the script ends in a way, such that atexit
is applied, data.close
is called. If I remember correctly you need the wrapper class, as finalize
contains the reference to data.close
, thus data would only be collect on exit.
See my question on stack overflow
from h5netcdf.
It sounds like you may have looked into this in more detail than I have!
Take a look here for the logic in xarray. Pull requests to improve this would be very welcome!
from h5netcdf.
Hey, I'm still running into this issue using xarray. My versions are xarray 0.15.0 and h5netcdf 0.7.4.
It seems that adding dataset.close()
doesn't fix the issue unless you also load()
/compute()
the dataset. My process is open_mfdataset
followed by close()
, but I still get a long list of those h5py messages (one for each file opened by open_mfdataset
).
@shoyer did this issue ever make it to the xarray issue tracker? I couldn't find it by searching. If it has could you reference that issue here so others who stumble upon it here can follow along there? This is the first issue that comes up when I google that ImportError: sys.meta_path is None, Python is likely shutting down
error message.
from h5netcdf.
I don't think has made it onto the xarray issue tracker yet. It would still be helpful to have a full script that reproduces the behavior for debugging potential fixes.
from h5netcdf.
@shoyer Will this be resolved with XARRAY_BACKEND_API = v2
? If we had a MWE, we could add tests for this.
from h5netcdf.
@floriankrb I can't reproduce this with h5netcdf
version 0.12.0
. If you still encounter this issue, please provide a minimal working example showing this behaviour. We can reopen then. Thanks.
from h5netcdf.
Related Issues (20)
- Corrupted headers when serialising using dask.distributed client HOT 9
- Test failures with NetCDF 4.9.0 HOT 6
- Remove h5py2 related code and CI builds HOT 3
- FAILED h5netcdf/tests/test_h5netcdf.py::test_group_names HOT 6
- AttributeError for '_phony_dim_count' when trying to convert a file made with h5py HOT 12
- Tests test_more_than_7_attr_creation_track_order and test_bool_slicing_length_one_dim fail in the test suite HOT 18
- very slow partial reading when saved with index shift HOT 10
- h5py minimum version update? HOT 5
- Improving performance for h5netcdf HOT 7
- Documentation request: Alternative way to obtain h5netcdf HOT 1
- Segmentation fault after upgrading to h5netcdf==1.1.0 HOT 14
- ValueError raised when attribute has type `h5py.Reference` HOT 5
- support for HDF5 dimension scales with null dataspace HOT 2
- Modifying attributes safely is not possible with all datasets. HOT 10
- Better Error for illegal variable names HOT 3
- md5 checksum mismatch for identical files/data HOT 7
- Question: does `h5netcdf` bring in the entire data from a netCDF file on a remote disk (like S3)? HOT 5
- Provide an example with time dimension readable by paraview HOT 3
- transient vs committed types HOT 5
- string decoding produces pairs of low surrogates HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from h5netcdf.