dcs4cop / xcube-cds Goto Github PK
View Code? Open in Web Editor NEWAn xcube plugin to generate data cubes from the Climate Data Store (CDS) API
License: MIT License
An xcube plugin to generate data cubes from the Climate Data Store (CDS) API
License: MIT License
Currently, the CDS Store does not specify any chunking when opening downloaded NetCDF files. This means that the entire datasets are always loaded into memory (see open_dataset
and open_mfdataset
documentation). The store should provide some option to chunk datasets. Mirroring NetCDF chunking would be a good start, but see caveats in this discussion.
Currently, the open parameter schemas include some optional parameters which only have one allowed value -- for instance crs
in both ERA5 and Soil Moisture, which can only be WGS84
. These should be removed from the schemas to make things easier for UI generation code.
Achive full compliancy with dcs4cop/xcube#330.
For the reanalysis-era5-land-monthly-means:monthly_averaged_reanalysis_by_hour_of_day
dataset (and, probably, the similarly structuredreanalysis-era5-single-levels-monthly-means:monthly_averaged_ensemble_members_by_hour_of_day
and reanalysis-era5-single-levels-monthly-means:monthly_averaged_reanalysis_by_hour_of_day
), data are sometimes returned with the time not monotonically increasing, causing them to fail cube validation.
With the merging of xcube PR #354 (see xcube issue #352), the CDS store plugin needs to be updated accordingly to maintain compatibility.
With the recently released version 0.18.0 version of xarray installed, any ERA5 request via xcube-cds fails with the error "ValueError: cannot guess the engine, try passing one explicitly".
When building xcube in docker using default xcube 0.6.1 Dockerfile, installation of xcube-cds
plugin fails due to the xcube requirements entry in the setup.py
file.
Perhaps the setup.py
requirements should be commented out like the xcube-sh plugin
Error message when running Docker build
source activate xcube && cd xcube-cds && python setup.py develop && sed "s/- xcube/# - xcube/g" -i environment.yml && mamba env update -n xcube
running develop
running egg_info
creating xcube_cds.egg-info
writing xcube_cds.egg-info/PKG-INFO
writing dependency_links to xcube_cds.egg-info/dependency_links.txt
writing requirements to xcube_cds.egg-info/requires.txt
writing top-level names to xcube_cds.egg-info/top_level.txt
writing manifest file 'xcube_cds.egg-info/SOURCES.txt'
reading manifest file 'xcube_cds.egg-info/SOURCES.txt'
writing manifest file 'xcube_cds.egg-info/SOURCES.txt'
running build_ext
Creating /opt/conda/envs/xcube/lib/python3.8/site-packages/xcube-cds.egg-link (link to .)
Adding xcube-cds 0.6.1.dev0 to easy-install.pth file
Installed /xcube/xcube-cds
Processing dependencies for xcube-cds==0.6.1.dev0
Searching for cdsapi>=0.2.7
Reading https://pypi.org/simple/cdsapi/
Downloading https://files.pythonhosted.org/packages/d6/9e/952b99737b2dfc56229306abdd8f353b9114480db29e379ad2a621b12e6b/cdsapi-0.4.0.tar.gz#sha256=e2fc7c06c18810b2dea38522e4d41470ab91607a155df7de6a6a8bceea90b39d
Best match: cdsapi 0.4.0
Processing cdsapi-0.4.0.tar.gz
Writing /tmp/easy_install-mfu_wykp/cdsapi-0.4.0/setup.cfg
Running cdsapi-0.4.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-mfu_wykp/cdsapi-0.4.0/egg-dist-tmp-u9y3tkvv
warning: no files found matching 'LICENSE'
warning: no files found matching '*.in' under directory 'tests'
Moving cdsapi-0.4.0-py3.8.egg to /opt/conda/envs/xcube/lib/python3.8/site-packages
Adding cdsapi 0.4.0 to easy-install.pth file
Installed /opt/conda/envs/xcube/lib/python3.8/site-packages/cdsapi-0.4.0-py3.8.egg
Searching for xcube>=0.5.0
Reading https://pypi.org/simple/xcube/
Couldn't find index page for 'xcube' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.org/simple/
No local packages or working download links found for xcube>=0.5.0
error: Could not find suitable distribution for Requirement.parse('xcube>=0.5.0')
The Soil Moisture demo notebook has far better-looking plots than its ERA5 counterpart (tailored orthographic projections with superimposed grid and coastlines). These improvements should be ported back to the ERA5 notebook.
Co-ordinated releases between xcube and xcube-cds (and other plugins) are complicated by the versioning: xcube-cds 0.x typically depends on xcube 0.x, but specifying xcube >= 0.x in the environment.yml breaks the build until the xcube 0.x conda-forge package becomes available -- meaning that xcube 0.x has to be both released and conda-packaged before xcube-cds 0.x can be tested.
The xcube-cds AppVeyor config already goes part of the way to resolving this problem: it actually installs xcube from the head of the master branch. However, before doing this, it also installs its own environment -- which includes the latest (and possibly as yet non-existent) xcube version -- then removes the conda-installed xcube. It would make more sense to do the following:
This ensures that all xcube-cds's transitive dependencies through xcube are satisfied but avoids trying to install xcube itself through conda.
Example notebooks currently instantiate the data store directly. Best practice is to use xcube.core.store.new_data_store('cds')
instead. The notebooks should be updated to use this idiom.
Add new cds dataset handler to access data at https://cds.climate.copernicus.eu/cdsapp#!/dataset/satellite-sea-ice-thickness
Running the test suite produces the following warnings:
test/test_store.py: 16 warnings
/home/pont/loc/envs/xcube-latest/lib/python3.7/site-packages/jsonschema/validators.py:931: DeprecationWarning: The types argument is deprecated. Provide a type_checker to jsonschema.validators.extend instead.
validator = cls(schema, *args, **kwargs)
test/test_store.py: 39 warnings
/home/pont/loc/envs/xcube-latest/lib/python3.7/site-packages/dateutil/rrule.py:476: DeprecationWarning: Using both 'count' and 'until' is inconsistent with RFC 5545 and has been deprecated in dateutil. Future versions will raise an error.
"raise an error.", DeprecationWarning)
These should be fixed before the libraries in question upgrade them to errors.
Travis builds are no longer working due to their new pricing model, and DCS4COP is switching to AppVeyor for CI. The xcube-cds .travis.yml
configuration therefore needs to be converted into an equivalent AppVeyor configuration, and AppVeyor needs to be configured to build automatically on pushes to this repository.
When the CDS Store is used in the xcube gen UI, the default end date for the data is set to 1970-01-01 (before the start date!). This is because, for continually updating datasets, the CDS Store gives an end date of None
(= null
in JSON), with the intended semantics of ‘the present’. The gen UI, on the other hand, evidently interprets this None
as ‘the Unix epoch’.
This is unfortunately an undefined point in the API specification: the store conventions document currently makes no mention of how a None
value in time_range
should be interpreted. To fix this bug, the following steps are necessary:
None
values in time_range
(for both start and end dates), or prohibit them entirely.Micromamba now seems stable enough for everyday use in CI builds, and it doesn't need to be installed with conda. We should use this in the AppVeyor build configuration, since currently about 30% of the build time consists merely of installing miniconda, then using miniconda to install mamba!
This issue is effectively the inverse of #28.
After further discussion, it has been decided that stores should not implement geographical subsetting for datasets which can't be subsetted via the store's backend. This is the case for the CDS Store's Soil Moisture dataset, which is not subsettable in the CDS API. Subsetting will instead be carried out by the gen2 feature, so the CDS Store should revert to offering only a fixed [-180, -90, 90, 180] bbox for soil moisture. Removal of the bbox parameter itself (along with other constant-valued open parameters) is covered in Issue #36.
At present, it is thoroughly documented, but only in the readme.
Currently the version number is only provided in xcube_cds.version.version
, which is difficult for users to find. In Python, the standard location is <packagename>.__version__
. xcube_cds
should follow this standard by providing the version number in xcube_cds.__version__
.
DataDescriptor.data_vars
a Mapping[str, VariableDescriptor]
(currently it's List[VariableDescriptor]
).cds_api_url
to endpoint_url
.Methods get_type_specifiers and get_type_specifiers_for_data both return tuples of type specifiers. They are supposed to return strings.
AppVeyor builds are failing on Mac as of 2022-03-09.
When I take a look at the variable names of a dataset via the xcube-cds plugin, I get different results when using store.get_open_data_params_schema(data_id)
and
store.describe_data(data_id)
To reproduce:
from xcube.core.store import new_data_store
store = new_data_store('cds')
data_id = 'reanalysis-era5-single-levels-monthly-means:monthly_averaged_reanalysis'
store.get_open_data_params_schema(data_id)
store.describe_data(data_id)
Is this intentional? When I compare both ways of accessing data variables with xcube-sh plugin, the result is the same for both manners.
When installing xcube-cds via conda-forge I have to manually add
$ conda install -c conda-forge jdcal
afterwards in order to make xcube-cds work. This should not happen :)
For the reanalysis-era5-land-monthly-means:monthly_averaged_reanalysis_by_hour_of_day
dataset (and, probably, the similarly structuredreanalysis-era5-single-levels-monthly-means:monthly_averaged_ensemble_members_by_hour_of_day
and reanalysis-era5-single-levels-monthly-means:monthly_averaged_reanalysis_by_hour_of_day
), the data structure is more or less unchanged from what the CDS API returns. Unfortunately this is not a helpful structure for these data. Ideally, one would want each of the 24 possible "hour of day" values to be implemented as a channel -- that is, you can select any subset of x (>= 0, <= 24) hours and receive x parallel time series at monthly resolution, with the timestamps placed in the middle of each month.
Instead, what we get is a single time series with 24 hourly values on the first day of every month, representing the averages for that month for the corresponding hours. (These values aren't monotonically increasing -- see Issue #5 -- but this doesn't significantly complicate the task of restructuring the data.)
When attempting to use the CDS store via the xcube.core.store.new_data_store
method:
> cds = new_data_store('cds', endpoint_url='https://cds.climate.copernicus.eu/api/v2', cds_api_key='12345:abcd1234-1234-5678-abcd-123456789abc')
...
ValidationError: Additional properties are not allowed ('cds_api_key', 'endpoint_url' were unexpected)
...
However, these parameters do work when supplied directly to the CDSDataStore constructor.
The CDS server has recently changed the API for the soil moisture dataset: the valid version specifiers are no longer
'v201706.0.0', 'v201812.0.0', 'v201812.0.1', 'v201912.0.0'
but
'v201706', 'v201812', 'v201912', 'v202012',
This results in an error when an old-style version specifier is supplied (or when none is supplied and the default is used), and new-style version specifiers can't be used since they are checked against the old JSON Schema.
Steps to reproduce:
from xcube.core.store import new_data_store
cds = new_data_store('cds')
ds = cds.open_data(
'satellite-soil-moisture:volumetric:monthly',
time_range=['2015-01-01', '2015-12-31']
)
Result:
Exception: the request you have submitted is not valid. Value 'v201912.0.0' not valid for parameter 'version', valid values are: v201706, v201812, v201912, v202012.
The variable names have changed too.
Currently, the environment file for xcube-cds specifies cdsapi >=0.2.7
. Newer versions are available: 0.3.1 has been out since November 2020, and 0.5.1 has just been released on GitHub -- I assume that a conda-forge release will follow shortly.
It's hard to determine exactly what improvements have been made since 0.2.7, since there's no changelog and the commit messages are very brief. But updating the minimum required version to 0.5.1 makes sense nevertheless, since it reduces the chance of incompatibility with the Copernicus server and simplifies the environment solving step during installation of the conda environment.
The repository should contain a Travis (or similar) configuration for automated testing.
Let the CDS data store return an open parameters schema according to https://github.com/dcs4cop/xcube/blob/master/docs/source/storeconv.md.
Consider, for example, the following request:
generated_cube = cds_store.open_data(
'satellite-soil-moisture:volumetric:monthly',
variable_names=['volumetric_surface_soil_moisture'],
time_range=['2015-01-01', '2016-01-31']
)
This requests a modest three months at monthly resolution, but at time of writing results in the following error from the CDS API backend:
2021-04-12 13:53:46,053 ERROR Message: the request you have submitted is not valid
2021-04-12 13:53:46,054 ERROR Reason: Request too large. Requesting 17856 items, limit is 12000
This request worked as recently as 2021-02-18, and the relevant parts of the CDS Store code have not changed since then, so evidently something has changed in the CDS server's implementation of its undocumented API.
Closer examination reveals that the actual request sent by the CDS store includes the following:
'time' = ['00:00', '01:00', '02:00', '03:00', '04:00', '05:00', '06:00', '07:00', '08:00', '09:00', '10:00', '11:00', '12:00', '13:00', '14:00', '15:00', '16:00', '17:00', '18:00', '19:00', '20:00', '21:00', '22:00', '23:00'],
'day' = ['01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31']
This should not be there for a monthly average request. Presumably it was just ignored by previous versions of the CDS API. Removing the time
parameter and setting day
appropriately (according to the requested time-span and averaging period) would probably fix the bug.
PR #44, addressing dcs4cop/xcube#420 , added minimal support for the include_attrs: Sequence[str] = None
parameter in DataStore.get_data_ids()
-- that is, the parameter is accepted but ignored. The parameter should also be handled correctly, as described in dcs4cop/xcube#420 and the det_data_ids
docstring in https://github.com/dcs4cop/xcube/blob/master/xcube/core/store/store.py.
Currently, the time_coverage_start
and time_coverage_end
attributes of the output dataset are produced by the combine_netcdf_time_limits
method, which calculates them as the minimum and maximum, respectively, of the corresponding attributes in the set of NetCDF files downloaded from the CDS API. Unfortunately, both the syntax and semantics of the values returned by the CDS API appears to be inconsistent. In one soil moisture test, the time_coverage_start
is 2014-12-31T12:00:00Z
(corresponding to the start of the requested time period); in another, with different parameters, time_coverage_start
is 19910805T000000Z
-- not only is the syntax here a different dialect of ISO-8601, but the actual time represented has no relation to the requested period -- rather, it seems to be the coverage start for the entire product!.
In principle, it would not be difficult for the CDS store to recalculate the values of time_coverage_start
and time_coverage_end
directly from the data itself. The desired semantics are presumably the first of those mentioned above (i.e. covering the requested period rather than the whole product), but we should document this officially, and also decide:
Pull request #64 applies an automated fix for CVE-2007-4559, an archive unpacking vulnerability in tarfile
's extractall
method. There is not much danger in the contexts in which this is called in xcube-cds, but it would be as well to implement the changes. The PR isn't suitable for merging as-is, mainly due to code duplication. For each of the three calls to extractall
in the codebase (in request-one-var-per-file.py
, satellite_sea_ice_thickness.py
, and satellite_soil_moisture.py
), it suggests the following replacement:
def is_within_directory(directory, target):
abs_directory = os.path.abspath(directory)
abs_target = os.path.abspath(target)
prefix = os.path.commonprefix([abs_directory, abs_target])
return prefix == abs_directory
def safe_extract(tar, path=".", members=None, *, numeric_owner=False):
for member in tar.getmembers():
member_path = os.path.join(path, member.name)
if not is_within_directory(path, member_path):
raise Exception("Attempted Path Traversal in Tar File")
tar.extractall(path, members, numeric_owner=numeric_owner)
safe_extract(tf, args.output_dir)
This, or something similar, should be factored out into a utility function and called at the appropriate points.
When requesting a cds cube with a spatial subset, an Validation Error occurs for some dataset ids: ValidationError: '5 is greater than the maximum of -180'
this happens for the following dataset ids:
to reproduce:
from xcube.core.store import find_data_store_extensions
from xcube.core.store import get_data_store_params_schema
from xcube.core.store import new_data_store
store = new_data_store('cds')
dataset = store.open_data('satellite-soil-moisture:saturation:daily',
variable_names=['soil_moisture_saturation'],
bbox=[5, 35, 6, 36],
spatial_res=0.25,
time_range=['1978-11-01', '1978-11-06'])
dataset
The stacktrace:
---------------------------------------------------------------------------
ValidationError Traceback (most recent call last)
<ipython-input-6-82c0cb0543ee> in <module>
3 bbox=[5, 35, 6, 36],
4 spatial_res=0.25,
----> 5 time_range=['1978-11-01', '1978-11-06'])
6 dataset
~/Desktop/projects/xcube-cds/xcube_cds/store.py in open_data(self, data_id, opener_id, **open_params)
768 self._assert_valid_opener_id(opener_id)
769 self._validate_data_id(data_id)
--> 770 return super().open_data(data_id, **open_params)
771
772 ###########################################################################
~/Desktop/projects/xcube-cds/xcube_cds/store.py in open_data(self, data_id, **open_params)
417
418 schema = self.get_open_data_params_schema(data_id)
--> 419 schema.validate_instance(open_params)
420 handler = self._handler_registry[data_id]
421
~/Desktop/projects/xcube/xcube/util/jsonschema.py in validate_instance(self, instance)
101 format_checker=jsonschema.draft7_format_checker,
102 # We have to explicitly sanction tuples as arrays.
--> 103 types=dict(array=(list, tuple)))
104
105 def to_instance(self, value: Any) -> Any:
~/miniconda3/envs/xcube/lib/python3.7/site-packages/jsonschema/validators.py in validate(instance, schema, cls, *args, **kwargs)
932 error = exceptions.best_match(validator.iter_errors(instance))
933 if error is not None:
--> 934 raise error
935
936
ValidationError: 5 is greater than the maximum of -180
Failed validating 'maximum' in schema['properties']['bbox']['items'][0]:
{'maximum': -180, 'minimum': -180, 'type': 'number'}
On instance['bbox'][0]:
5
Currently, xcube-cds does not allow direct specification of the CDS API URL and key; the cdsapi Client class instead reads them directly from a configuration file or environment variables. It should be possible to pass the API URL and key directly to the CDSDataOpener
and CDSDataStore
constructor in cases where the configuration file and environment variable settings are not present or need to be overridden.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.