esmvalgroup / esmvaltool_sample_data Goto Github PK
View Code? Open in Web Editor NEWSamples of publicly available datasets for use with ESMValTool
License: Apache License 2.0
Samples of publicly available datasets for use with ESMValTool
License: Apache License 2.0
Here are the list of datasets I think would be nice, all from CMIP6, because it has a good license, for all models that provide the variable
Timeseries data (#4)
Map data
Profile data
To get a lot of variation, it might be nice to use different variables/experiments/frequencies for all 6 sets listed above.
I use this little snippet for testing the data in the __init__.py
file. It would be a good idea to build some basic unit tests around it.
VERBOSE = True
for mip_table in (
'Amon',
'day',
):
print()
print(f'Loading `{mip_table}`')
ts = load_timeseries_cubes(mip_table)
first_cube = ts[0]
for i, cube in enumerate(ts):
print(i)
cube.regrid(grid=first_cube, scheme=iris.analysis.Linear())
A few datasets have deviating vertical coordinates of [100000.00000001 92500.00000001]
:
./esmvaltool_sample_data/data/timeseries/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/historical/r1i1p1f1/Amon/ta/gn/v20191115
./esmvaltool_sample_data/data/timeseries/CMIP6/CMIP/CSIRO-ARCCSS/ACCESS-CM2/historical/r1i1p1f1/Amon/ta/gn/v20191108
and [100000.00000001 85000.00000001]
:
./esmvaltool_sample_data/data/timeseries/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/historical/r1i1p1f1/day/ta/gn/v20191115
./esmvaltool_sample_data/data/timeseries/CMIP6/CMIP/CSIRO-ARCCSS/ACCESS-CM2/historical/r1i1p1f1/day/ta/gn/v20191108
Addressing this issue is needed for:
ESMValGroup/ESMValCore#956
Once #13 gets merged, it would be nice to set up github actions to run the tests.
Some datasets have multiple versions, like the one below.
│ │ ├── CCCma
│ │ │ └── CanESM5
│ │ │ └── historical
│ │ │ └── r1i1p1f1
│ │ │ ├── Amon
│ │ │ │ └── ta
│ │ │ │ └── gn
│ │ │ │ ├── v20190306
│ │ │ │ └── v20190429
│ │ │ └── day
│ │ │ └── ta
│ │ │ └── gn
│ │ │ ├── v20190306
│ │ │ └── v20190429
The readme gives some hints to use this dataset in your ESMValTool projects
by adding the rootpath
to the config-user.yml
.
The instructions are unclear and incomplete. The drs specification seems to be BADC / ETHZ, but I am unable to get it working following the instructions.
One of the datasets has a very large unmasked value (5.813931e+36
), which causes issues with the multimodel statistics.
# Loading #15: esmvaltool_sample_data/data/timeseries/CMIP6/CMIP/E3SM-Project/E3SM-1-1/historical/r1i1p1f1/Amon/ta/gr/v20191211
# cube.shape=(780, 2, 2, 2) cube.data.min()=235.27495 cube.data.max()=5.813931e+36
# cube.coord("time").units.calendar='365_day'
To support testing the multimodel statistics preprocessor function with the data from this repository, the following still needs to be done
Importable data loader #3:
load_timeseries()
which returns a cubelist (use cube_helper.load
for loading the data?)esmvaltool_sample_data
Download script in #4:
.
in the paths with a /
for all subdirectories of esmvaltool_sample_data/data/timeseriesesmvaltool_sample_data/sample_data.py
)Add metadata #11 :
setup.py
Other:
download_sample_data.py
and move it to rootIt would be nice to add instructions to the readme on how to install this package.
The datasets have different lengths of the time axis, different calendars, and different horizontal axes. This makes them difficult to group and work with.
CMIP6/CMIP/NIMS-KMA/KACE-1-0-G/historical/r1i1p1f1/Amon/ta/gr/v20191028
(1980, 2, 2, 1)
, calendar: 360_day
CMIP6/CMIP/KIOST/KIOST-ESM/historical/r1i1p1f1/Amon/ta/gr1/v20191104
(1980, 2, 2, 1)
, calendar: 365_day
CMIP6/CMIP/NCAR/CESM2-WACCM/historical/r1i1p1f1/Amon/ta/gn/v20190227
(1980, 2, 3, 2)
, calendar: 365_day
CMIP6/CMIP/NASA-GISS/GISS-E2-1-H/historical/r1i1p1f1/Amon/ta/gn/v20190403
(1368, 2, 1, 1)
, calendar: 365_day
CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/historical/r1i1p1f1/Amon/ta/gn/v20191115
(780, 2, 2, 1)
, calendar: proleptic_gregorian
CMIP6/CMIP/NOAA-GFDL/GFDL-ESM4/historical/r1i1p1f1/Amon/ta/gr1/v20190726
(780, 2, 2, 2)
, calendar: 365_day
CMIP6/CMIP/CMCC/CMCC-CM2-SR5/historical/r1i1p1f1/Amon/ta/gn/v20200616
(780, 2, 3, 2)
, calendar: 365_day
CMIP6/CMIP/CCCma/CanESM5/historical/r1i1p1f1/Amon/ta/gn/v20190429
(1980, 2, 2, 1)
, calendar: 365_day
CMIP6/CMIP/BCC/BCC-CSM2-MR/historical/r1i1p1f1/Amon/ta/gn/v20181126
(1020, 2, 2, 2)
, calendar: 365_day
CMIP6/CMIP/INM/INM-CM4-8/historical/r1i1p1f1/Amon/ta/gr1/v20190605
(780, 2, 1, 2)
, calendar: 365_day
CMIP6/CMIP/NCC/NorESM2-MM/historical/r1i1p1f1/Amon/ta/gn/v20191108
(780, 2, 3, 2)
, calendar: 365_day
CMIP6/CMIP/E3SM-Project/E3SM-1-1/historical/r1i1p1f1/Amon/ta/gr/v20191211
(780, 2, 2, 2)
, calendar: 365_day
CMIP6/CMIP/CAS/FGOALS-f3-L/historical/r1i1p1f1/Amon/ta/gr/v20190927
(780, 2, 2, 2)
, calendar: 365_day
CMIP6/CMIP/NASA-GISS/GISS-E2-1-G/historical/r1i1p1f1/Amon/ta/gn/v20180827
(1368, 2, 1, 1)
, calendar: 365_day
CMIP6/CMIP/MPI-M/MPI-ESM1-2-HR/historical/r1i1p1f1/Amon/ta/gn/v20190710
(780, 2, 2, 3)
, calendar: proleptic_gregorian
CMIP6/CMIP/AS-RCEC/TaiESM1/historical/r1i1p1f1/Amon/ta/gn/v20200623
(1980, 2, 3, 2)
, calendar: 365_day
CMIP6/CMIP/INM/INM-CM5-0/historical/r1i1p1f1/Amon/ta/gr1/v20190610
(780, 2, 1, 2)
, calendar: 365_day
CMIP6/CMIP/NASA-GISS/GISS-E2-1-G-CC/historical/r1i1p1f1/Amon/ta/gn/v20190815
(1368, 2, 1, 1)
, calendar: 365_day
CMIP6/CMIP/CAS/FGOALS-g3/historical/r1i1p1f1/Amon/ta/gn/v20190818
(804, 2, 1, 2)
, calendar: 365_day
CMIP6/CMIP/NUIST/NESM3/historical/r1i1p1f1/Amon/ta/gn/v20190630
(1980, 2, 1, 2)
, calendar: gregorian
CMIP6/CMIP/CAMS/CAMS-CSM1-0/historical/r1i1p1f1/Amon/ta/gn/v20190708
(900, 2, 2, 2)
, calendar: 365_day
CMIP6/CMIP/CMCC/CMCC-CM2-HR4/historical/r1i1p1f1/Amon/ta/gn/v20200904
(780, 2, 3, 2)
, calendar: 365_day
CMIP6/CMIP/NCAR/CESM2/historical/r1i1p1f1/Amon/ta/gn/v20190308
(1980, 2, 3, 2)
, calendar: 365_day
CMIP6/CMIP/CCCR-IITM/IITM-ESM/historical/r1i1p1f1/Amon/ta/gn/v20191226
(780, 2, 1, 2)
, calendar: julian
CMIP6/CMIP/E3SM-Project/E3SM-1-0/historical/r1i1p1f1/Amon/ta/gr/v20191220
(780, 2, 2, 2)
, calendar: 365_day
CMIP6/CMIP/AWI/AWI-CM-1-1-MR/historical/r1i1p1f1/Amon/ta/gn/v20181218
(780, 2, 2, 3)
, calendar: proleptic_gregorian
CMIP6/CMIP/THU/CIESM/historical/r1i1p1f1/Amon/ta/gr/v20200417
(1980, 2, 3, 2)
, calendar: 365_day
CMIP6/CMIP/NOAA-GFDL/GFDL-CM4/historical/r1i1p1f1/Amon/ta/gr1/v20180701
(780, 2, 2, 2)
, calendar: 365_day
CMIP6/CMIP/FIO-QLNM/FIO-ESM-2-0/historical/r1i1p1f1/Amon/ta/gn/v20191204
(780, 2, 3, 2)
, calendar: 365_day
CMIP6/CMIP/CCCma/CanESM5/historical/r1i1p1f1/Amon/ta/gn/v20190306
(1980, 2, 2, 1)
, calendar: 365_day
CMIP6/CMIP/IPSL/IPSL-CM6A-LR/historical/r1i1p1f1/Amon/ta/gr/v20180803
(1980, 2, 2, 1)
, calendar: gregorian
CMIP6/CMIP/NCC/NorCPM1/historical/r1i1p1f1/Amon/ta/gn/v20200724
(1980, 2, 2, 1)
, calendar: 365_day
CMIP6/CMIP/HAMMOZ-Consortium/MPI-ESM-1-2-HAM/historical/r1i1p1f1/Amon/ta/gn/v20190627
(780, 2, 1, 2)
, calendar: proleptic_gregorian
CMIP6/CMIP/MPI-M/MPI-ESM1-2-LR/historical/r1i1p1f1/Amon/ta/gn/v20190710
(780, 2, 1, 2)
, calendar: proleptic_gregorian
CMIP6/CMIP/CAS/CAS-ESM2-0/historical/r1i1p1f1/Amon/ta/gn/v20200502
(1980, 2, 2, 2)
, calendar: 365_day
CMIP6/CMIP/MIROC/MIROC6/historical/r1i1p1f1/Amon/ta/gn/v20190311
(780, 2, 1, 2)
, calendar: gregorian
CMIP6/CMIP/MRI/MRI-ESM2-0/historical/r1i1p1f1/Amon/ta/gn/v20190308
(780, 2, 2, 2)
, calendar: proleptic_gregorian
CMIP6/CMIP/SNU/SAM0-UNICON/historical/r1i1p1f1/Amon/ta/gn/v20190323
(780, 2, 3, 2)
, calendar: 365_day
CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/Amon/ta/gn/v20190815
(780, 2, 2, 1)
, calendar: 365_day
CMIP6/CMIP/E3SM-Project/E3SM-1-1-ECA/historical/r1i1p1f1/Amon/ta/gr/v20200624
(780, 2, 2, 2)
, calendar: 365_day
CMIP6/CMIP/BCC/BCC-ESM1/historical/r1i1p1f1/Amon/ta/gn/v20181217
(1980, 2, 2, 1)
, calendar: 365_day
CMIP6/CMIP/CSIRO-ARCCSS/ACCESS-CM2/historical/r1i1p1f1/Amon/ta/gn/v20191108
(780, 2, 2, 1)
, calendar: proleptic_gregorian
CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/historical/r1i1p1f1/Amon/ta/gr/v20200310
(780, 2, 3, 3)
, calendar: proleptic_gregorian
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.