Comments (12)
Update 2021-05-21: @efucile will provide list of centers providing NWP.
@josusky and @tomkralidis to develop MVP.
from wcmp.
@tomkralidis and @josusky this is a list of GDPFS centres. There is a column saying which of them is supposed to provide data in WIS. This can be used to check which of the centres in the list is also providing data in the catalogue.
GDPFS Centers
from wcmp.
Just for the record,, the whole WIS metadata catalogue can be downloaded from https://gisc.dwd.de/oaidownload/wis-catalogue.tar.gz. I guess that can be considered and authoritative source :-)
from wcmp.
Thanks @josusky. To clarify, is this the result of DWD GISC harvest of all GISCs? How often is the .tar.gz updated? cc @jsieland
from wcmp.
To clarify, is this the result of DWD GISC harvest of all GISCs? How often is the .tar.gz updated? cc @jsieland
Yes, it's everything. At https://gisc.dwd.de/wisportal/# (see "Metadata") you can choose between all or only GISC Toulouse or only GISC Offenbach. Should be updated every night.
from wcmp.
I am pasting here part of the discussion with Tom over Slack:
When I go to https://gisc.dwd.de/wisportal/# and search for "NWP model GRIB" I get (allegedly) 110978 results. A more machine friendly approach is to use SRU:
https://sru.dwd.de/SRU2JDBC/sru?operation=searchRetrieve&version=1.1&startRecord=1&maximumRecords=5&query=title%20=%20model%20and%20abstract%20=%20NWP&stylesheet=xsl/dwd-sru.xsl&x-dwd-stylesheetDetailLevel=1
this surprisingly finds only 2985 matches (the request definitely needs tuning). Such request could be done from a script. The number of responses per requests is usually limited so we would need to repeat the request.
The other alternative is to run a script locally on top of the catalogue retrieved as the *.tar.gz. That might be faster but requires own implementation of a search algorithm.
from wcmp.
Aside: I've captured the WIS catalogue location information in the wiki for future use: https://github.com/wmo-im/wcmp/wiki/WISMetadataCatalogue#wis-10-metadata-catalogue
Please feel free to update.
from wcmp.
An initial implementation can be found in https://github.com/wmo-im/pywiscat; @josusky to review/update accordingly, at which point we'll be able to provide an analysis of NWP data.
from wcmp.
Different search result numbers occure due to differences in search query
https://sru.dwd.de/SRU2JDBC/sru?operation=searchRetrieve&version=1.1&startRecord=1&maximumRecords=5&query=title%20=%20model%20and%20abstract%20=%20NWP&stylesheet=xsl/dwd-sru.xsl&x-dwd-stylesheetDetailLevel=1
--> title=model AND abstract=NWP
Search over
https://gisc.dwd.de/
"NWP model GRIB"
--> 111 023 matches
searches over all indexed fields for "NWP" OR "model" OR "GRIB"
Search "NWP AND model AND GRIB" (but still over all indexed fields)
--> 24 957 matches
[SolR-supported operators: AND – alternative symbol: &&, NOT – alternative symbol: !, OR – alternative symbol: || [DEFAULT]]
Of interest could possibly also be our new REST API in the WIS Portal. A first version of a doc is available at
https://gisc-test.dwd.de/restapi.html
if you have any questions, we will be happy to help you.
from wcmp.
COR: /search/startSearch --> GET
from wcmp.
Hi, a first very crude test (looking only for the word "GRIB") yielded the following result (in the JSON output non-ASCII characters are encoded):
{
"NMC FRANCE - M\u00e9t\u00e9o-France": 1531,
"NMC UNITED KINGDOM - Met Office": 15069,
"ECMWF": 2755,
"GISC Tokyo - Japan Meteorological Agency": 21372,
"Max-Planck-Institut fuer Meteorologie": 46,
"Deutscher Wetterdienst": 1246,
"National Meteorological Information Center, CMA": 126,
"WMO Lead Centre for Long-Range Forecast Multi-Model Ensemble": 120,
"FSBE \u00abAviamettelecom of Roshydromet\u00bb": 2182,
"Deutsches Klimarechenzentrum": 8,
"Japan Meteorological Agency": 193,
"OSI SAF": 28,
"European Centre for Medium-Range Weather Forecasts": 7,
"Forschungszentrum Karlsruhe": 3,
"Commonwealth Scientific & Industrial Research Organisation": 4,
"University of Hohenheim": 2,
"Canadian Centre for Climate Modelling and Analysis": 2,
"NOAA": 1,
"EUMETSAT": 25,
"H SAF": 14,
"WMO/WIS/GISC Tokyo": 10,
"Deutscher Wetterdienst (RD)": 2,
"Institute for Meteorology, Freie Universit\u00e4t Berlin": 1,
"DCPC-Adriatic Marine Meteorological Centre": 8,
"ZAMG - Central Institute for Meteorology and Geodynamics": 2,
"Agenzia Regionale Preventione e Ambiente dell'Emilia-Romagna": 2,
"Deutscher Wetterdienst (ZAK)": 2,
"CNMCA (Pratica di Mare)": 2,
"NMC BULGARIA - National Institute of Meteorology and Hydrology": 3,
"University of Toulouse": 2,
"Max-Planck-Institut fuer Meteorologie (MD)": 5,
"National Institute for Environmental Studies": 6,
"Met Office Hadley Centre": 4,
"Institut f\u00fcr Meteorologie der Freien Universit\u00e4t Berlin": 3,
"Instituto Nacional de Meteorologia": 1,
"Centre National de Recherches M\u00e9t\u00e9orologiques": 2,
"Istituto Superiore per la Protezione e la Ricerca Ambientale (ex APAT)": 2,
"Met Office": 2,
"Institute of Atmospheric Sciences and Climate": 2,
"Meteo-France": 1,
"Geophysical Fluid Dynamics Laboratory/NOAA": 2,
"Environment Canada": 2,
"WMO/WIS/DCPC Tokyo (Global Producing Centre for long-range forecast)": 1,
"Federal Office of Meteorology and Climatology MeteoSwiss": 2,
"South East European Virtual Climate Change Center (SEEVCCC)": 3,
"ARPA-Servizio IdroMeteorologico": 2,
"Agenzia Regionale per la Protezione dell'Ambiente Ligure": 1,
"Helmholtz-Zentrum Geesthacht, Zentrum f\u00fcr Material- und K\u00fcstenforschung GmbH": 1,
"WMO": 1,
"DCPC Rome (RTH)": 1,
"Czech hydrometeorological institude": 1,
"NMC KENYA - Kenya Meteorological Department": 1
}
this shows several potential issues. Several centers are listed more than once, for example, "ECMWF" is obviously the same thing as "European Centre for Medium-Range Weather Forecasts". And there is granularity again. JMA published 21372 records while DWD "only" 1246, but that does not mean that JMA is doing its job by an order of magnitude better :-)
from wcmp.
@efucile , sorry for the delay. I have run an updated version of pywiscat (that groups the output by citation authority extracted from the file identifier URI) after our last teleconference but forgot to publish the result. Here it is (this time UTF-8 encoded, thus the non-ASCII characters are more readable):
"" : {
"ARPA-Servizio IdroMeteorologico" : 2,
"Agenzia Regionale Preventione e Ambiente dell'Emilia-Romagna" : 2,
"Agenzia Regionale per la Protezione dell'Ambiente Ligure" : 1,
"CNMCA (Pratica di Mare)" : 2,
"Canadian Centre for Climate Modelling and Analysis" : 2,
"Centre National de Recherches Météorologiques" : 2,
"Commonwealth Scientific & Industrial Research Organisation" : 4,
"Deutscher Wetterdienst" : 1,
"Deutscher Wetterdienst (RD)" : 2,
"Deutscher Wetterdienst (ZAK)" : 2,
"Deutsches Klimarechenzentrum" : 8,
"Environment Canada" : 2,
"Federal Office of Meteorology and Climatology MeteoSwiss" : 2,
"Forschungszentrum Karlsruhe" : 3,
"Geophysical Fluid Dynamics Laboratory/NOAA" : 2,
"Helmholtz-Zentrum Geesthacht, Zentrum für Material- und Küstenforschung GmbH" : 1,
"Institut für Meteorologie der Freien Universität Berlin" : 3,
"Institute for Meteorology, Freie Universität Berlin" : 1,
"Institute of Atmospheric Sciences and Climate" : 2,
"Instituto Nacional de Meteorologia" : 1,
"Istituto Superiore per la Protezione e la Ricerca Ambientale (ex APAT)" : 2,
"Max-Planck-Institut fuer Meteorologie" : 46,
"Max-Planck-Institut fuer Meteorologie (MD)" : 5,
"Met Office Hadley Centre" : 4,
"National Institute for Environmental Studies" : 6,
"University of Hohenheim" : 2,
"University of Toulouse" : 2,
"ZAMG - Central Institute for Meteorology and Geodynamics" : 2
},
"cn.cma.wmc-bj" : {
"National Meteorological Information Center, CMA" : 126
},
"cz.chmi.dcpc" : {
"Czech hydrometeorological institude" : 1
},
"de.dwd.gpc" : {
"Deutscher Wetterdienst" : 16
},
"fr.meteo" : {
"NMC FRANCE - Météo-France" : 369
},
"fr.meteo.dcpc-copernicus" : {
"ECMWF" : 905
},
"fr.meteo.dcpc-eer" : {
"NMC FRANCE - Météo-France" : 7
},
"fr.meteo.dcpc-lrf" : {
"NMC FRANCE - Météo-France" : 2
},
"fr.meteo.dcpc-nwp" : {
"NMC FRANCE - Météo-France" : 64
},
"hr.ammc.dcpc" : {
"DCPC-Adriatic Marine Meteorological Centre" : 8
},
"int.ecmwf" : {
"ECMWF" : 116,
"European Centre for Medium-Range Weather Forecasts" : 7
},
"int.eumetsat" : {
"ECMWF" : 2,
"EUMETSAT" : 25,
"H SAF" : 14,
"Met Office" : 2,
"Meteo-France" : 1,
"NOAA" : 1,
"OSI SAF" : 28,
"WMO" : 1
},
"int.wmo.wis" : {
"Deutscher Wetterdienst" : 1229,
"ECMWF" : 1732,
"FSBE «Aviamettelecom of Roshydromet»" : 2182,
"GISC Tokyo - Japan Meteorological Agency" : 21356,
"Japan Meteorological Agency" : 193,
"NMC BULGARIA - National Institute of Meteorology and Hydrology" : 3,
"NMC FRANCE - Météo-France" : 1089,
"NMC KENYA - Kenya Meteorological Department" : 1,
"NMC UNITED KINGDOM - Met Office" : 15069
},
"it.meteoam.dcpc" : {
"DCPC Rome (RTH)" : 1
},
"jp.go.jma.wis.dcpc-geogr" : {
"GISC Tokyo - Japan Meteorological Agency" : 16
},
"jp.go.jma.wis.dcpc-gpc" : {
"WMO/WIS/DCPC Tokyo (Global Producing Centre for long-range forecast)" : 1,
"WMO/WIS/GISC Tokyo" : 5
},
"jp.go.jma.wis.dcpc-sat" : {
"WMO/WIS/GISC Tokyo" : 1
},
"jp.go.jma.wis.dcpc-tcc" : {
"WMO/WIS/GISC Tokyo" : 4
},
"org.wmolc" : {
"WMO Lead Centre for Long-Range Forecast Multi-Model Ensemble" : 120
},
"rs.gov.hidmet" : {
"South East European Virtual Climate Change Center (SEEVCCC)" : 3
}
}
from wcmp.
Related Issues (20)
- Move KPI examples HOT 2
- Improve wording in examples paragraph of KPI 6 HOT 5
- update Part 5 of the WIS Guide HOT 17
- test/assess/provide feedback on pywcmp HOT 6
- remove status from KPIs
- KPI occurences of XPaths HOT 2
- Temporal extent with both time as "indeterminatePosition" HOT 2
- Multi-word and multi-line keywords HOT 3
- remove references to GitHub in KPIs and guide once community metadata page is implemented HOT 2
- Add Info for Metadata ISO 19115 HOT 4
- review the WIS Guide Part V and send feedback before July 9 HOT 4
- create KPI for gmd:fileIdentifier
- WIS Metadata Metrics Dashboard HOT 1
- KPI_7 results are 0/0 for all records HOT 5
- KPI_2: good title total score multiples by number of identification sections HOT 11
- Change WCMP KPI-1 Criteria HOT 1
- WIS 1.x Metadata KPI report HOT 1
- Prepare KPIs for publication into WMO-No. 1061 HOT 6
- add GitHub Action to build KPI docx and pdf in gh-pages branch HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from wcmp.