Vanity package for geopython projects
pip install geopython
>>> import geopython
pygeometa is a Python package to generate metadata for geospatial datasets
Home Page: https://geopython.github.io/pygeometa
License: Other
Vanity package for geopython projects
pip install geopython
>>> import geopython
As discussed offline with @tomkralidis , we could eventually create a website for pygeometa (for docs, news via blog, etc). Could be hosted on http://geopython.github.io/pygeometa and leverage Jekyll.
More discussions required before proceeding.
Attempting to install pygeometa from source using pip fails, seemingly because setup.py imports 'pygeometa'.. the package which is to be built.
(test1)bjv@host:~$ pip install git+https://github.com/geopython/pygeometa.git#egg=pygeometa
You are using pip version 7.0.3, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pygeometa from git+https://github.com/geopython/pygeometa.git#egg=pygeometa
Cloning https://github.com/geopython/pygeometa.git to /tmp/pip-build-sj02i_4c/pygeometa
p11-kit: invalid config filename, will be ignored in the future: /etc/pkcs11/modules/gnome-keyring-module
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 20, in <module>
File "/tmp/pip-build-sj02i_4c/pygeometa/setup.py", line 53, in <module>
import pygeometa
File "/tmp/pip-build-sj02i_4c/pygeometa/pygeometa/__init__.py", line 52, in <module>
from jinja2 import Environment, FileSystemLoader
ImportError: No module named 'jinja2'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-sj02i_4c/pygeometa
Our local variable for french in the yaml definition are "fr", but only "fra" is supported in ECDC. When generating the xml, some keywords are created with:
<gmd:LocalisedCharacterString locale="#fr"
<gmd:LocalisedCharacterString locale="#fra"
The result is that in ECDC, those keywords are not translated in french...
I suggest we modify a little bit https://github.com/geopython/pygeometa/blob/master/pygeometa/templates/iso19139-hnap/main.j2#L202-205 .
to:
{% else %}
{% for kw1, kw2 in zip(keywords[0], keywords[1]) %}
{% if record['metadata']['language_alternate'] == "fr" %}
{% set language_alternate = "fra" %}
{% endif %}
{{ cs.get_freetext('keyword', language_alternate, [kw1, kw2]) }}
{% endfor %}
{% endif %}
I think this is the easier solution.
Other solution:
thoughts @tomkralidis @josegar74 ?
In the output XML, for the HNAP profile, the French locale must be #fra instead of #locale-fr
Thanks for the fix @tomkralidis -- Alex
HNAP2.3 has mandatory requirements for 'Corporate information'. There are 3 parameters: Branch, Directorate and Project. The first two are mandatory and 'Project' is optional.
The ECDC provides a dropdown list for selecting Branch and Directorate, along with a freetext box for Project (in both language). In the pygeometa spirit, I suggest specifying 'Corporate info' directly in the MCF file. Parameter names could be HNAP_Branch, HNAP_Directorate and HNAP_Project-en / HNAP_Project-fr.
Here's an example of an output XML which has the 'Corporate info' in it. This section comes right after /gmd:supplementalInformation
<napec:EC_CorporateInfo>
<napec:EC_Branch>
<napec:EC_Branch_TypeCode codeListValue="meteorological_service_canada" codeList="http://www.ec.gc.ca/data_donnees/standards/schemas/napec#EC_Branch"/>
</napec:EC_Branch>
<napec:EC_Directorate>
<napec:EC_Directorate_TypeCode codeListValue="weather_environmental_operations" codeList="http://www.ec.gc.ca/data_donnees/standards/schemas/napec#EC_Directorate"/>
</napec:EC_Directorate>
<napec:EC_Project xsi:type="gmd:PT_FreeText_PropertyType">
<gco:CharacterString>Test project in English</gco:CharacterString>
<gmd:PT_FreeText>
<gmd:textGroup>
<gmd:LocalisedCharacterString locale="#fra">Projet test en français</gmd:LocalisedCharacterString>
</gmd:textGroup>
</gmd:PT_FreeText>
</napec:EC_Project>
<napec:GC_Security_Classification>
<napec:GC_Security_Classification_TypeCode codeListValue="" codeList="http://www.ec.gc.ca/data_donnees/standards/schemas/napec#GC_Security_Classification"/>
</napec:GC_Security_Classification>
<napec:EC_Program>
<napec:EC_Program_TypeCode codeListValue="" codeList="http://www.ec.gc.ca/data_donnees/standards/schemas/napec#EC_Program"/>
</napec:EC_Program>
</napec:EC_CorporateInfo>
</napec:MD_DataIdentification>
Let me know if you need additional info. Thanks! -- Alex
(issue moved from initial repos to GitHub) Feature request. Not a blocker. Would constitute a significantly useful new feature.
The existing mcf file inheritance feature via base_mcf= is useful but has a limitation that mcf files can have only a single 'parent' and a single 'child'. This is a tree structure. This limitation means that whole branches of the mcf file tree would need to be duplicated to accommodate, as an example, the combination of dissemination methods for similar datasets.
Instead of modifying the existing template inheritance capability (i.e. base_mcf=), Tom proposed to use the more flexible 'include' command to allow multiple mcf file inclusions by pygeometa. This ticket is the request for the implementation of such 'multiple mcf file inclusion' capability.
Feature request. It would be useful to have keywords in the MCF files to be substituted when pygeometa is ran.
As an example, if the keyword
Examples of where
datestamp=$date$
would become datestamp=2016-09-13
at running timerights_en=Copyright (c) $year$ Her Majesty the Queen in Right of Canada
would become rights_en=Copyright (c) 2016 Her Majesty the Queen in Right of Canada
Documentation currently indicate:
generate_metadata.py --mcf=path/to/file.mcf --schema_local=/path/to/my-schema --output some_file.xml
=
to --output
and thus changing togenerate_metadata.py --mcf=path/to/file.mcf --schema_local=/path/to/my-schema --output=some_file.xml
Agreed? Thx
When the example 'language_alternate' directive is removed from an MCF (i.e.: creating a unilingual MCF), or if the directive is set to an empty value (e.g.: "language_alternate="), iso19139 Jinja2 template appears to raise UndefinedError
Example:
>>> from pygeometa import render_template
>>> xml = render_template('unilingual.mcf', 'iso19139')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/bjv/git/pygeometa/pygeometa/__init__.py", line 202, in render_template
software_version=__version__).encode('utf-8')
File "/home/bjv/git/pygeometa/my-env/lib/python3.4/site-packages/jinja2/environment.py", line 989, in render
return self.environment.handle_exception(exc_info, True)
File "/home/bjv/git/pygeometa/my-env/lib/python3.4/site-packages/jinja2/environment.py", line 754, in handle_exception
reraise(exc_type, exc_value, tb)
File "/home/bjv/git/pygeometa/my-env/lib/python3.4/site-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "/home/bjv/git/pygeometa/pygeometa/templates/iso19139/main.j2", line 163, in top-level template code
{% for kw1, kw2 in zip(keywords[0].split(','), keywords[1].split(',')) %}
jinja2.exceptions.UndefinedError: 'None' has no attribute 'split'
HNAP2.3 needs the WMS layer also specified in the gmd:linkage gmd:URL element of the output XML.
Current output XML:
<gmd:CI_OnlineResource>
<gmd:linkage>
<gmd:URL>http://geo.weather.gc.ca/geomet/?lang=E&service=WMS&request=GetCapabilities</gmd:URL>
</gmd:linkage>
While HNAP2.3 needs:
<gmd:CI_OnlineResource>
<gmd:linkage>
<gmd:URL>http://geo.weather.gc.ca/geomet/?lang=E&service=WMS&request=GetCapabilities&layers=GDPS.DIAG_NW&legend_format=image/png&feature_info_type=text/plain</gmd:URL>
</gmd:linkage>
Let me know if you need additional details.
Seems like running pygeometa self tests fail after a fresh installation.
Beginning of the error message:
/my-env/pygeometa/tests$ python run_tests.py
__main__.PygeometaTest.test_get_charstring: Test support of unilingual or multilingual value(s)
.__main__.PygeometaTest.test_get_supported_schemas: Test supported schemas
.__main__.PygeometaTest.test_mcf_model: test mcf model and types
.__main__.PygeometaTest.test_nested_mcf: test nested mcf support
.__main__.PygeometaTest.test_pretty_print: Test pretty-printing
.__main__.PygeometaTest.test_read_mcf: Test reading MCFs, strings or dict
.__main__.PygeometaTest.test_render_template: test template rendering
--- Logging error ---
Traceback (most recent call last):
File "/usr/lib/python3.4/logging/__init__.py", line 978, in emit
msg = self.format(record)
File "/usr/lib/python3.4/logging/__init__.py", line 828, in format
return fmt.format(record)
File "/usr/lib/python3.4/logging/__init__.py", line 573, in format
record.exc_text = self.formatException(record.exc_info)
File "/usr/lib/python3.4/logging/__init__.py", line 523, in formatException
traceback.print_exception(ei[0], ei[1], tb, None, sio)
File "/usr/lib/python3.4/traceback.py", line 169, in print_exception
for line in _format_exception_iter(etype, value, tb, limit, chain):
File "/usr/lib/python3.4/traceback.py", line 146, in _format_exception_iter
for value, tb in values:
File "/usr/lib/python3.4/traceback.py", line 125, in _iter_chain
context = exc.__context__
AttributeError: 'NoneType' object has no attribute '__context__'
Copying @RousseauLambertLP . Thanks!
Currently pygeometa's API allows for only passing an MCF file by filepath:
from pygeometa import render_template
xml_string = render_template('/path/to/file.mcf', schema='iso19139')
Allow for passing of either plain old strings or ConfigParser objects:
from pygeometa import render_template
mcf_string = '...' # some string
xml_string = render_template_string(mcf_string, schema='iso19139')
from pygeometa import render_template
mcf_cp = '...' # some ConfigParser object
xml_string = render_template_string(mcf_cp, schema='iso19139')
Documentation specify that the CRS needs to be an 'EPSG code identifier'. Can pygeometa ingest anything else at the moment? Can it ingest OGC WKT directly? If not, should this be a feature request? Thanks -- Alex
Feature request for the support of the HNAP schema, specifically the implementation of language in HNAP which doesn't use gmd:language
but rather uses xlink:role
for specifying language.
Possible implementation (not from me) proposes supporting tokens such as [distribution:wms:eng-CAN]
and [distribution:wms:fra-CAN]
, thus if there's three tokens (1:2:3), pygeometa can output the last token as xlink:role
Feature request for pygeometa to be able to import ISO metadata records from external sources and create corresponding pygeometa MCF records in the yaml format.
Example use cases:
Potential bug, or at least a parameter missing from the output HNAP XML. Under gmd:MD_Distribution, there should be a gmd:distributionFormat. Currently, distributionformat is nowhere in pygeometa's output XML.
Example I've been provided:
<gmd:distributionInfo>
<gmd:MD_Distribution>
<gmd:distributionFormat>
<gmd:MD_Format>
<gmd:name>
<gco:CharacterString>dsa</gco:CharacterString>
</gmd:name>
<gmd:version>
<gco:CharacterString>sfda</gco:CharacterString>
</gmd:version>
</gmd:MD_Format>
</gmd:distributionFormat>
...
Let me know if you need more details, thanks! -- Alex
This is a question, not a change request.
HNAP2.3 use an aggregation of 3 controlled fields for the description of online resources. It does not allow providing an actual description of the online resource in its description parameter. My questions:
Here's an example of what HNAP2.3 wants as a description of an online resource: Web Service;WMS;eng
Thanks for your thoughts and comments -- Alex
There should be a mechanism in pygeometa that validates that the MCF version stated in an input MCF file is indeed supported by pygeometa.
When trying to use a creation or publication date older than 1900, I get an error:
File "/home/ops/afss/lor/ENV/ven-p2/bin/pygeometa", line 9, in <module>
load_entry_point('pygeometa==0.4.dev0', 'console_scripts', 'pygeometa')()
File "/home/ops/afss/lor/ENV/ven-p2/local/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/home/ops/afss/lor/ENV/ven-p2/local/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/home/ops/afss/lor/ENV/ven-p2/local/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/ops/afss/lor/ENV/ven-p2/local/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/ops/afss/lor/ENV/ven-p2/local/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/home/ops/afss/lor/ENV/ven-p2/local/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/ops/afss/lor/ENV/ven-p2/local/lib/python2.7/site-packages/pygeometa-0.4.dev0-py2.7.egg/pygeometa/core.py", line 377, in generate_metadata
schema_local=schema_local)
File "/home/ops/afss/lor/ENV/ven-p2/local/lib/python2.7/site-packages/pygeometa-0.4.dev0-py2.7.egg/pygeometa/core.py", line 325, in render_template
software_version=VERSION).encode('utf-8')
File "/home/ops/afss/lor/ENV/ven-p2/local/lib/python2.7/site-packages/Jinja2-2.10-py2.7.egg/jinja2/environment.py", line 1008, in render
return self.environment.handle_exception(exc_info, True)
File "/home/ops/afss/lor/ENV/ven-p2/local/lib/python2.7/site-packages/Jinja2-2.10-py2.7.egg/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/home/ops/afss/lor/ENV/ven-p2/lib/python2.7/site-packages/pygeometa-0.4.dev0-py2.7.egg/pygeometa/templates/iso19139-hnap/main.j2", line 149, in top-level template code
{% set datestamp = date|normalize_datestring %}
File "/home/ops/afss/lor/ENV/ven-p2/local/lib/python2.7/site-packages/pygeometa-0.4.dev0-py2.7.egg/pygeometa/core.py", line 122, in normalize_datestring
datestring2 = datestring.strftime('%Y-%m-%dT%H:%M:%SZ')
ValueError: year=1871 is before 1900; the datetime strftime() methods require year >= 1900
Looks like the problem is with the datestring
library.
There are most probably errors in the MCF_Reference.md file regarding which properties are indeed mandatory when it comes localization.
Example, 'organization' is flagged as mandatory, but 'organization_en' and 'organization_fr' as well. Same for 'url', 'url_en' and 'url_fr'. I suggest that properties flagged as mandatory should be the ones indicated as such for the ISO19115:2003 schema.
Makes sense @tomkralidis ?
Missing xsd for the HNAP schema.
<gmd:MD_Metadata xsi:schemaLocation="http://www.ec.gc.ca/data_donnees/standards/schemas/n
apec schema.xsd">
Does it make sense to tag the current pygeometa version on GitHub as 0.1.0? It currently has no version number and this is annoying when trying to reference to a pygeometa version. Furthermore, pygeometa's output currently includes:
<gco:CharacterString>This metadata record was generated by pygeometa-0.1.0 (https://github.com/geopython/pygeometa)</gco:CharacterString>
It specifically states 0.1.0, so it would make sense that this value is in sync with release tags in the code repository. Makes sense? Thanks! -- Alex
Request for the adjustment of pygeometa's HNAP XML output for dates. As of HNAP 2.3, both publication and creation dates are mandatory (with their respective 'Date Type').
In a pygeometa MCF file, in the [identification] section, if I provide:
date=1991-03-12T00:00:00Z
datetype=publication
date=1996-01-01T00:00:00Z
datetype=creation
the first date is ignored (probably overwritten by the second one).
A potential solution would be to, in MCF files, specifying dates with parameters such as:
creationdate=value
publicationdate=value
A creationdate parameter in an MCF would create both the date and datetype in the XML.
Better ideas? Thanks -- Alex
Feature request. Currently, an empty XML output file is generated if pygeometa can't parse the whole MCF file because the MCF file contains syntax errors. Would be nice if the output XML file is written to file at the end of pygeometa's process, thus if pygeometa ends prematurely in a ParsingError, at least there won't be an empty file created.
This is different from feature request #5 which requires more code. Thanks -- Alex
When pygeometa generates the xml metadata files, it creates a <gmd:distribution format>
for each value in distribution
even if the distribution format already exists.
Example in MSC case:
Under distribution
in the YAML file we have 3 distribution types:
For each of these distribution type we have a format_en
and format_fr
tag. pygeometa creates a <gmd:distribution format>
tag in the xml for each of these format_xx
even if it already exist in the xml. This creates many <gmd:distribution format>
with the same value. Note that each <gmd:distribution format>
has an English and French value.
I think we need to modify the code so that pygeometa checks if the format already exists in the xml, and if it does, it doesn't create another <gmd:distribution format>
with the same value.
Copying @alexandreleroux and @tomkralidis for comments.
Feature request for the capability to launch pygeometa and process multiple MCF files located in a user-specified directory with a single command.
This 'batch mode' could be invoked directly from pygeometa with the --batch
argument. If --batch
is specified, then pygeometa is ran for every mcf files in the directory specified with --mfc=
and all the outputs files are named the same as input files but end with .xml and are saved in the directory specified by --output
. Other pygeometa arguments such as --schema=
are applied to all MCF files processed in batch.
An initial version of this batch mode can simply loop on all mcf files and generate the corresponding xml files. The batch mode should skip base_mcf mcf files, which can't be processed on their own.
Future versions could:
--mfc=
folder to process .mcf files located in subfolders as well and output the same directory structure to --output=
--log=
argument--log=
locationThoughts / comments?
Feature request. (moved from the to-do list to the issue tracker)
Would be useful to have a validation mechanism for MCF files. Validating things such as: MCF syntax, missing non-optional values based on schema, duplicate base_mcf, etc
Feature request as discussed offline.
Ability to specify the output XML filename and location from a parameter when running pygeometa.
Example:
generate_metadata.py --mcf=sample.mcf --schema=iso19139-hnap --outputxml=sample.xml
One advantage of proceeding this way is being able to handle errors before the outputxml file is created.
Documentation will need to be updated accordingly (I can do that :-))
Request for supporting 'inapplicable' for [identification]:language values.
In the case where a dataset is made of numerical values, it makes sense for the dataset's language to be set to 'none' or its ISO19115 equivalent. pygeometa needs to support this value.
Unlike what's mentioned in the doc, pygeometa doesn't seem to support chain nesting.
It looks like it's only reading one level of MCF nesting.
For example, first I create a MCF for the radar model using a base_mcf for the contact information. Then I'm trying to create a specific radar layer MCF using the radar model MCF and it fails because pygeometa is not able to read the information from the contact file.
contact.yml:
Contact:
blablabla
radar_model.yml:
base_mcf: ../contact.yml
Identification:
titre:
abstract:
radar_layer.yml:
base_mcf: ../radar_model.yml
Identification:
titre: another title
abstract: another abstract
Then I get an error because I have no contact key in my radar_layer.yml:
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'contact'
If I copy manually the whole contact.yml file into my radar_layer.yml, I don't get an error...
copying @tomkralidis and @alexandreleroux for comments
I looked at https://github.com/geopython/pygeometa/blob/master/pygeometa/templates/iso19139-hnap/main.j2#L421-454 and saw that only if the CRS value is not equal to '4326' the BBOX value should be written on the same line. If the CRS is set to 4326, it should use the {else} tag where the max lat, lon and min lat lon should be written separately on different lines.
We have a few models which uses CRS 4326 and they all have the same behaviour.
Ex for GDPS (xml):
<gmd:polygon>
<gml:Polygon gml:id="P001" srsDimension="2" srsName="4326">
<gml:exterior>
<gml:LinearRing>
<gml:posList srsDimension="2" srsName="4326">-180 -90 -180 90 180 90 180 -90 -180 -90</gml:posList>
</gml:LinearRing>
</gml:exterior>
</gml:Polygon>
In the YAML:
spatial:
datatype: grid
geomtype: solid
crs: 4326
bbox: -180, -90, 180, 90
Copying @alexandreleroux and @tomkralidis for comments.
Hello,
Not sure if this issue is specifically with pygeometa or if there is another issue somewhere in my workflow. I'm working to create metadata files for all of my companies geospatial data. As a test, I created a test yaml file based on the sample one from this repository, and updated it with information pertinent for my use. Pygeometa works fine and creates an xml file if I have everything hardcoded into that yaml file.
However, I had hoped to be able to update the yaml on the fly as I go through the data files so that I can update it with the extent information, file type (vector, grid, or something else) and other information based on the file. When I do this, I can update the yaml file as expected and save it to be able to be used by pygeometa. If I do this though, I recieve the error as mentioned in the title above. So once again, if I manually update my yaml file and save it and use pygeometa it works but if I try to do it on the fly it does not. Any help would be useful if anyone can see something in my code that would be cause this issue to arise. Thanks in advance.
System: Win7
Env: JupyterLab notebook for testing
Python Version: 3.6.6
import yaml
from osgeo import gdal, ogr, osr
import datetime
import os
from pygeometa.core import render_template
path = "path_to_my_geospatial_data"
# Then I grab a bunch of the information about the geo datafile to input into my *.yml file
# Next open the basic file
with open("test.yml") as f:
data = yaml.load(f)
# change values in data to account for the information I got from the geo datafile
# Now save to a new temporary .yml file
with open("test_new.yml", "w") as ff:
yaml.dump(data, ff)
ff.close()
# Finish with giving the new .yml file to pygeometa to create the metadata xml
xml_string = render_template("test_new.yml", schema='iso19139')
with open('test_new.xml', 'w') as x:
x.write(xml_string)
If pygeometa is to support the 'WMO Core Profile', users will be able to use pygeometa to fulfil WMO WIS requirements.
HNAP2.3 requires the use of xlink:role to specify language for WMS Map Resources. Current pygeometa does not duplicate WMS Map Resources as required by HNAP2.3.
Here's an example of a desired HNAP2.3 output XML where you can see that gmd:transfer Options is duplicated for the WMS Map Resource to add French via xlink:role="urn:xml:lang:fra-CAN":
<gmd:transferOptions>
<gmd:MD_DigitalTransferOptions>
<gmd:onLine xlink:role="urn:xml:lang:eng-CAN" xlink:title="tor1d5678012e1497">
<gmd:CI_OnlineResource>
<gmd:linkage>
<gmd:URL>http://dd.weather.gc.ca/model_gem_global/25km/grib2/lat_lon/</gmd:URL>
</gmd:linkage>
<gmd:protocol>
<gco:CharacterString>WWW:LINK</gco:CharacterString>
</gmd:protocol>
<gmd:name xsi:type="gmd:PT_FreeText_PropertyType">
<gco:CharacterString>MSC Datamart</gco:CharacterString>
<gmd:PT_FreeText>
<gmd:textGroup>
<gmd:LocalisedCharacterString locale="#fra">Datamart du SMC</gmd:LocalisedCharacterString>
</gmd:textGroup>
</gmd:PT_FreeText>
</gmd:name>
<gmd:description xsi:type="gmd:PT_FreeText_PropertyType">
<gco:CharacterString>The Meteorological Service of Canada (MSC) HTTP data server is a source of several raw meteorological data types and forecast data. This service is aimed at specialized users with good meteorological and IT knowledge, and is mainly meant to be accessed in an automatic manner via the internet (e.g. with scripts). The server's URL is: http://dd.weather.gc.ca/</gco:CharacterString>
<gmd:PT_FreeText>
<gmd:textGroup>
<gmd:LocalisedCharacterString locale="#fra">Le Service météorologique du Canada (SMC) met à la disposition des usagers spécialisés une multitude de données météorologiques et prévisionnelles sur son serveur de données HTTP, à l'adresse : http://dd.meteo.gc.ca/ Les usagers qui désirent accéder à ces données doivent avoir une bonne connaissance de la météorologie et de l'informatique afin d'en faire bon usage. Le mode d'accès privilégié est via des routines d'accès automatiques (scripts).</gmd:LocalisedCharacterString>
</gmd:textGroup>
</gmd:PT_FreeText>
</gmd:description>
<gmd:function>
<gmd:CI_OnLineFunctionCode codeList="http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_OnLineFunctionCode" codeListValue="download" codeSpace="ISOTC211/19115">download</gmd:CI_OnLineFunctionCode>
</gmd:function>
</gmd:CI_OnlineResource>
</gmd:onLine>
<gmd:onLine xlink:role="urn:xml:lang:eng-CAN" xlink:title="tor1d5678012e1551">
<gmd:CI_OnlineResource>
<gmd:linkage>
<gmd:URL>http://geo.weather.gc.ca/geomet/?lang=E&service=WMS&request=GetCapabilities&layers=GDPS.ETA_TT&legend_format=image/png&feature_info_type=text/plain</gmd:URL>
</gmd:linkage>
<gmd:protocol>
<gco:CharacterString>OGC:WMS</gco:CharacterString>
</gmd:protocol>
<gmd:name xsi:type="gmd:PT_FreeText_PropertyType">
<gco:CharacterString>MSC GeoMet</gco:CharacterString>
<gmd:PT_FreeText>
<gmd:textGroup>
<gmd:LocalisedCharacterString locale="#fra">SMC GeoMet</gmd:LocalisedCharacterString>
</gmd:textGroup>
</gmd:PT_FreeText>
</gmd:name>
<gmd:description xsi:type="gmd:PT_FreeText_PropertyType">
<gco:CharacterString>Web Service;WMS;eng</gco:CharacterString>
<gmd:PT_FreeText>
<gmd:textGroup>
<gmd:LocalisedCharacterString locale="#fra">Service Web;WMS;eng</gmd:LocalisedCharacterString>
</gmd:textGroup>
</gmd:PT_FreeText>
</gmd:description>
<gmd:function>
<gmd:CI_OnLineFunctionCode codeList="http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_OnLineFunctionCode" codeListValue="download" codeSpace="ISOTC211/19115">download</gmd:CI_OnLineFunctionCode>
</gmd:function>
</gmd:CI_OnlineResource>
</gmd:onLine>
</gmd:MD_DigitalTransferOptions>
</gmd:transferOptions>
<gmd:transferOptions>
<gmd:MD_DigitalTransferOptions>
<gmd:onLine xlink:role="urn:xml:lang:fra-CAN" xlink:title="tor1d5678012e1611">
<gmd:CI_OnlineResource>
<gmd:linkage>
<gmd:URL>http://geo.weather.gc.ca/geomet/?lang=F&service=WMS&request=GetCapabilities&layers=GDPS.ETA_TT&legend_format=image/png&feature_info_type=text/plain</gmd:URL>
</gmd:linkage>
<gmd:protocol>
<gco:CharacterString>OGC:WMS</gco:CharacterString>
</gmd:protocol>
<gmd:name xsi:type="gmd:PT_FreeText_PropertyType">
<gco:CharacterString>MSC GeoM</gco:CharacterString>
<gmd:PT_FreeText>
<gmd:textGroup>
<gmd:LocalisedCharacterString locale="#fra">SMC GeoMet</gmd:LocalisedCharacterString>
</gmd:textGroup>
</gmd:PT_FreeText>
</gmd:name>
<gmd:description xsi:type="gmd:PT_FreeText_PropertyType">
<gco:CharacterString>Web Service;WMS;fra</gco:CharacterString>
<gmd:PT_FreeText>
<gmd:textGroup>
<gmd:LocalisedCharacterString locale="#fra">Service Web;WMS;fra</gmd:LocalisedCharacterString>
</gmd:textGroup>
</gmd:PT_FreeText>
</gmd:description>
</gmd:CI_OnlineResource>
</gmd:onLine>
</gmd:MD_DigitalTransferOptions>
</gmd:transferOptions>
If you need additional info for supporting this, let me know! Thanks -- Alex
In a pygeometa MCF file, we must specify the Identification 'url' parameter. We cannot currently specify different URLs for different languages. At least for HNAP, pygeometa stores this information in the output XML as gmd:supplementalInformation. However, for HNAP, this information needs to be provided in the output XML in both language.
Currently, pygeometa generates:
<gmd:supplementalInformation>
<gco:CharacterString>http://weather.gc.ca/grib/grib2_glb_25km_e.html</gco:CharacterString>
</gmd:supplementalInformation>
and the following is expected:
<gmd:supplementalInformation xsi:type="gmd:PT_FreeText_PropertyType">
<gco:CharacterString>http://weather.gc.ca/grib/grib2_glb_25km_e.html</gco:CharacterString>
<gmd:PT_FreeText>
<gmd:textGroup>
<gmd:LocalisedCharacterString locale="#fra">http://weather.gc.ca/grib/grib2_glb_25km_f.html</gmd:LocalisedCharacterString>
</gmd:textGroup>
</gmd:PT_FreeText>
</gmd:supplementalInformation>
One possibility would be to support url_en and url_fr in MCF files. If no url_en or url_fr exists, then url is used. This would allow specifying language-specific URLs in pygeometa MCF files.
Let me know if you need any additional info. Thanks -- Alex
Our current base_mcf
support requires the base MCF to be a fully qualified MCF. Update base_mcf
support so that base MCF files are relative from where they are defined. This results in, by design, MCF snippets which are not required to be fully qualified MCF files. In addition, inheritence should continue to be supported (i.e. if value in base MCF and not in child MCF, then set; if in child and in base, override).
I tried to import some metadata xml files to ECDC but I get some errors. One of them is because we use a variable #locale-fra.
Jose Garcia working on ECDC told me that we should use #fra instead.
@tomkralidis @alexandreleroux should I update https://github.com/geopython/pygeometa/blob/master/pygeometa/templates/iso19139-hnap/main.j2 to output #fra?
Feature request for the capability to specify a post-processing script to be ran after pygeometa is done generating the output xml. The post-processing script could be invoked with --postprocessing=script-name.py
. The pygeometa arguments would be sent to the post-processing script, which could thus identified which input mcf file has been specified and which output xml file has been generated.
Usage example: this would allow pygeometa to automatically launch Python scripts to apply processing which is not within pygeometa's scope. This post-processing capability could be used in conjunction with the batch mode (feature request in #63).
In a future version, the --postprocessing
argument could also write in the pygeometa log.
Thoughts on the pertinence of this feature request?
Feature request to have pygeometa log messages to be exposed when running from the command line. The user should be able to specify the log level to output (e.g. info, warning, error, debug).
We get an error related to the missing MD_Format from the distribution info. I don't know if this is an HNAP-only issue or not. And I admit I know what is the desired value for the parameter (provided we already provide distribution info elsewhere!).
Here's the gmd:MD_Format tag missing in a code sample that has been provided to me, along with some context before+after:
<gmd:distributionInfo>
<gmd:MD_Distribution>
<gmd:distributionFormat>
<gmd:MD_Format>
<gmd:name>
<gco:CharacterString>dsa</gco:CharacterString>
</gmd:name>
<gmd:version>
<gco:CharacterString>sfda</gco:CharacterString>
</gmd:version>
</gmd:MD_Format>
</gmd:distributionFormat>
<gmd:distributor>
<gmd:MD_Distributor>
<gmd:distributorContact>
<gmd:CI_ResponsibleParty>
Let me know if this makes sense to you, and if so, if you have questions or comments. Thanks
As discussed with @tomkralidis ; pygeometa's MCF parameters should support a generic design pattern for language.
Example for the 'format' parameter:
Same applies for all parameters (eg description, description_en, description_fr)
pygeometa currently setting and emitting a single topiccategory
/gmd:topicCategory
. Support multiple elements so users can identify more than one topic category.
In the pygeometa MCFs, there's a value for the MCF version as stated in the documentation:
mcf:
version: 1.0.0
Given we changed the MCF format from INI to YAML, does it make sense to specify mcf.version: 2.0.0 now? Thanks. Copying @RousseauLambertLP
So
Should we update pygeometa’s magic
By default pygeometa writes to stdout
. Add --output
parameter which writes to specified file. This allows for catching errors and exiting gracefully without shell redirecting.
Since latest change 99d6b7b , I get an error parsing options.
Trace:
(my-env)/PATH/my-env/pygeometa$ pygeometa generate-metadata --mcf /local/home/data/afssalx/content/discovery-metadata/mcf/climate/msc_climate-normals.yml --schema=schema=iso19139-hnap --output msc_climate-normals.xml
Usage: pygeometa generate-metadata [OPTIONS]
Error: Invalid value for "--schema": invalid choice: schema=iso19139-hnap. (choose from iso19139, iso19139-hnap, wmo-cmp, wmo-wigos)
With another schema such as iso19139, I get the same error.
As a metadata manager
I want to create "initial" or "stub" metadata records in ISO XML format using OGC Web Services (OWS) as the source for metadata
So that I can reference these "stub" metadata records in my OWS (eg WMS Metadata URL) or import them into a catalogue (eg GeoNetwork)
Use case:
An organization already has one or more OWS services. Data managers have already created services with minimum metadata. This organization wants to further its data governance maturity by adding a catalogue. Catalogue systems are able to harvest services and create MD (eg Geonetwork), however any edits made to the MD records are overwritten on each harvest. Creating stub records from the service will act as a starting point to import into such catalogues.
HNAP2.3 has mandatory requirements for 'Information classification'. There are 5 parameters:
ECDC provides dropdown lists of value values for these parameters. In the pygeometa spirit, I suggest specifying 'Information classification' directly in the MCF file. Parameter names could be HNAP_info-cat, HNAP_geography, HNAP_info-content, HNAP_info-program and HNAP_GCSecLevel
Example of output XML which has 'Information classification' specified in it. For Program and GC Security Level, the associated code goes in the same section EC_CorporateInfo mentioned in issue #19 :
<napec:GC_Security_Classification>
<napec:GC_Security_Classification_TypeCode codeListValue="unclassified" codeList="http://www.ec.gc.ca/data_donnees/standards/schemas/napec#GC_Security_Classification"/>
</napec:GC_Security_Classification>
<napec:EC_Program>
<napec:EC_Program_TypeCode codeListValue="weather_observations_forecasts_and_warnings" codeList="http://www.ec.gc.ca/data_donnees/standards/schemas/napec#EC_Program"/>
</napec:EC_Program>
</napec:EC_CorporateInfo>
</napec:MD_DataIdentification>
For info-cat, geography and content, they go in the gmd:descriptivekeywords section, such as:
<gmd:descriptiveKeywords>
<gmd:MD_Keywords>
<gmd:keyword xmlns:srv="http://www.isotc211.org/2005/srv" xsi:type="gmd:PT_FreeText_PropertyType">
<gco:CharacterString>Weather and Climate</gco:CharacterString>
<gmd:PT_FreeText>
<gmd:textGroup>
<gmd:LocalisedCharacterString locale="#fra">Temps et climat</gmd:LocalisedCharacterString>
</gmd:textGroup>
</gmd:PT_FreeText>
</gmd:keyword>
<gmd:type xmlns:srv="http://www.isotc211.org/2005/srv">
<gmd:MD_KeywordTypeCode codeList="http://nap.geogratis.gc.ca/metadata/register/napMetadataRegister.xml#IC_101" codeListValue="theme"/>
</gmd:type>
<gmd:thesaurusName>
<gmd:CI_Citation id="local.theme.EC_Information_Category">
<gmd:title>
<gco:CharacterString>local.theme.EC_Information_Category</gco:CharacterString>
</gmd:title>
<gmd:date>
<gmd:CI_Date>
<gmd:date>
<gco:Date>2012-05-25</gco:Date>
</gmd:date>
<gmd:dateType xmlns:srv="http://www.isotc211.org/2005/srv">
<gmd:CI_DateTypeCode codeList="http://nap.geogratis.gc.ca/metadata/register/napMetadataRegister.xml#IC_87" codeListValue="RI_367">publication; publication</gmd:CI_DateTypeCode>
</gmd:dateType>
</gmd:CI_Date>
</gmd:date>
<gmd:citedResponsibleParty/>
</gmd:CI_Citation>
</gmd:thesaurusName>
</gmd:MD_Keywords>
</gmd:descriptiveKeywords>
<gmd:descriptiveKeywords>
<gmd:MD_Keywords>
<gmd:keyword xmlns:srv="http://www.isotc211.org/2005/srv" xsi:type="gmd:PT_FreeText_PropertyType">
<gco:CharacterString>International</gco:CharacterString>
<gmd:PT_FreeText>
<gmd:textGroup>
<gmd:LocalisedCharacterString locale="#fra">Internationale</gmd:LocalisedCharacterString>
</gmd:textGroup>
</gmd:PT_FreeText>
</gmd:keyword>
<gmd:type xmlns:srv="http://www.isotc211.org/2005/srv">
<gmd:MD_KeywordTypeCode codeList="http://nap.geogratis.gc.ca/metadata/register/napMetadataRegister.xml#IC_101" codeListValue="place"/>
</gmd:type>
<gmd:thesaurusName>
<gmd:CI_Citation id="local.place.EC_Geographic_Scope">
<gmd:title>
<gco:CharacterString>local.place.EC_Geographic_Scope</gco:CharacterString>
</gmd:title>
<gmd:date>
<gmd:CI_Date>
<gmd:date>
<gco:Date>2012-05-25</gco:Date>
</gmd:date>
<gmd:dateType xmlns:srv="http://www.isotc211.org/2005/srv">
<gmd:CI_DateTypeCode codeList="http://nap.geogratis.gc.ca/metadata/register/napMetadataRegister.xml#IC_87" codeListValue="RI_367">publication; publication</gmd:CI_DateTypeCode>
</gmd:dateType>
</gmd:CI_Date>
</gmd:date>
<gmd:citedResponsibleParty/>
</gmd:CI_Citation>
</gmd:thesaurusName>
</gmd:MD_Keywords>
</gmd:descriptiveKeywords>
<gmd:descriptiveKeywords>
<gmd:MD_Keywords>
<gmd:keyword xmlns:srv="http://www.isotc211.org/2005/srv" xsi:type="gmd:PT_FreeText_PropertyType">
<gco:CharacterString>Environmental Model</gco:CharacterString>
<gmd:PT_FreeText>
<gmd:textGroup>
<gmd:LocalisedCharacterString locale="#fra">Modèle environnemental</gmd:LocalisedCharacterString>
</gmd:textGroup>
</gmd:PT_FreeText>
</gmd:keyword>
<gmd:type xmlns:srv="http://www.isotc211.org/2005/srv">
<gmd:MD_KeywordTypeCode codeList="http://nap.geogratis.gc.ca/metadata/register/napMetadataRegister.xml#IC_101" codeListValue="theme"/>
</gmd:type>
<gmd:thesaurusName>
<gmd:CI_Citation id="local.theme.EC_Content_Scope">
<gmd:title>
<gco:CharacterString>local.theme.EC_Content_Scope</gco:CharacterString>
</gmd:title>
<gmd:date>
<gmd:CI_Date>
<gmd:date>
<gco:Date>2012-05-25</gco:Date>
</gmd:date>
<gmd:dateType xmlns:srv="http://www.isotc211.org/2005/srv">
<gmd:CI_DateTypeCode codeList="http://nap.geogratis.gc.ca/metadata/register/napMetadataRegister.xml#IC_87" codeListValue="RI_367">publication; publication</gmd:CI_DateTypeCode>
</gmd:dateType>
</gmd:CI_Date>
</gmd:date>
<gmd:citedResponsibleParty/>
</gmd:CI_Citation>
</gmd:thesaurusName>
</gmd:MD_Keywords>
</gmd:descriptiveKeywords>
Let me know if you need additional info. Thanks! -- Alex
Long term feature request, no rush to implement this on my end.
It would be useful if pygeometa was to support YAML as input format for metadata in addition to the MCF format.
Possible bug, at least a feature request for HNAP2.3 compliance.
In current HNAP XML output, for the EPSG we have:
<gmd:role gco:nilReason="missing"/>
while ECDC expects something along the lines of:
<gmd:role xmlns:srv="http://www.isotc211.org/2005/srv" xmlns:napec="http://www.ec.gc.ca/data_donnees/standards/schemas/napec">
<gmd:CI_RoleCode codeList="http://nap.geogratis.gc.ca/metadata/register/napMetadataRegister.xml#IC_90" codeListValue="RI_413">originator; auteur</gmd:CI_RoleCode>
</gmd:role>
In this example, I chose 'Originator' as role, however, the ECDC proposes 15 possible values, such as Author, Distributor, Editor, Owner, Publisher, Resource provider, Rights holder, etc.
Let me know how I can help, thanks -- Alex
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.