mjanez / ckanext-schemingdcat Goto Github PK

• publisher_info template
• contact_info template

ckanext-dcat

• Migrate profiles from mjanez/ckanext-dcat:
  • spain_dcat/es_dcat: NTI-RISP profile.
  • es_dcat_ap_2: Adapt new spain_dcat_ap_2 profile to DCAT-AP 2.1.1.
  • es_dcat_ap_3: Adapt the new Spanish profile to DCAT-AP 3.0.0.
  • es_geodcat_ap_3: Adapt new Spanish profile to GeoDCAT-AP 3.0.0
  • eu_dcat_ap_2: Adapt the new EU profile to DCAT-AP 2.1.1
  • eu_dcat_ap_3: Adapt the new EU profile to DCAT-AP 3.0.0.
• mjanez/ckanext-dcat#26
• #99
• #92
• Improve aux profiles tools
  • Processing Catalogs
  • Multilang rdfy
  • Utils
  • Default configs
  • Add fields
  • #80
  • Fix spatial range of dct:Location properties mjanez/ckanext-dcat#4

ckanext-scheming

• Enhance schemas to ensure compatibility with new profiles:
  • geodcat_ap_es to GeoDCAT-AP 3.0.0
  • geodcat_ap to GeoDCAT-AP 3.0.0
  • dcat_ap to DCAT-AP 3.0.0

To do

• Improve multilingual jinja texts with i18n. Info: https://docs.ckan.org/en/2.9/extensions/translating-extensions.html

Fixes

• Filter by location
  

Based on the controlled list, expand the list of available metadata languages http://publications.europa.eu/resource/authority/language

Schemas:

• https://github.com/mjanez/ckanext-scheming_dcat/blob/main/ckanext/scheming_dcat/schemas/README.md

Example:
https://github.com/mjanez/ckanext-scheming_dcat/blob/919e4bd63284071e3d4ee6ca014bd039fbd40af9/ckanext/scheming_dcat/schemas/geodcatap_es/geodcatap_es_dataset.yaml#L2202-L2218

To do

• Update metadata element in schemas.
• Update schemas info.

Info

mjanez/ckanext-dcat@d9a823e
https://semiceu.github.io/DCAT-AP/releases/2.2.0-hvd/

• Refactor all controllers to controller.py:

  • package_controller.py

• ATOM Feed

  • Add Feed plugin:

    def before_map(self, map):
        with SubMapper(map, controller='ckanext.scheming_dcat.controller:SchemingDcatFeedController') as m:
            m.connect('/feeds/organization/{id}.atom', action='organization')
        map.connect(
            'general', '/feeds/dataset/{pkg_id}.atom',
            controller='ckanext.scheming_dcat.controller:SchemingDcatFeedController',
            action='dataset',
        )
        return map

  • Add FeedController for ATOM Feed:

    class SchemingDcatFeedController(FeedController):
        def general(self):
            data_dict, params = self._parse_url_params()
            data_dict['q'] = '*:*'
    
            item_count, results = _package_search(data_dict)
    
            navigation_urls = self._navigation_urls(params,
                                                    item_count=item_count,
                                                    limit=data_dict['rows'],
                                                    controller='feed',
                                                    action='general')
    
            feed_url = self._feed_url(params,
                                      controller='feed',
                                      action='general')
    
            alternate_url = self._alternate_url(params)
    
            return self.output_feed(
                results,
                feed_title=_(u'Open Government Dataset Feed'),
                feed_description='',
                feed_link=alternate_url,
                feed_guid=_create_atom_id(
                    u'/feeds/dataset.atom'),
                feed_url=feed_url,
                navigation_urls=navigation_urls
            )
    
        def dataset(self, pkg_id):
            try:
                context = {'model': model, 'session': model.Session,
                           'user': c.user, 'auth_user_obj': c.userobj}
                get_action('package_show')(context,
                                            {'id': pkg_id})
            except NotFound:
                abort(404, _('Dataset not found'))
    
            data_dict, params = self._parse_url_params()
    
            data_dict['fq'] = '{0}:"{1}"'.format('id', pkg_id)
    
            item_count, results = _package_search(data_dict)
    
            navigation_urls = self._navigation_urls(params,
                                                    item_count=item_count,
                                                    limit=data_dict['rows'],
                                                    controller='feed',
                                                    action='dataset',
                                                    id=pkg_id)
    
            feed_url = self._feed_url(params,
                                      controller='feed',
                                      action='dataset',
                                      id=pkg_id)
    
            alternate_url = self._alternate_url(params, id=pkg_id)
    
            return self.output_feed(
                results,
                feed_title=_(u'Open Government Dataset Feed'),
                feed_description='',
                feed_link=alternate_url,
                feed_guid=_create_atom_id(
                    u'/feeds/dataset/{0}.atom'.format(pkg_id)),
                feed_url=feed_url,
                navigation_urls=navigation_urls
            )
    
        def output_feed(self, results, feed_title, feed_description,
                        feed_link, feed_url, navigation_urls, feed_guid):
    
            author_name = config.get('ckan.feeds.author_name', '').strip() or \
                config.get('ckan.site_id', '').strip()
            author_link = config.get('ckan.feeds.author_link', '').strip() or \
                config.get('ckan.site_url', '').strip()
    
            feed = _FixedAtom1Feed(
                title=feed_title,
                link=feed_link,
                description=feed_description,
                language=u'en',
                author_name=author_name,
                author_link=author_link,
                feed_guid=feed_guid,
                feed_url=feed_url,
                previous_page=navigation_urls['previous'],
                next_page=navigation_urls['next'],
                first_page=navigation_urls['first'],
                last_page=navigation_urls['last'],
            )
    
            for pkg in results:
                feed.add_item(
                    title= h.get_translated(pkg, 'title'),
                    link=h.url_for(controller='package', action='read', id=pkg['id']),
                    description= h.get_translated(pkg, 'notes'),
                    updated=date_str_to_datetime(pkg.get('metadata_modified')),
                    published=date_str_to_datetime(pkg.get('metadata_created')),
                    unique_id=_create_atom_id(u'/dataset/%s' % pkg['id']),
                    author_name=pkg.get('author', ''),
                    author_email=pkg.get('author_email', ''),
                    categories=''.join(e['value']
                                       for e in pkg.get('extras', [])
                                       if e['key'] == lx('keywords')).split(','),
                    enclosure=webhelpers.feedgenerator.Enclosure(
                        self.base_url + url(str(
                            '/api/action/package_show?id=%s' % pkg['name'])),
                        unicode(len(json.dumps(pkg))),   # TODO fix this
                        u'application/json')
                )
            response.content_type = feed.mime_type
            return feed.writeString('utf-8')

• Fix duplicate attribution in widget map. #44
  

• Update the facet templates to include labels of the schema URIs like: tag_uri, conforms_to, etc. instead of raw URI names.

  https://github.com/mjanez/ckanext-scheming_dcat/blob/919e4bd63284071e3d4ee6ca014bd039fbd40af9/ckanext/scheming_dcat/schemas/geodcatap_es/geodcatap_es_dataset.yaml#L645-L658

  

• Improve metadata forms:

  

Adapt custom OGC harvester for INSPIRE Endpoints.

• OGC Harvester
• CSW Harvester and ckanext-spatial/csw_harvester
• XML Harvester and 'ckanext-spatial/doc_harvester`

Migrated from: mjanez/ckanext-scheming#10

Steps

GeoDCAT-AP Schema

• Add generic continents
  to: spatial_uri field.

• Add generic icon for non-spanish uris spatial_uris.html template.

• Add icons for countries.

DCAT-AP Schema

• Add generic continents
  and european countries
  to: spatial_uri field.

DCAT-AP Info

Fix tests and add module testing to Github Actions.

https://github.com/mjanez/ckanext-scheming_dcat/blob/6dddad2a3a9df550aa27459e717a77b407e7eaf7/test.ini#L1-L56

1. ISO 8601: https://www.w3.org/TR/xmlschema11-2/#duration

duration is a datatype that represents durations of time. The concept of duration being captured is drawn from those of [ISO 8601], specifically durations without fixed endpoints. For example, "15 days" (whose most common lexical representation in duration is "'P15D'") is a duration value; "15 days beginning 12 July 1995" and "15 days ending 12 July 1995" are not duration values. duration can provide addition and subtraction operations between duration values and between duration/dateTime value pairs, and can be the result of subtracting dateTime values. However, only addition to dateTime is required for XML Schema processing and is defined in the function ·dateTimePlusDuration·.

Maintain consistency in the naming of field_names, utilizing EU INSPIRE themes as a default, and adding suffixes to any national values.

• theme: INSPIRE Themes in all schemas. Spatial data themes, as defined in the Annexes of the INSPIRE Directive, compatibles with DCAT and also DCAT-AP/GeoDCAT-AP. (E.g: https://www.w3.org/TR/vocab-dcat-3/#ex-spatial-coverage-bbox)
• theme_{code}: For instance, theme_eu for MDR Themes or theme_es for spanish NTI-RISP vocabulary themes.

Adaptation is also required:

• ckan-docker (Solr fields, .env, etc.)
• ckan-ogc theme/theme_inspire (Harvester/Ingester fields)
• ckan-pycsw (theme == <gmd:keyword> from Thesaurus.)

Internally all extra fields are stored as strings. If you are attempting to save and restore other types of data you will need to use output validators.

For example if you use a simple "yes/no" question, you will need to let ckanext-scheming know that this data needs to be stored and retrieved as a boolean. This is acheieved using validators and output_validators keys.

 - field_name: is_camel_friendly
   label: Is this camel friendly?
   required: true
   preset: select
   choices:
     - value: false
       label: "No"
     - value: true
       label: "Yes"
   validators: scheming_required boolean_validator
   output_validators: boolean_validator

Info: https://github.com/ckan/ckanext-scheming#output_validators

Schemas

• Fix all multiple fields in custom schemas.
  • DCAT
  • DCAT-AP
  • GeoDCAT-AP (EU/ES)

Components of ckan-docker deployment

• ckan-docker: Check API actions like package_show.
• ckan-ogc: Check ingested JSON Lists.
• ckan-pycsw: Check harvested JSON Lists.

The output of information in the API needs to be fixed to show more information about groups and organisations, or to generate a specific action.

Extensible to tags.

Concretely the multilingual fields.

Example

https://hermes.tragsatec.es/catalogo/api/3/action/group_list?all_fields=True

{
            "approval_status": "approved",
            "created": "2023-10-18T16:12:32.494529",
            "description": "",
            "display_name": "",
            "id": "64b56bd3-27da-4d20-b3c7-4cd3de945789",
            "image_display_url": "https://hermes.tragsatec.es/catalogo/dataset/6439fa33-1fe9-43ce-adf3-3a2e3177246a/resource/523a3d6a-9dbd-4059-b74e-20cce785f9c9/download/af_2.jpg",
            "image_url": "https://hermes.tragsatec.es/catalogo/dataset/6439fa33-1fe9-43ce-adf3-3a2e3177246a/resource/523a3d6a-9dbd-4059-b74e-20cce785f9c9/download/af_2.jpg",
            "is_organization": false,
            "name": "7_alternative-fuels",
            "num_followers": 0,
            "package_count": 10,
            "state": "active",
            "title": "",
            "type": "group"
        }

tags

{
            "id": "96d4a6d8-b9d2-470e-aad6-57689396882d",
            "name": "theme",
            "tags": [
                {
                    "id": "e50c57e9-54c5-4400-806f-3e30b313abd9",
                    "name": "ac",
                    "vocabulary_id": "96d4a6d8-b9d2-470e-aad6-57689396882d",
                    "display_name": "ac"
                },
                {
                    "id": "228741b0-c86c-4ee4-8f8a-3c83e45aaa71",
                    "name": "ad",
                    "vocabulary_id": "96d4a6d8-b9d2-470e-aad6-57689396882d",
                    "display_name": "ad"
                },
                {
                    "id": "a5f92a8d-0f07-4c8b-8241-516fb985d8d2",
                    "name": "af",
                    "vocabulary_id": "96d4a6d8-b9d2-470e-aad6-57689396882d",
                    "display_name": "af"
                },
}

Context info

Description:
Graphic that provides an illustration of the dataset

• Pygeometa MCF Core schema

• XML Schema ISO19139 gmd:graphicOverview:

  <gmd:identificationInfo>
  ...
    <gmd:graphicOverview>
      <gmd:MD_BrowseGraphic>
        <gmd:fileName>
          <gco:CharacterString>{{ record['identification']['browsegraphic']|e }}</gco:CharacterString>
          </gmd:fileName>
      </gmd:MD_BrowseGraphic>
    </gmd:graphicOverview>
  </gmd:identificationInfo>

Tasks

Basic

• Add element to schemas.

- field_name: graphic_overview
  label:
    en: Graphic overview of the dataset
    es: Descripción gráfica del conjunto de datos
  display_snippet: link_name.html
  form_placeholder: http://example.com/dataset.jpg
  help_text:
    en: "Graphic that provides an illustration of the dataset."
    es: "Gráfico que ilustra el conjunto de datos."

• Add element to ckan-pycsw ckan GeoDCAT-AP schema
• Update ckan-pycsw pygeometa INSPIRE ISO19139 schema

Enhancements

• Add a custom snippet to upload image from web or use /form_snippet/upload.html

References

DCAT-AP:

• https://github.com/SEMICeu/DCAT-AP/tree/master/releases
• https://semiceu.github.io/DCAT-AP/releases/3.0.0/#validation-of-dcat-ap

EU Vocabularies: https://op.europa.eu/en/web/eu-vocabularies/dcat-ap

Validator:

• https://www.itb.ec.europa.eu/shacl/dcat-ap/upload
• https://github.com/ISAITB/validator-resources-dcat-ap/tree/master#

DCAT-AP Country profile:

• https://github.com/diggsweden/DCAT-AP-SE
• https://github.com/opendata-swiss/dcat_ap_ch

SHACLs: https://github.com/ISAITB/validator-resources-dcat-ap/blob/baca3adf63d31ee415fa5e769249053ae211414c/resources/config.properties

DCAT-AP Validator Validation Cases

The different cases to validate in the DCAT-AP Validator are based on the level of completeness of the checks and the incorporation of background knowledge (vocabularies). Each case is designed for a specific data exchange scenario.
The following describes each case and recommends which one you should use for a CKAN catalog:

Case 1: DCAT-AP Base Zero (no background knowledge)

Includes all constraints required for technical coherence, excluding range class membership constraints and controlled vocabulary usage.

SHACL Profiles:

• 2.1.1
• 3.0.0

Case 2: DCAT-AP Ranges Zero (no background knowledge)

Includes all range class membership constraints.

SHACL Profiles:

• 2.1.1
• 3.0.0

Case 3: DCAT-AP Base (with background knowledge)

Extends Case 1 with background knowledge, including all vocabularies used in DCAT-AP.

SHACL Profiles:

• 2.1.1: shapes and imports
• 3.0.0: shapes and imports

Case 4: DCAT-AP Ranges (with background knowledge)

Extends Case 2 with background knowledge, adding validation of range class membership and vocabulary standards compliance.

SHACL Profiles:

• 2.1.1: range and imports
• 3.0.0: range and imports

Case 5: DCAT-AP Recommendations (with background knowledge)

Includes all constraints related to recommended properties.

SHACL Profiles:

• 2.1.1: shapes recommended and imports
• 3.0.0: shapes recommended and imports

Case 6: DCAT-AP Controlled Vocabularies

Includes all constraints related to controlled vocabularies.

SHACL Profiles:

• 2.1.1: vocabularies shape and imports
• 3.0.0: vocabularies shape and mdr imports

Case 7: DCAT-AP Full (with background knowledge)

The union of Cases 3, 4, 5, and 6.

SHACL Profiles:

• 2.1.1: shapes, shapes recommended, imports, range and deprecateduris
• 3.0.0: shapes, shapes recommended, imports, range and deprecateduris

Recommendation:

For most use cases, Case 3: DCAT-AP Base (with background knowledge) is recommended. It provides comprehensive validation of basic coherence and vocabulary standards compliance.
If your CKAN catalog uses controlled vocabularies, consider using Case 6: DCAT-AP Controlled Vocabularies or Case 7: DCAT-AP Full (with background knowledge) for more exhaustive validation.
Remember, the choice of the appropriate validation case depends on your specific needs and data exchange context.

Fix forms and validations to include tabs instead of stages in package_form templates.

• Allow download codelists of EU Publications. See ckan-mqa

Improve DCAT schemas to include all codelist elements from MARC Genre Terms: https://docs.google.com/spreadsheets/d/1p4h5y82-jiH68qGY_WeCV6HuyJKmuyC1XLjVbwjsAdg/edit#gid=1165291034

• #41
• [ ]

• Update multilingual field schema for geodcatap_es

• Fields

  • title
  • notes
  • provenance
  • lineage_source
  • ...

• field properties

  • preset
  • form_languages

Create a new schema based on the Spanish GeoDCAT-AP YAML for an international context without the mandatory elements of the NTI-RISP standard, only DCAT-AP.

Note
Info about the latest mapping: Schema GeoDCAT-AP.

Steps to complete a ckan_geodcatap_eu.yaml version from ckan_geodcatap.yaml:

Metadata properties:

• Update theme_es of the NTI-RISP (ES) to theme_eu (MDR Data Themes) https://github.com/mjanez/ckanext-scheming/blob/9dcb4e3bacc86efa1accd2853d4a520d740ebf6e/ckanext/scheming/ckan_geodcatap.yaml#L142-L495
• Update multi-value from codelist spatial_uri of the NTI-RISP (ES) to a stored as text value. For example to use geonames values: https://www.geonames.org/2510769/kingdom-of-spain.html

  dct:spatial

  Vocabs: DR Continents Named Authority List , MDR Countries Named Authority List , MDR Places Named Authority List , Geonames.

  Definition: The MDR Name Authority Lists must be used for continents, countries and places that are in those lists; if a particular location is not in one of the mentioned Named Authority Lists, Geonames URIs must be used.

  DCAT-AP

https://github.com/mjanez/ckanext-scheming/blob/9dcb4e3bacc86efa1accd2853d4a520d740ebf6e/ckanext/scheming/ckan_geodcatap.yaml#L1339-L1857

• Add european languages from MDR Authority Language https://github.com/mjanez/ckanext-scheming/blob/9dcb4e3bacc86efa1accd2853d4a520d740ebf6e/ckanext/scheming/ckan_geodcatap.yaml#L513-L532

• #2

• Adapt schemas
• contact
• publisher
• maintainer
• author
• temporal_coverage
• spatial_coverage
• Adapt harvesters.
• Check profiles
• Fix templates.
• Adapt ckan-pycsw

Ref: ckanext-dcat/dcat_ap_full