Overview • Installation • Configuration • Schemas • Harvesters • Running the Tests
This CKAN extension provides functions and templates specifically designed to extend ckanext-scheming and includes DCAT and Harvest enhancements to adapt CKAN Schema to GeoDCAT-AP.
Warning
Requires mjanez/ckanext-dcat, ckan/ckanext-scheming and ckan/ckanext-spatial to work properly.
Tip
It is recommended to use with: ckan-docker deployment or only use ckan-pycsw to deploy a CSW Catalog.
Enhancements:
- Could use schemas for
ckanext-schemingin the plugin like CKAN GeoDCAT-AP custom schemas - Improve the search functionality in CKAN for custom schemas. It uses the fields defined in a scheming file to provide a set of tools to use these fields for scheming, and a way to include icons in their labels when displaying them. More info:
ckanext-schemingdcat - Add improved harvesters for custom metadata schemas integrated with
ckanext-harvestin CKAN usingmjanez/ckan-ogc. - Add Metadata downloads for Linked Open Data formats (
mjanez/ckanext-dcat) and Geospatial Metadata (ISO 19139, Dublin Core, etc. withmjanez/ckan-pycsw) - Add custom i18n translations to
datasets,groups,organizationsin schemas, e.g: GeoDCAT-AP (ES).1 - Add a set of useful helpers and templates to be used with Metadata Schemas.
- Update the base theme of CKAN to use with the enhancements of this extension.
- Modern UI inspired on
datopian/ckanext-datopian. - LOD/OGC Endpoints based on avalaible profiles (DCAT) and CSW capabilities with
mjanez/ckan-pycsw.
This plugin is compatible with CKAN 2.9 or later and needs the following plugins to work properly:
# Install latest stable release of:
## ckan/ckanext-scheming: https://github.com/ckan/ckanext-scheming/tags (e.g. release-3.0.0)
pip install -e git+https://github.com/ckan/[email protected]#egg=ckanext-scheming
## mjanez/ckanext-dcat: https://github.com/mjanez/ckanext-dcat/tags (e.g. 1.2.0-geodcatap)
pip install -e git+https://github.com/mjanez/[email protected]#egg=ckanext-dcat
pip install -r https://raw.githubusercontent.com/mjanez/ckanext-dcat/master/requirements.txt
## ckan/ckckanext-spatial: https://github.com/ckan/ckanext-spatial/tags (e.g. v2.1.1)
pip install -e git++https://github.com/ckan/[email protected]/#egg=ckanext-spatial#egg=ckanext-spatial
pip install -r https://raw.githubusercontent.com/ckan/ckanext-spatial/v2.1.1/requirements.txt
## ckan/ckckanext-harvest: https://github.com/ckan/ckanext-harvest/tags (e.g. v1.5.6)
pip install -e git++https://github.com/ckan/[email protected]#egg=ckanext-spatial
pip install -r https://raw.githubusercontent.com/ckan/ckanext-harvest/v1.5.6/requirements.txtcd $CKAN_VENV/src/
# Install the scheming_dataset plugin
pip install -e "git+https://github.com/ckan/ckanext-schemingdcat.git#egg=ckanext-schemingdcat"Set the plugin:
# Add the plugin to the list of plugins
ckan.plugins = ... spatial_metadata ... dcat ... schemingdcatWarning
When using schemingdcat extension,scheming should not appear in the list of plugins loaded in CKAN. But dcat and spatial should.
Set the schemas you want to use with configuration options:
# Each of the plugins is optional depending on your use
ckan.plugins = schemingdcat_datasets schemingdcat_groups schemingdcat_organizationsTo use CSW Endpoint in ckanext-schemingdcat:
schemingdcat.geometadata_base_uri = http://localhost:81/csw
ckanext.dcat.base_uri = http://localhost:81/catalogTo use custom schemas in ckanext-scheming:
# module-path:file to schemas being used
scheming.dataset_schemas = ckanext.schemingdcat:schemas/geodcatap/geodcatap_dataset.yaml
scheming.group_schemas = ckanext.schemingdcat:schemas/geodcatap/geodcatap_group.json
scheming.organization_schemas = ckanext.schemingdcat:schemas/geodcatap/geodcatap_org.json
# URLs may also be used, e.g:
#
# scheming.dataset_schemas = http://example.com/spatialx_schema.yaml
# Preset files may be included as well. The default preset setting is:
scheming.presets = ckanext.schemingdcat:schemas/geodcatap/geodcatap_presets.json
# The is_fallback setting may be changed as well. Defaults to false:
scheming.dataset_fallback = falseAdd the custom Harvesters to the list of plugins as you need:
ckan.plugins = ... spatial_metadata ... dcat ... schemingdcat ... harvest ... schemingdcat_ckan_harvester schemingdcat_csw_harvester ...You can update the endpoints.yaml file to add your custom OGC/LOD endpoints, only has 2 types of endpoints: lod and ogc, and the profile avalaible in ckanext-dcat Preferably between 4 and 8.
Examples:
-
LOD endpoint: A Linked Open Data endpoint is a DCAT endpoint that provides access to RDF data. More information about the catalogue endpoint, how to use the endpoint, (e.g.
https://{ckan-instance-host}/catalog.{format}?[page={page}]&[modified_since={date}]&[profiles={profile1},{profile2}]&[q={query}]&[fq={filter query}], and more atckanext-dcat- name: euro_dcat_ap_2_rdf display_name: RDF DCAT-AP type: lod format: rdf image_display_url: /images/icons/endpoints/euro_dcat_ap_2.svg description: RDF DCAT-AP Endpoint for european data portals. profile: euro_dcat_ap_2 profile_label: DCAT-AP version: null
-
OGC Endpoint: An OGC CSW endpoint provides a standards-based interface to discover, browse, and query metadata about spatial datasets and data services. More info about the endpoint at OGC: Catalogue Servicestandard/cat/
- name: csw_inspire display_name: CSW INSPIRE 2.0.2 type: ogc format: xml image_display_url: /images/icons/endpoints/csw_inspire.svg description: OGC-INSPIRE Endpoint for spatial metadata. profile: spain_dcat profile_label: INSPIRE version: 2.0.2
To configure facets, there are no mandatory sets in the config file for this extension. The following sets can be used:
schemingdcat.facet_list: [list of fields] # List of fields in scheming file to use to faceting. Use ckan defaults if not provided.
schemingdcat.default_facet_operator: [AND|OR] # OR if not defined
schemingdcat.icons_dir: (dir) # images/icons if not definedAs an example for facet list, we could suggest:
schemingdcat.facet_list = "theme groups theme_es dcat_type owner_org res_format publisher_name publisher_type frequency tags tag_uri conforms_to spatial_uri"The same custom fields for faceting can be used when browsing organizations and groups data:
schemingdcat.organization_custom_facets = true
schemingdcat.group_custom_facets = trueThis two last settings are not mandatory. You can omit one or both (or set them to false), and the default fields for faceting will be used instead.
-
Clear the index in solr:
ckan -c [route to your .ini ckan config file] search-index clear -
Modify the schema file on Solr (schema or managed schema) to add the multivalued fields added in the scheming extension used for faceting. You can add any field defined in the schema file used in the ckanext-scheming extension that you want to use for faceting. You must define each field with these parameters:
type: string- to avoid split the text in tokens, each individually "faceted".uninvertible: false- as recomended by solr´s documentationdocValues: true- to ease recovering faceted resourcesindexed: true- to let ckan recover resources under this facetstored: true- to let the value to be recovered by queriesmultiValued: well... it depends on if it is a multivalued field (several values for one resource) or a regular field (just one value). Use "true" or "false" respectively.
E.g.
ckanext-iepnbextension are ready to use these multivalued fields. You have to add this configuration fragment to solr schema in order to use them:<!-- Extra fields --> <field name="tag_uri" type="string" uninvertible="false" docValues="true" indexed="true" stored="true" multiValued="true"/> <field name="conforms_to" type="string" uninvertible="false" docValues="true" indexed="true" stored="true" multiValued="true"/> <field name="lineage_source" type="string" uninvertible="false" docValues="true" indexed="true" stored="true" multiValued="true"/> <field name="lineage_process_steps" type="string" uninvertible="false" docValues="true" indexed="true" stored="true" multiValued="true"/> <field name="reference" type="string" uninvertible="false" docValues="true" indexed="true" stored="true" multiValued="true"/> <field name="theme" type="string" uninvertible="false" docValues="true" indexed="true" stored="true" multiValued="true"/> <field name="theme_es" type="string" uninvertible="false" docValues="true" multiValued="true" indexed="true" stored="true"/> <field name="metadata_profile" type="string" uninvertible="false" docValues="true" multiValued="true" indexed="true" stored="true"/> <field name="resource_relation" type="string" uninvertible="false" docValues="true" indexed="true" stored="true" multiValued="true"/>
[!NOTE] You can ommit any field you're not going to use for faceting, but the best policy could be to add all values at the beginning.
The extra fields depend on your schema
Be sure to restart Solr after modify the schema.
-
Restart CKAN.
-
Reindex solr index:
ckan -c [route to your .ini ckan config file] search-index rebuild-fastSometimes solr can issue an error while reindexing. In that case I'd try to restart solr, delete index ("search-index clear"), restart solr, rebuild index, and restart solr again.
Ckan needs to "fix" multivalued fields to be able to recover values correctly for faceting, so this step must be done in order to use faceting with multivalued fields.
Icons for each field option in the scheming file can be set in multiple ways:
- Set a root directory path for icons for each field using the
icons_dirkey in the scheming file. - If
icons_diris not defined, the directory path is guessed starting from the value provided for theschemingdcat.icons_dirparameter in the CKAN config file, adding the name of the field as an additional step to the path (public/images/icons/{field_name). - For each option, use the
iconsetting to provide the last steps of the icon path from the field's root path defined before. This value may be just a file name or include a path to add to the icon's root directory. - If
iconis not used, a directory and file name are guessed from the option's value. - Icons files are tested for existence when using
schemingdcat_schema_iconfunction to get them. If the file doesn't exist, the function returnsNone. Icons can be provided by any CKAN extension in itspublicdirectory. - Set a
default iconfor a field using the default_icon setting in the scheming file. You can get it usingschemingdcat_schema_get_default_iconfunction, and it is your duty to decide when and where to get and use it in a template.
Update the base theme of CKAN to use with the enhancements of this extension.
With this plugin, you can customize the group, organization, and dataset entities in CKAN. Adding and enabling a schema will modify the forms used to update and create each entity, indicated by the respective type property at the root level. Such as group_type, organization_type, and dataset_type. Non-default types are supported properly as is indicated throughout the examples.
Are available to use with this extension a number of custom schema, more info: schemas/README.md
Schema Enhancements: We've made several improvements to our schema to provide a better metadata and metadata group management. Here are some of the key changes:
- Form Groups: We've introduced the use of
form_groupsand improveform_pagesin our schemas. This allows us to group related fields into the same form, making it easier to navigate and manage metadata. - Metadata Management Improvements: We've improved how we manage metadata in our schema. It's now easier to add, remove, and modify metadata, allowing us to keep our data more organized and accessible.
- Metadata Group Updates: We've made changes to how we handle metadata groups (
form_groups). It's now easier to group related metadata, helping us keep our data more organized and making it easier to find specific information.
For more details on these enhancements check Form Groups documentation, please refer to the schema files in ckanext/schemingdcat/schemas.
schemas/geodcatp_es with specific extensions for spatial data and GeoDCAT-AP/INSPIRE metadata profiles.
Note
RDF to CKAN dataset mapping: GeoDCAT-AP (ES) to CKAN
schemas/dcat based
on: DCAT.
Note
RDF to CKAN dataset mapping: DCAT to CKAN
schemas/dcatap based on: DCAT-AP for the european context.
Note
RDF to CKAN dataset mapping: DCAT-AP (EU) to CKAN
schemas/geodcatap based on: GeoDCAT-AP for the european context.
Note
RDF to CKAN dataset mapping: GeoDCAT-AP (EU) to CKAN
Form groups are a way to group related fields together in the same form. This makes it easier to navigate and manage metadata. A form group is defined with the following elements:
form_group_id: A unique identifier for the form group. For example,contact.label: A human-readable label for the form group. This can be provided in multiple languages. For example:label: en: Contact information es: Información de contacto
fa_icon: An optional Font Awesome icon that can be used to visually represent the form group. For example,fa-address-book.
Here is an example of a form group definition:
form_group_id: contact
label:
en: Contact information
es: Información de contacto
fa_icon: fa-address-bookFields can be added to a form group by specifying the form_group_id in the field definition. The form_group_id should match the form_group_id of the form group that the field should be part of.
Here is an example of a field that is part of the general_info form group:
field_name: owner_org
label:
en: Organization
es: Organización
required: True
help_text:
en: Entity (organisation) responsible for making the Dataset available.
es: Entidad (organización) responsable de publicar el conjunto de datos.
preset: dataset_organization
form_group_id: general_infoIn this example, the owner_org field will be part of the general_info form group.
In production, when gather and consumer processes are running, the following command are used to start and stop the background processes:
ckan harvester run: Starts any harvest jobs that have been created by putting them onto the gather queue. Also checks running jobs - if finished it changes their status to Finished.
To testing harvesters in development, you can use the following command:
-
ckan harvester run-test {source-id/name}: This does all the stages of the harvest (creates job, gather, fetch, import) without involving the web UI or the queue backends. This is useful for testing a harvester without having to fire up gather/fetch_consumer processes, as is done in production.[!WARNING] After running the
run-testcommand, you should stop all background processes forgatherandconsumerto avoid conflicts.
The plugin includes a harvester for remote CKAN instances using the custom schemas provided by schemingdcat and ckanext-scheming. This harvester is a subclass of the CKAN Harvester provided by ckanext-harvest and is designed to work with the schemingdcat plugin to provide a more versatile and customizable harvester for CKAN instances.
To use it, you need to add the schemingdcat_ckan_harvester plugin to your options file:
ckan.plugins = harvest schemingdcat schemingdcat_datasets ... schemingdcat_ckan_harvesterThe Scheming DCAT CKAN Harvester supports the same configuration options as the CKAN Harvester, plus the following additional options:
dataset_field_mapping/distribution_field_mapping(Optional): Mapping field names from local to remote instance, all info at: CKAN Harvester Field mapping structurefield_mapping_schema_version(Mandatory if existsdataset_field_mapping/distribution_field_mapping): Schema version of the field_mapping to ensure compatibility with older schemas. The default is2.schema(Optional): The name of the schema to use for the harvested datasets. This is theschema_nameas defined in the scheming file. The remote and local instances must have the same dataset schema. If not provided, thedataset_field_mapping/distribution_field_mappingis needed to mapping fields.allow_harvest_datasets(Optional): Iftrue, the harvester will create new records even if the package type is from the harvest source. Iffalse, the harvester will only create records that originate from the instance. Default isfalse.remote_orgs(Optional): [WIP]. Onlyonly_local.remote_groups(Optional): [WIP]. Onlyonly_local.clean_tags: By default, tags are stripped of accent characters, spaces and capital letters for display. Setting this option toFalsewill keep the original tag names. Default isTrue.
And example configuration might look like this:
{
"api_version": 2,
"clean_tags": false,
"default_tags": [{"name": "inspire"}, {"name": "geodcatap"}],
"default_groups": ["transportation", "hb"],
"default_extras": {"encoding":"utf8", "harvest_description":"Harvesting from Sample Catalog", "harvest_url": "{harvest_source_url}/dataset/{dataset_id}"},
"organizations_filter_include": ["remote-organization"],
"groups_filter_include":[],
"override_extras":false,
"user":"harverster-user",
"api_key":"<REMOTE_API_KEY>",
"read_only": true,
"remote_groups": "only_local",
"remote_orgs": "only_local",
"schema": "geodcatap",
"allow_harvest_datasets":false,
"field_mapping_schema_version":2,
"dataset_field_mapping": {
"title": {
"field_name": "my_title"
},
"title_translated": {
"languages": {
"en": {
"field_name": "my_title-en"
},
"es": {
"field_name": "my_title"
}
}
},
"private": {
"field_name": "private"
},
"tag_string": {
"field_name": ["theme_a", "theme_b", "theme_c"]
},
"theme_es": {
"field_value": "http://datos.gob.es/kos/sector-publico/sector/medio-ambiente"
},
"tag_uri": {
"field_name": "keyword_uri",
// "field_value" extends the original list of values retrieved from the remote file for all records.
"field_value": ["https://www.example.org/codelist/a","https://www.example.org/codelist/b", "https://www.example.org/codelist/c"]
},
"my_custom_field": {
// If you need to map a field in a remote dict to the "extras" dict, use the "extras_" prefix to indicate that the field is there.
"field_name": "extras_remote_custom_field"
},
},
}The dataset_field_mapping/distribution_field_mapping is structured as follows (multilingual version):
{
...
"field_mapping_schema_version": 2,
"<dataset_field_mapping>/<distribution_field_mapping>": {
"<schema_field_name>": {
"languages": {
"<language>": {
<"field_value": "<fixed_value>/<fixed_value_list>">,/<"field_name": "<excel_field_name>/<excel_field_name_list>">
},
...
},
...
},
...
}
}<schema_field_name>: The name of the field in the CKAN schema.<language>: (Optional) The language code for multilingual fields. This should be a valid ISO 639-1 language code. This is now nested under thelanguageskey.
<fixed_value>/<fixed_value_list>: (Optional) A fixed value or a list of fixed values that will be assigned to the field for all records.- Field labels: Field name:
<field_name>/<field_name_list>: (Optional) The name of the field in the remote file or a list of field names.
For fields that are not multilingual, you can directly use field_name without the languages key. For example:
{
...
"field_mapping_schema_version": 2,
"<dataset_field_mapping>/<distribution_field_mapping>": {
"<schema_field_name>": {
<"field_value": "<fixed_value>/<fixed_value_list>">,/<"field_name": "<excel_field_name>/<excel_field_name_list>">
},
...
}
}Important
The field mapping can be done either at the dataset level using dataset_field_mapping or at the resource level using distribution_field_mapping. The structure and options are the same for both. The field_mapping_schema_version is 2 by default, but needs to be set to avoid errors.
There are two types of fields that can be defined in the configuration:
- Regular fields: These fields have a field label to define the mapping or a fixed value for all its records.
- Properties: A field can have one of these three properties:
- Fixed value fields (
field_value): These fields have a fixed value that is assigned to all records. This is defined using thefield_valueproperty. Iffield_valueis a list,field_namecould be set at the same time, and thefield_valueextends the list obtained from the remote field. - Field labels: Field name:
- Name based fields (
field_name): These fields are defined by their name in the Excel file. This is defined using thefield_nameproperty, or if you need to map a field in a remote dict to theextrasdict, use theextras_prefix to indicate that the field is there.
- Name based fields (
- Fixed value fields (
- Properties: A field can have one of these three properties:
- Multilingual Fields (
languages): These fields have different values for different languages. Each language is represented as a separate object within the field object (es,en, ...). The language object can havefield_valueandfield_nameproperties, just like a normal field.
Example Here are some examples of configuration files:
- Field names: With
field_nameto define the mapping based on names of attributes in the remote sheet (my_title,org_identifier,keywords).
{
"api_version": 2,
"clean_tags": false,
...
# other properties
...
"field_mapping_schema_version": 2,
"dataset_field_mapping": {
"title": {
"field_name": "my_title"
},
"title_translated": {
"languages": {
"en": {
"field_name": "my_title-en"
},
"de": {
"field_value": ""
},
"es": {
"field_name": "my_title"
}
}
},
"private": {
"field_name": "private"
},
"theme": {
"field_name": ["theme", "theme_eu"]
},
"tag_custom": {
"field_name": "keywords"
},
"tag_string": {
"field_name": ["theme_a", "theme_b", "theme_c"]
},
"theme_es": {
"field_value": "http://datos.gob.es/kos/sector-publico/sector/medio-ambiente"
},
"tag_uri": {
"field_name": "keyword_uri",
// "field_value" extends the original list of values retrieved from the remote file for all records.
"field_value": ["https://www.example.org/codelist/a","https://www.example.org/codelist/b", "https://www.example.org/codelist/c"]
},
"my_custom_field": {
// If you need to map a field in a remote dict to the "extras" dict, use the "extras_" prefix to indicate that the field is there.
"field_name": "extras_remote_custom_field"
}
}
}###TODO: Scheming DCAT CSW INSPIRE Harvester
A harvester for remote CSW catalogues using the INSPIRE ISO 19139 metadata profile. This harvester is a subclass of the CSW Harvester provided by ckanext-spatial and is designed to work with the schemingdcat plugin to provide a more versatile and customizable harvester for CSW endpoints and GeoDCAT-AP CKAN instances.
To use it, you need to add the schemingdcat_csw_harvester plugin to your options file:
ckan.plugins = harvest schemingdcat schemingdcat_datasets ... schemingdcat_csw_harvesterA harvester for remote Google spreadsheets and Onedrive Excel files with Metadata records. This harvester is a subclass of the Scheming DCAT Base Harvester provided by ckanext-schemingdcat to provide a more versatile and customizable harvester for Excel files that have metadata records in them.
To use it, you need to add the schemingdcat_xls_harvester plugin to your options file:
ckan.plugins = harvest schemingdcat schemingdcat_datasets ... schemingdcat_xls_harvesterRemote Google Sheet/Onedrive Excel metadata upload Harvester supports the following options:
storage_type- Mandatory: The type of storage to use for the harvested datasets asonedriveorgspread. Default isonedrive.dataset_sheet- Mandatory: The name of the sheet in the Excel file that contains the dataset records.field_mapping_schema_version: Schema version of the field_mapping to ensure compatibility with older schemas. The default is2.dataset_field_mapping/distribution_field_mapping: Mapping field names from local to remote instance, all info at: Field mapping structurecredentials: Thecredentialsparameter should be used to provide the authentication credentials. The credentials depends on thestorage_typeused.- For
onedrive: The credentials parameter should be a dictionary with the following keys:username: A string representing the username.password: A string representing the password. - For
gspreadorgdrive: The credentials parameter should be a string containing the credentials inJSONformat. You can obtain the credentials by following the instructions provided in the Google Workspace documentation.
- For
distribution_sheet: The name of the sheet in the Excel file that contains the distribution records. If not provided, the harvester will only create records for the dataset sheet.datadictionary_sheet: The name of the sheet in the Excel file that contains the data dictionary records. If not provided, the harvester will only create records for the dataset sheet.api_version: You can force the harvester to use either version 1 or 2 of the CKAN API. Default is2.default_tags: A list of tags that will be added to all harvested datasets. Tags don't need to previously exist. This field takes a list of tag dicts which allows you to optionally specify a vocabulary. Default is[].default_groups: A list of group IDs or names to which the harvested datasets will be added to. The groups must exist in the local instance. Default is[].default_extras: A dictionary of key value pairs that will be added to extras of the harvested datasets. You can use the following replacement strings, that will be replaced before creating or updating the datasets:{dataset_id}{harvest_source_id}{harvest_source_url}Will be stripped of trailing forward slashes (/){harvest_source_title}{harvest_job_id}{harvest_object_id}
override_extras: Assign default extras even if they already exist in the remote dataset. Default isFalse(only non existing extras are added).user: User who will run the harvesting process. Please note that this user needs to have permission for creating packages, and if default groups were defined, the user must have permission to assign packages to these groups.read_only: Create harvested packages in read-only mode. Only the user who performed the harvest (the one defined in the previous setting or the 'harvest' sysadmin) will be able to edit and administer the packages created from this harvesting source. Logged in users and visitors will be only able to read them.force_all: By default, after the first harvesting, the harvester will gather only the modified packages from the remote site since the last harvesting Setting this property to true will force the harvester to gather all remote packages regardless of the modification date. Default isFalse.clean_tags: By default, tags are stripped of accent characters, spaces and capital letters for display. Setting this option toFalsewill keep the original tag names. Default isTrue.source_date_format: By default the harvester usesdateutilto parse the date, but if the date format of the strings is particularly different you can use this parameter to specify the format, e.g.%d/%m/%Y. Accepted formats are: COMMON_DATE_FORMATS
The dataset_field_mapping/distribution_field_mapping is structured as follows (multilingual version):
{
...
"field_mapping_schema_version": 2,
"<dataset_field_mapping>/<distribution_field_mapping>": {
"<schema_field_name>": {
"languages": {
"<language>": {
<"field_value": "<fixed_value>/<fixed_value_list>">,/<"field_name": "<excel_field_name>/<excel_field_name_list>">/< "field_position": "<excel_column>/<excel_column_list>">
},
...
},
...
},
...
}
}<schema_field_name>: The name of the field in the CKAN schema.<language>: (Optional) The language code for multilingual fields. This should be a valid ISO 639-1 language code. This is now nested under thelanguageskey.
<fixed_value>/<fixed_value_list>: (Optional) A fixed value or a list of fixed values that will be assigned to the field for all records.- Field labels: Field position or field name:
<field_position>/<field_position_list>: (Optional) The position of the field in the remote file, represented as a letter or a list of letters (e.g., "A", "B", "C").<field_name>/<field_name_list>: (Optional) The name of the field in the remote file or a list of field names.
For fields that are not multilingual, you can directly use field_name or field_position without the languages key. For example:
{
...
"field_mapping_schema_version": 2,
"<dataset_field_mapping>/<distribution_field_mapping>": {
"<schema_field_name>": {
<"field_value": "<fixed_value>/<fixed_value_list>">,/<"field_name": "<excel_field_name>/<excel_field_name_list>">/< "field_position": "<excel_column>/<excel_column_list>">
},
...
}
}Important
The field mapping can be done either at the dataset level using dataset_field_mapping or at the resource level using distribution_field_mapping. The structure and options are the same for both. The field_mapping_schema_version is 2 by default, but needs to be set to avoid errors.
There are two types of fields that can be defined in the configuration:
- Regular fields: These fields have a field label/position to define the mapping or a fixed value for all its records.
- Properties: A field can have one of these three properties:
- Fixed value fields (
field_value): These fields have a fixed value that is assigned to all records. This is defined using thefield_valueproperty. Iffield_valueis a list,field_nameorfield_positioncould be set at the same time, and thefield_valueextends the list obtained from the remote field. - Field labels: Field position or field name:
- Position based fields (
field_position): These fields are defined by their position in the Excel file. This is defined using thefield_positionproperty. - Name based fields (
field_name): These fields are defined by their name in the Excel file. This is defined using thefield_nameproperty.
- Position based fields (
- Fixed value fields (
- Properties: A field can have one of these three properties:
- Multilingual Fields (
languages): These fields have different values for different languages. Each language is represented as a separate object within the field object (es,en, ...). The language object can havefield_value,field_positionandfield_nameproperties, just like a normal field.
Example Here are some examples of configuration files:
-
Field positions: With
field_positionto define the mapping based on positions of attributes in the remote sheet (A,B,AA, etc.).{ "storage_type": "gspread", "dataset_sheet": "Dataset", "distribution_sheet": "Distribution", ... # other properties ... "field_mapping_schema_version": 2, "dataset_field_mapping": { "title": { "field_position": "A" }, "title_translated": { "languages": { "en": { "field_position": "AC" }, "de": { "field_value": "" }, "es": { "field_position": "A" } } }, "private": { "field_position": "F" }, "theme": { "field_position": ["G", "AA"], }, "tag_custom": { "field_position": "B" }, "tag_string": { "field_position": ["A", "B", "AC"] }, "theme_es": { "field_value": "http://datos.gob.es/kos/sector-publico/sector/medio-ambiente" }, "tag_uri": { "field_position": "Z", // "field_value" extends the original list of values retrieved from the remote file for all records. "field_value": ["https://www.example.org/codelist/a","https://www.example.org/codelist/b", "https://www.example.org/codelist/c"] }, } }- Field names: With
field_nameto define the mapping based on names of attributes in the remote sheet (my_title,org_identifier,keywords).
{ "storage_type": "gspread", "dataset_sheet": "Dataset", "distribution_sheet": "Distribution", ... # other properties ... "field_mapping_schema_version": 2, "dataset_field_mapping": { "title": { "field_name": "my_title" }, "title_translated": { "languages": { "en": { "field_name": "my_title-en" }, "de": { "field_value": "" }, "es": { "field_name": "my_title" } } }, "private": { "field_name": "private" }, "theme": { "field_name": ["theme", "theme_eu"] }, "tag_custom": { "field_name": "keywords" }, "tag_string": { "field_name": ["theme_a", "theme_b", "theme_c"] }, "theme_es": { "field_value": "http://datos.gob.es/kos/sector-publico/sector/medio-ambiente" }, "tag_uri": { "field_name": "keyword_uri", // "field_value" extends the original list of values retrieved from the remote file for all records. "field_value": ["https://www.example.org/codelist/a","https://www.example.org/codelist/b", "https://www.example.org/codelist/c"] }, } } - Field names: With
Important
All *_translated fields need their fallback non-suffix field as simple field, e.g:
...
"title": {
"field_position": "A"
},
"title_translated": {
"languages": {
"en": {
"field_value": ""
},
"es": {
"field_position": "A"
}
}
},
...To run the tests:
pytest --ckan-ini=test.ini ckanext/schemingdcat/tests
Footnotes
-
An improvement to [
ckanext-fluent] (https://github.com/ckan/ckanext-fluent) to allow more versatility in multilingual schema creation and metadata validation. ↩
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent
