CKAN crawler shouldn't attempt crawl user info

We currently crawl user info when crawl org. some ckan sources have ban this.

We seems not use this info anyway --- should just from the extra query parameter below:

Auto detect fields from CKAN package extra data array

The following fields:

• contact_point
• license_title
• spatial_coverage
• temporal_coverage_from
• temporal_coverage_to

are available on CKAN via plugin ckanext-dgu and might be available as "extra" data on CKAN instance without the plugin installed.

Those fields are critical for Magda's searching capability (especially, spatial_coverage, temporal_coverage and license_title).

The ckan connector's field mapping template should be updated to search for those fields from extra data as well.

Filter data via custom fields

Some data portal might create their own customised field through CKAN customisation.

e.g. this dataset https://catalogue.data.wa.gov.au/api/3/action/package_show?id=mining-tenements-dmirs-003
contains a custom field access_level, which we might need to filter data by.

This ticket is about creating a new feature to filter datasets harvested by custom fields.

Possible solution:

• Option 1: This generic proposal might solve this: magda-io/magda#3100
• Option 2: alternatively, we can work out a quick solution by filtering by ckan API query (needs to test if it's supports that custom field)

Update

We adopted solution option 1: magda-io/magda#3100

Recognise harvest_url & harvest_portal fields

Some clients may have external scripts running to aggregate data from external sources to their CKAN portal.

And store original portal source info as extra fields: harvest_url & harvest_portal fields:

e.g. this dataset (ckan package):

https://uatweb.datansw.links.com.au/data/api/3/action/package_show?id=f187bde0-c7c7-446d-bef0-a53ac2ba71d5

on Magda: https://uatweb.datansw.links.com.au/search/dataset/ds-nsw-ckan-f187bde0-c7c7-446d-bef0-a53ac2ba71d5/details

We need make ckan connector supports those fields so we can display the correct original source on Magda web UI

Investigate VIC portal harvesting issue

Seems VIC portal might respond 0 count for package_search request intermittently:

{"help":"https://discover.data.vic.gov.au/api/3/action/help_show?name=package_search","success":true,"result":{"count":0,"facets":{},"results":[],"sort":"id asc","search_facets":{}}}

The issue seems hard to fix from vic portal (seems a cache (section.io) issue)

We will retry the request when 0 count response is received in order to fix the issue.