bcgov / ckanext-bcgov

BC Data Catalogue source code, main ckan extension

ckanext-bcgov

This extension provides the basic customized features on the BC Data Catalogue, such as schema updates, theme changes, etc.

Ckan core and other required ckan extensions including python modules are included as part of requirement.txt.

Installation

1. Activate virtual environment, e.g.

   $ . /usr/lib/ckan/default/bin/activate
   

2. Install the extension. Switch to ckanext-bcgov extension directory and run the following command:

   python setup.py develop
   

3. Update the config file and add the following plugins to ckan.plugins list : edc_app edc_geo edc_ngeo edc_webservice edc_disqus.

4. Add the following lines to ini file to the search setting section if they don’t exist:

# solr related settings
search.facets.limit = 500
search.facets.default = 20
ckan.search.show_all_types = true
ckan.api_key = your-sysadmin-api-key

# licenses and sectors JSON files, e.g.:
licenses_group_url = https://${BCDC_LICENSE_API_ENDPOINT}/bcdc_licenses.json
sectors_file_url = https://${BCDC_LICENSE_API_ENDPOINT}/bcdc_sectors.json

# (optional) Environment name
edc.environment_name = MYDEVBOX

# Dashboard settings
bcgov.dashboard.api_url = https://argg.apps.gov.bc.ca/int/

# OFI Service endpoint
bcgov.ofi.api.public_url = https://apps.gov.bc.ca/pub/dwds-ofi
bcgov.ofi.api.secure_url = https://apps.gov.bc.ca/pub/dwds-ofi/secure
bcgov.ofi.api.convert_to_single_res = true

# POW Service Endpoints
bcgov.pow.public_url = https://apps.gov.bc.ca/pub/dwds-ofi
# Siteminder enabled POW is enabled.
bcgov.pow.secure_url = https://apps.gov.bc.ca/ext/dwds-pow
bcgov.pow.pow_ui_path = /jsp/dwds_pow_current_order.jsp?

# POW Settings
bcgov.pow.env = prod
bcgov.pow.past_orders_nbr = 5
bcgov.pow.custom_aoi_url = http://maps.gov.bc.ca/ess/hm/aoi/
bcgov.pow.persist_config = true
bcgov.pow.enable_mow = false
bcgov.pow.user_pow_ofi = true
bcgov.pow.order_source = bcdc

# POW Order Defaults
bcgov.pow.order_details.aoi_type = 0
bcgov.pow.order_details.aoi =
bcgov.pow.order_details.clipping_method_type_id = 1
bcgov.pow.order_details.ordering_application = BCDC
bcgov.pow.order_details.format_type = 3
bcgov.pow.order_details.csr_type = 4
bcgov.pow.order_details.item.metadata_url = https://catalogue.data.gov.bc.ca/dataset/

Note:

• For format_type see #400 (comment)
• For CSR types see https://apps.gov.bc.ca/pub/dwds-ofi/info/crsTypes

4. Update (or create) import.ini file inside ckanext-bcgov/ckanext/bcgov/scripts/config. Add api_key, site_url options (they should be the same as in your CKAN .ini file).

5. Create default vocabularies

   cd ckanext-bcgov/ckanext/bcgov/scripts
   $ python create_vocabs.py
   

Note: The following data files in ckanext-bcgov/ckanext/bcgov/scripts/data is required:

    edc-vocabs.json

6. Create organizations

   $ cd ckanext-bcgov/ckanext/bcgov/scripts
   $ python create_orgs.py
   

Note: The following data files in ckanext-bcgov/ckanext/bcgov/scripts/data is required:

    org_suborg_sector_mapping_forEDC.csv


Originally converted from SVN Source

Original Repo Copyright 2015, Province of British Columbia.

ckanext-bcgov's Issues

Singular (non-plural) word stems are not returned by search

When users search for a singular version of a word - e.g. home - the search does not return any records that include the stem of the word - e.g. homes.

demo.ckan.org (currently CKAN 2.5.2) does return records that include word stems .

Is this a SOLR version or configuration issue?

email notifications not being sent

In test and delivery email notifications are not being sent to organization's editors/admin when a record's state is moved to pending publish from draft.

Here is the log.

[pid: 24147|app: 0|req: 10/18] 142.34.140.32 () {48 vars in 752 bytes} [Thu Jul 7 13:32:29 2016] POST /api/3/action/resource_show => generated 1159 bytes in 62 msecs (HTTP/1.1 200) 4 headers in 130 bytes (1 switches on core 5)
u'

As a BC Data Catalogue admin for the Ministry of Jobs, Tourism and Skills Training - DataBC, please be advised that the publication state of this record: https://cat.data.gov.bc.ca/dataset/autotest-geographic269 is now "PENDING PUBLISH".

If you are no longer an admin for Ministry of Jobs, Tourism and Skills Training - DataBC or if you have a question or concern regarding this message please contact DataBC Catalogue Services [email protected] .

Thanks.

'
[pid: 24147|app: 0|req: 11/19] 142.34.140.32 () {58 vars in 2799 bytes} [Thu Jul 7 13:32:28 2016] POST /dataset/edit/autotest-geographic269 => generated 303 bytes in 1442 msecs (HTTP/1.1 302) 5 headers in 198 bytes (1 switches on core 2)
Exception in thread Thread-8:
Traceback (most recent call last):
File "/s00/lib64/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/s00/lib64/python2.7/threading.py", line 763, in run
self.__target(_self.__args, *_self.__kwargs)
File "/tmp/bcdcsrc/src/ckanext-bcgov/ckanext/bcgov/logic/action.py", line 234, in check_record_state
send_state_change_notifications(members, email_dict, site_title, site_url)
File "/tmp/bcdcsrc/src/ckanext-bcgov/ckanext/bcgov/logic/action.py", line 119, in send_state_change_notifications
raise MailerException("SMTP server does not support STARTTLS")
MailerException: SMTP server does not support STARTTLS

Rename resource storage location names or have this auto-populate

New editors often don't know what to select for 'Resource Storage Location'. Changing EDC Data Store to 'Catalogue File/Data Store' might be more helpful. Ideally, this should be automatic - if a file is uploaded this should be auto-populated. Same if the data is stored on an ftp site (external) or the link is set to pub.data.

update thin header footer style as the blog/engagement site

Using this page as a model ... http://engage.gov.bc.ca/govtogetherbc/ make the following changes to the UI:

1. Remove thin gold vertical bar next to sunrise
2. Remove search magnifying glass.
3. Make div id=header-main shorter
4. Only add the hamburger when page is in mobile view.
5. Drop the standard gov navigation IA from the hamburger menu - just move the catalogue (div.level2navigation and div.level3navigation content to the hamburger icon ).

ckan-2.3.4 patching

ga-report extension requires old oauth2client version (1.4.12)

Linked to CITZEDC-839

Underscores in resource filenames should be maintained when uploaded.

As a user, when I upload a file with underscores in the filename I would like them to be maintained.

An issue has been identified on core ckan/ckan#2597 however it appears to be dormant. Consider offering to address if original poster has not provided PR.

Authorization error page should tell users to login

Editors who don't know to login are notifying us that they can't access draft records. We should add a note to the https://cat.data.gov.bc.ca/authorization-error that asks editors to login in to access records in draft.

Application record template label update and field removal

CITZEDC-525

Review and implement patch 2.3.4

LInked to CITZEDC-842

March 31 patch release for 2.3
More information about CKAN releases can be found here:
http://docs.ckan.org/en/latest/maintaining/upgrading/index.html#ckan-releases

Parent organization cannot be determined from child organization

As a user, when I am viewing a child organization page (e.g. http://catalogue.data.gov.bc.ca/organization/enterprise-data-services ) I would like to be able to see the parent organization name and have a hyperlink to it, so that I know the context of the child organization and can quickly access the parent records.

Some example CKAN templates that implement the hierarchy extension include:

• Data.gov -
  has organized child organizations as searches that reference the parent organization - e.g. https://catalog.data.gov/dataset/food-access-research-atlas
• Data.gov.uk
  presents a hierarchy icon (top right) that presents a window to explore the organization (publisher) hierarchy - e.g. https://data.gov.uk/publisher/health-and-social-care-information-centre

layout bug in iPhone 5

extra blue stripe on the right side screen and class "icon-sitemap" should be in hamburger menu
see screen attached

server 500 error on unpublished dataset

linked to CITZEDC-863

update recline.js to use ESRI leaflet that consumes the BC tilecache

MapQuest changed its terms of use for the default leaflet tilecache. fork the reclineview extension and replace the leaflet code in recline.js with the ESRI leaflet code that consumes our own tilecache.

Email notifications being sent to removed users

Linked to CITZEDC-843

setup CI between GH and CAD

Remove mislabelled Data Created field from resource view page templates

On the resource view pages - ({domain}/dataset/dataset-name/resource/...) the field labelled Data Created - is actually the date the package/record was published. It should be removed from the resource view page templates.

resource_update for larger files results in duplicate rows to datastore resources

Using resource_update of the FileStore API to replace larger files results in duplicate rows in DataStore resources after DataPusher completes update.

See CITZEDC-819

add license badge

Remind users to login when challenged with Unauthorized Page

When some editors receive publication state change emails they forget they can login to see the record. The following change to the authorization notice page is recommended.

The prompt specified here

can be changed to...

Either you need to login to view this page or you are not authorized to access this page.
If you believe this is an error please contact
[email protected].

add Swagger UI console as an extension for API resource preview

CITZEDC-806

Estimate the effort to add Swagger UI Console as an extension that allows users to explore APIs that have Swagger definitions files. We would like to know the estimated effort to create an extension that would use the Swagger UI console within CKAN - possibly as an iFrame - to explore APIs that are added as resources. The assumption would be that we would identify the resource as a swagger JSON file format. To see an example of using the Swagger UI Console - look at this record for the BCDC API - http://catalogue.data.gov.bc.ca/dataset/bc-data-catalogue-api and select the second resource called API Console.

Day of week (& other date fields) recast as timestamps

ckan/datapusher#75

Keywords field isn't required to create new record

The keyword field (tags) is mandatory - and should require a minimum entry of one keyword. This validation is no longer working. It is now possible to navigate to the add data (resource) page without an entry in this field.

Original source: CITZEDC-821

Investigate timing for editors browsing organization view

Navigation by editors of the organization view (https://catalogue.data.gov.bc.ca/organization ) is slow.
Identify options for improving the timing to be closer to that provided to public/admin (<2 secs).

This is linked to CIZTEDC-874.

@Mbrownshoes will provide time statistics for various roles as a baseline.

Adjust footer height to be consistent with engagement style

Please match the height of #footerAdminSection to http://engage.gov.bc.ca/spillresponse/
(60)

Hyperlink from custodial agency of dataset record should reference organization

As a user, when I am viewing a dataset record and I select the hyperlink for the custodial agency (Published by the ...) I would like to be able to see a list of their datasets and have the opportunity to learn more about the publishing organization. The existing custodial agency hyperlink on the dataset page takes the user to the search page and adds a facet for the organization (e.g. http://catalogue.data.gov.bc.ca/dataset?organization=enterprise-data-services) If this hyperlink referenced the custodial agency page under the organizations tab (e.g. http://catalogue.data.gov.bc.ca/organization/enterprise-data-services ) the user will be able to discover more information about the custodial organization.

Stop email notifications when an editor has been removed from an organization

When an editor is removed from an organization using the organization manage feature (member_remove) that the individual should no longer receive notifications. Currently, the individual also needs to be deleted, but in some cases we have people who are editors in multiple organizations and we may want to remove them from one org but not delete them.

Right now a user needs to be 'non-active' before they stop receiving a notification, but ideally notifications would stop once they are removed as an editor for that org. We might still want them to be 'active', perhaps as an editor for another organization, but the notifications should stop for the org they have been removed from.

Improve email message sent on state changes

Linked to CITZEDC-862

Geographic data extraction API testing

Conduct testing against OFI (Order Fulfiller Interface) to confirm it meets requirements for use in next sprint. Provide feedback to TL by July 18.

Related tickets include CITZEDC-608, CITZEDC-732, CITZEDC-733,

Remove hyperlinks to dataset type icons, sectors and resource formats to improve UX

As a user I find the hyperlinks presented for each dataset in the datasets list / search pages ( http://catalogue.data.gov.bc.ca/dataset and http://catalogue.data.gov.bc.ca/organization/[orgname]) inconsistent and confusing.

1. When I click on a Dataset Type Icon, I am adding that as a facet filter to what I have filtered,
2. When I click on the sector name (e.g. "Natural Resources") I am applying a global search - wiping out what I have already added as filters,
3. When I click the resource type icons (w.g. WMS, CSV) - I am taken to the record.

To resolve this confusion and improve consistency - suggestion is to remove these hyperlinks and just ensure that hovertext is in place.

Hovertext for the Dataset Type icons:

• Tabular Record
• Geographic Record
• API Record
• Application Record

Hovertext for Sectors:

• 'Sector'

Hovertext for Format icons:

• 'Data Format'

Linked to https://apps.bcgov/int/jira/browse/CITZEDC-805

Auto-complete search should match facets and use relevancy as sort order

As a user, when I receive an auto-complete response from the search interface I expect that it will obey the facet filter displayed for the search box that it is launched from - and that it returns records according to relevancy sort order.

Currently the auto-complete disregards facet filters - so users are finding datasets that are not publicly downloadable.

When the auto-complete is used from a place where facets are not shown - it should also prefer records that have download public attribute.

CITZEDC-776

Recording and displaying last date of upload for filestore resources

As a user, I would like to know when the file resources were uploaded into the data catalogue so that I know how current they are. This should be displayed on the resource show page and possibly on the package page if it can fit.

Background:
The core resource_show API call indicates the following elements are present in the response last_modified: , cache_last_updated: ,* webstore_last_updated:* however they are all NULL.

Core CKAN - is also not populating these fields for a resource http://demo.ckan.org/api/3/action/resource_show?id=b36d7da2-8a9e-4912-8f2b-b58c29a847ef

UK is using Archiver and QA extensions and recording dates within nested objects
https://data.gov.uk/api/3/action/resource_show?id=2ac8abba-4a71-4f12-af1b-57ad0e36b6a4

There seems to have been lots of discussion about this issue - as it pertained to the DataStore - ckan/ckan#2519 - it would also be helpful to know when the DataStore was last updated for a resource. How is this determined?

Before we use one of the above referenced element to refer to the date the last time the file resource was uploaded we should confirm how these are used in the code.

Certainly the date of updates for resource record revisions are being tracked - although they may not be being displayed by the resource_show API call.

CITZEDC-808

CITZEDC-808

handle mixed content

opt1: force all https
opt2: force https only for logged in users.

Improve email notification regarding record state changes

Sometimes users added to an organization with an editor role are confused when they receive email notification messages from the application on record state changes. It would help editors and system administrators if the messages were more explicit.

Existing message Sample:

The following record is "Pending Publication" for Ministry of Advanced Education - Enterprise Data Services

Record https://.data.gov.bc.ca/dataset/record-title-goes-here, Record Title Goes Here

Please review and act as required.

A new message with the following form should be prepared:

As a BC Data Catalogue editor for the Ministry of Advanced Education - Enterprise Data Services, please be advised that the publication state of this record: https://.data.gov.bc.ca/dataset/record-title-goes-here is now "Pending Publication" .

If you are no longer an editor for Ministry of Advanced Education - Enterprise Data Services or if you have a question or concern regarding this message please contact DataBC Catalogue Services [email protected] .

Thanks.
.........................................................................................................

This is the code that generates the message.
https://github.com/bcgov/ckanext-bcgov/blob/b8a9ba0492697622199604bbf9824e783c6ab1b6/ckanext/bcgov/logic/action.py

Add a spinner that indicates to a user progress is underway - lock edit fields

When users are updating records or uploading resources they need to be aware of the time needed by the application to complete the task. Sometimes this leads to data corruption.

An indicator such as spin.js may reduce the chance that they will attempt to update when a transaction is in progress.

As a secondary investigate locking fields.

layout bug in iPhone 5

blue stripe on the right side page, also class "icon-sitemap" should be in hamburger menu. see screen attached.

resource/package title background blue needs match with header

footer doesn't stick to foot when for short page content

sample see about page

Test and Delivery are loading very slowly

When logged in as an editor in test or delivery and change the ‘Order by’ field from ‘published date’ the page is taking >10 s to load. This is not observed in Prod, where the page loads in ~2 s.

This is a critical issue with the 1.3 release. @gjlawran @ll911 @Darv72

Review production logs

Linked to CITZEDC-751

update the 'access only' license url

The url for the standard gov license changed. I've updated this and will submit a pull request.

test

Popular count is displaying total views instead of recent views.

The flames icon is displaying for any dataset with more than 10 views since launch - it is supposed to just display packages with more than 10 recent views. Where recent is defined as 14 days. CITZEDC-662

Fork geoview repo - update with openlayers - make pull request - and sync/clone

Linked to CITZEDC-814

new users unable to login to CAD

CITZEDC-876

add additional resource format for gml, gft, arcgis_rest

added

when duplicating a record, clicking 'update dataset' should lead to add resource page

As an Editor, when I duplicate a record I am not reminded to add a data resource - the system offers the 'Update Dataset' button instead of 'Next: Add Data' This can lead to 'ghost' records with no data resources. You should have to add a resource when duplicating.

change imap link to IMAPBC

Linked to CITZEDC-844

On the geographic dataset page template - modify the button text from "View in iMap" to "View in iMapBC"
View in iMap
View in iMapBC

Modifications needed for header

LInked to CITZEDC-854