openwis / openwis4

openwis4

openwis4's Issues

Harvesting Report 'Download Detailed Report' issue - not showing details of those that have Errors

From @woollattd on March 23, 2016 11:37

The option to 'Download Detailed Report' for a harvest task and a synchronisation task in the Admin Portal is a great addition, it covers what has been 'added'; 'updated' and 'removed' but... it is missing something vital...it doesn't include those that had 'errors'.
Having a list of all the errors included and the reason they failed would make it easier to chase the GISC concerned about their metadata.
The first 100 errors are shown in the report window but there is no user interaction available (no copy/paste) and not all errors will be shown if there are more than 100 of them.

It might also be a good idea to include the other headings from the Report window, such as:
Unkown Schema
OpenWIS Not Compliant
Does Not Validate
Ignored

These would be very useful to know as well.

Copied from original issue: OpenWIS/openwis#165

RSS/ATOM feed for metadata changes

From @lmika-bom on June 23, 2015 5:56

Brought up during discussions with user of OpenWIS that is consuming metadata to produce derivative work.

At the moment, OpenWIS has a notion of "Whats New": a display of metadata record IDs that have been recently added or changed. It might be useful to set-up a machine readable version of this list --- something that users can subscribe to trigger automated processes when metadata changes. A good technology for providing this feature would be an RSS or ATOM feed.

Consider adding such a feed to the user portal. It might be necessary to add a number of feeds based on what updates the user wants to receive:

- Metadata types (non-stopgap metadata, all metadata)
- Update operations (newly added metadata, updated metadata)

Copied from original issue: OpenWIS/openwis#77

New behaviour required for data received where there is no metadata record

From @6a6d74 on July 4, 2016 14:33

Discussion at ICT-ISS (29 June - 1 July 2016) covered how GISCs should respond when a data record (e.g. bulletin / message) is received but no associated metadata record can be found in the WIS DAR catalogue.

The Technical Regulations (WMO No. 1060) will be amended (pending approval at CBS16, November 2016) such that the GISC should send a notification to the originating centre that the metadata is missing.

The missing metadata statistics also need to be included in the data sent to the WIS Common Dashboard.

Dave Thomas (WMO) also noted that OpenWIS "doesn't behave right" according to the WIS Tech Regs. OpenWIS will create a "dummy metadata record" rather than rejecting the data record and creating a log-entry to this effect.

I'm not sure what the current behaviour actually is ...

Suggest reviewing this for OpenWIS 4.x

Copied from original issue: OpenWIS/openwis#198

BlackListingSeuilWarning : wrong parameter type (int => long) in blacklisting warning

From @jpjanecek-mto on December 15, 2016 15:23

Copied from original issue: OpenWIS/openwis#243

Improvements to the sub-selection parameters

From @lmika-bom on August 13, 2015 5:25

OpenWIS allows for the creation of a sub-selection parameter UI for a DCPC by calling a SOAP method on a harness (the sub-selection parameter harness). Although this is helpful, it has a few limitations. Before the adaptation of the GeoNetwork 2.11, it might be worth looking into making some of the following improvements:

1. Selecting geographic region that crosses the anti-meridian: the sub-selection parameter harness can show a map for the user to select a geographical region. At the moment, the user is unable to declare a region that crosses the anti-meridian. In certain cases, it might be necessary for users to do so, particularly users near the anti-meridian, so it would be nice to allow for this.
2. Lat/Long entry boxes for the map field: instead of (or in addition to) item 1, it would be nice to include some entry boxes with map fields which will allow the user to specify the N, S, E and W point of the selected geographic region. That way, if the user knows the exact boundaries of the region they wish to extract, they could simply enter the boundary as decimal degrees (for example) instead of zoom and pan the map to the region of interest before selecting it by dragging the mouse.
3. Full support for cross field validation: (this might be a hard one to do and will probably involve some changes to the harness API). At the moment, the sub-selection parameter harness can define a limited form of validation in which the valid choices for a particular field are dictated by the choices made in the field immediately above it. In cases where a more sophisticate validation scheme is required, it would be nice for fields to use any other sub-selection field when running validation checks.

Copied from original issue: OpenWIS/openwis#85

58. v4 COM - OpenWIS version 4.1 development release package

OpenWIS version 4.1 development release:

Logout link not working correctly

The logout action has been modified in CGN 3.2.0 as mentioned here:

• API / Login and logout are now /signin and /signout instead of j_spring_security_(check|logout).

The OW4 code uses the old URL which is no longer available. It should be updated

77. v4 RET - DDS1 Download offer

The migrated version only shows the ‘subscribe’ action in the sample entries of the Search screen. It is possible that the metadata itself needs to have specific characteristics in order for the ‘Deliver’ and ‘Download’ actions to become available. Further investigation is required.

Furthermore, it is very important to point out that GN v3.2.0 appears to feature its own download mechanism. This shows a Download button in the UI but not in the same place as OpenWIS. It is possible that OpenWIS would like to integrate this functionality into its own delivery methods.

This task was part of the OpenWISv4/GeoNetworksv3 work package assigned to GeoCat in 2015. The documentation produced as part of that work describes the new functionality delivered.

The outcome for this feature was:

2015 status: MO Test; Percent complete: 100; Priority: Normal; Questions outstanding: nil;

Extra info:

GeoNetwork shall be amended such that, when the conditions outlined below are met, a user is presented with the opportunity to download or subscribe to the dataset described by the metadata record they are viewing.

The user may subscribe to the dataset under the following conditions:

The user has the necessary download permissions for the dataset
The dataset is managed by the OpenWIS Data Service (either cache or a local data service)- as determined via attributes from the summary record within the Product Metadata Table1.
A ‘subscribe’ option shall be provided by the portal enabling a new subscription to be requested (see enhancement DDS2 below).

Furthermore, the user may download or request immediate delivery of the dataset if the following conditions are also met:

The user is not blacklisted 2(e.g. they have not already exceeded their daily download threshold)
Data product instances are currently available3.
Options ‘deliver’ (see enhancement DDS3 below) and ‘download’ (see enhancement DDS4 below) shall be provided by the portal.

88. v4 ACC - User right access on metadata editor

See OpenWIS/openwis/issue#73

LM - Please confirm this works in GeoNetworks 3. If so, close this ticket.

PR - It would be useful if MF, who have real users, could mock-up their v3 user population on a v4 instance and then check they can assign rights properly, and that they work as they think they should (see issue #73). This will have to wait until we have a v4 instance worth testing. For now, I will park this under TEST and assign to MF for later when an instance is available.

Removing harvesting and keep associated metadata

From @benjaminsaclier on December 17, 2015 13:49

At present time when a harvesting task is removed all the metadata which are "linked" to this harvesting task are removed also.
It would be better if administrator have the choice when removing a harvesting task to keep or not the metadata.
If administrator decides to keep the metadata then the metadata table should be updated for all associated MD for the following fields:
'isharvested' => 'n'
harvestingtask => 'NULL'

Benjamin

Copied from original issue: OpenWIS/openwis#118

46. v4 DIS - MM3 Metadata categories and category-based harvesting

This task was part of the OpenWISv4/GeoNetworksv3 work package assigned to GeoCat in 2015. The documentation produced as part of that work describes the new functionality delivered.

The outcome for this feature was:

2015 status: MO Test; Percent complete: 100; Priority: Normal; Questions outstanding: nil;

Extra info:

In order to provide greater control in the harvesting of metadata between WMO Information System Centres (WIS Centres), OpenWIS provides the facility to group metadata records into named subsets, or categories. Each category of metadata records can be offered for harvesting by a third party as a distinct subset of the total set of metadata records within the OpenWIS catalogue.

GeoNetwork shall be amended such that:

A metadata record may be assigned to a category
Metadata harvesting can be configured for each metadata category; e.g. specific metadata harvesting configurations can be specified for the categories of metadata, enabling OpenWIS to expose named subsets of metadata for harvesting by third parties.
The GeoNetwork role Administrator can define new categories.
The GeoNetwork role Administrator can modify category assignment of published metadata records1
The GeoNetwork role User Administrator can assign the permission to use categories to specific Groups and define a default category for each Group.
The GeoNetwork role Editor can assign a metadata record that they have permission to modify to a specific category.
Note that permissions are assumed to cascade in the order: Guest (non-authenticated user), Registered User, Editor, Content Reviewer, User Administrator and Administrator. For example, a user with role Content Reviewer can perform all the actions assigned to a user with role Editor plus a few more.

79. v4 RET - DDS4 Direct download via HTTP

See the other DDS tasks.

This task was part of the OpenWISv4/GeoNetworksv3 work package assigned to GeoCat in 2015. The documentation produced as part of that work describes the new functionality delivered.

The outcome for this feature was:

2015 status: MO Test; Percent complete: 60; Priority: Normal; Questions outstanding: 100% OR 60% Done?;

Extra info:

GeoNetwork shall be amended such that a user with the appropriate permissions may download data product instances that are currently available from the OpenWIS Data Service directly from the portal via HTTP.

The portal will provide a web-based user interface enabling a user to configure their download request for the specified dataset; allowing them to select specific data product instances from the currently available setError: Reference source not found and specify how these are to be packaged for download (e.g. as a single compressed file, or concatenated according to a WMO-specified scheme). Direct downloads are then processed identically to ad-hoc requests; the portal will package the user supplied information and the relevant product metadata record (retrieved via the ProductMetadataService SOAP web-service) into an ad-hoc request object and send it for validation and processing to the OpenWIS Data Service using the createRequest operation from the RequestService SOAP web-service.

Note that by re-using the existing services for ad-hoc requests, all the necessary usage statistics and download history are automatically gathered for that user. The absence of dissemination information within the request object means that no further action will be taken within the OpenWIS Data Service once the data products have been extracted, packaged and moved to the Staging Post.

The asynchronous nature of the ad-hoc request process means that it is not straightforward to determine when the packaged data products have been moved to the Staging Post ready for download. For the purposes of this technical design the monitorExtraction operation of the ProcessedRequestService SOAP web service can be used to poll for completion. Once the ad-hoc request process has completed, the portal will be updated to provide a hyperlink to the location of the packaged data products within the Staging Post. The URL is determined by concatenating the Staging Post URL (retrieved from configuration) and the URI for the data product package (retrieved using the getUri() operation on the processed request object returned from the getProcessedRequest or getProcessedRequestForAdhoc operations from the ProcessedRequestService SOAP web-service).The user is then able to download the packaged data products.

OpenWIS shall notify the relevant Groups that the dataset has been downloaded according to the notify permission defined for the metadata record associated with the downloaded dataset.

Remove FTPReplication Tool

From @lmika-bom on November 22, 2015 21:8

It was decided during the TC teleconference on 20 Nov 2015 that the FTP Replication tool included in OpenWIS is no longer required. Investigate and remove this tool from the project.

Copied from original issue: OpenWIS/openwis#105

51. v4 DIS - DM1 Default metadata insertion

This task was part of the OpenWISv4/GeoNetworksv3 work package assigned to GeoCat in 2015. The documentation produced as part of that work describes the new functionality delivered.

The outcome for this feature was:

2015 status: MO/GeoCat Review; Percent complete: 100; Priority: Normal; Questions outstanding: Does the status remain UNPUBLISHED?;

Extra info:

GeoNetwork shall be amended such that a software agent is able to insert a metadata record into the catalogue wherein it is associated with a specific Group and its status remains unpublished.

Feature: Discovery

52. v4 DIS - MM4 Maintain product metadata table

This task was part of the OpenWISv4/GeoNetworksv3 work package assigned to GeoCat in 2015. The documentation produced as part of that work describes the new functionality delivered.

The outcome for this feature was:

2015 status: MO/GeoCat Review; Percent complete: 0; Priority: High; Questions outstanding: 0% Done? There is a conversation with Jeremy and Leon in Redmine, but not sure how it concluded;

Extra info:

This enhancement is to be implemented as a complementary component to GeoNetwork as the functionality described herein is specific to OpenWIS. Implementation of this enhancement should be loosely coupled with the GeoNetwork code base enabling the source code for this enhancement to be maintained independently. It is highly desirable to be able to bind the capabilities described by this enhancement within GeoNetwork using deployment-time configuration rather than at compilation-time.

The OpenWIS Data Service holds a summary record for each metadata record in the catalogue. This information is stored in the Product Metadata Table and drives the processing of the Data Service on arrival and dissemination of data product instances.

The definition of the summary record, expressed as a complex type within XML Schema, is provided below:

...
Information for the summary record is located within the main metadata record according to XPath statements.

When a new metadata record is added to the OpenWIS catalogue, a corresponding entry in the Product Metadata Table shall be added using the createProductMetadata operation of the ProductMetadataService SOAP web service.

When an existing metadata record is updated within the OpenWIS catalogue, the corresponding entry in the Product Metadata Table shall be updated using the updateProductMetadata operation of the ProductMetadataService SOAP web service.

When an existing metadata record is deleted within the OpenWIS catalogue, the corresponding entry in the Product Metadata Table shall be removed using the deleteProductMetadata, deleteProductMetdataByURN or deleteProductMetadatasWithURN operations of the ProductMetadataService SOAP web service. The latter operation provides for bulk removal.

Note that the obsoleting of subscriptions associated with a deleted metadata record is undertaken by the OpenWIS Data Service.

45. v4 SYS - MM2 Limit on OAI-PMH metadata harvest response

This task was part of the OpenWISv4/GeoNetworksv3 work package assigned to GeoCat in 2015. The documentation produced as part of that work describes the new functionality delivered.

The outcome for this feature was:

2015 status: MO Test; Percent complete: 90; Priority: Normal; Questions outstanding: nil;

Extra info: nil.

44. v4 SYS - ADMIN4 Monitoring and statistics

This task was part of the OpenWISv4/GeoNetworksv3 work package assigned to GeoCat in 2015. The documentation produced as part of that work describes the new functionality delivered.

The outcome for this feature was:

2015 status: MO/GeoCat Review; Percent complete: 90; Priority: Normal; Questions outstanding: 100% or 90%. Are the System information and statistics exportable in a machine readable form?;

Extra info:

GeoNetwork shall be amended to display the system information and statistics available via the MonitoringService SOAP web-service. System information and statistics should be exportable in a machine readable form.

Write recommended hardware specs doc

From @rogers492 on October 26, 2016 20:44

Assuming vagrant

Copied from original issue: OpenWIS/openwis#225

Need a better timeout mechanism on socketRead

From @ywang-bom on August 10, 2015 0:9

Synchronistation jobs can sometimes hang for unreasonably long time (serveral days). It would be helpful to implement a better timeout mechanism to terminate the job earlier and allow it to have a chance to restart in the next schedule.

Since the jobs are scheduled to run every 24 hours, the upper-limit of the timeout shall not be longer than that. In practice, if no data is comming from a connection in the first few minutes, it is very likely the data will not come at all.

The stack dump is as follows:

"pool-7-thread-4" prio=10 tid=0x086aa800 nid=0x3327 runnable [0xa36d1000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
        - locked <0xbd2f8af8> (a java.io.BufferedInputStream)
        at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:77)
        at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:105)
        at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1115)
        at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1832)
        at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1590)
        at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:995)
        at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397)
        at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
        at org.fao.oaipmh.requests.Transport.doExecute(Transport.java:257)
        at org.fao.oaipmh.requests.Transport.execute(Transport.java:194)
        at org.fao.oaipmh.requests.AbstractRequest.sendRequest(AbstractRequest.java:92)
        at org.fao.oaipmh.requests.ListRecordsRequest.execute(ListRecordsRequest.java:63)
        at org.fao.oaipmh.requests.ListRecordsRequest.execute(ListRecordsRequest.java:37)
        at org.openwis.metadataportal.kernel.harvest.oaipmh.OaipmhHarvester.harvest(OaipmhHarvester.java:156)
        at org.openwis.metadataportal.kernel.harvest.exec.HarvesterRunnable.run(HarvesterRunnable.java:78)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:662)

Copied from original issue: OpenWIS/openwis#84

Email Dissemination issue - embedded in body option needs to be removed

From @woollattd on March 18, 2016 10:19

This option has never worked in OpenWIS. It also isn't a good idea if it did since it would be posssible to set this for .bin files
it is better to just leave the ability to attach a file.

Copied from original issue: OpenWIS/openwis#158

User right access on metadata editor

From @benjaminsaclier on May 12, 2015 10:7

At present time an openwis user which have the permission to edit metadata has the ability to:
1/ add a new metadata in any set of catalogue
2/ move a metadata (which is owner) to any other set
3/ add a new metadata using any uniform resource names (urn).
This behavior is not compliant with WMO and is a risk on metadata management on Openwis.

The proposition of evolution is the following :
1/ Have the ability to associate an openwis user/editor to a specific set, in that case the user can only add/modified/delete metadata from this set.
2/ Have the possibility to specific urn regulars expressions for a specific user, in that case the user can only add/modified/delete metatada where urn matching the regular expression, example the following regulars expressions only authorize a user to work on Meteo-France metadata:
"x-wmo:bulletin:int.wmo.wis::......LFPW$"
":x-wmo:bulletin:int.wmo.wis::[TAZW].*_C_LFPW"

Benjamin

Copied from original issue: OpenWIS/openwis#73

need a way to update subscription delivery addresses for multiple subscriptions for a user in bulk

From @woollattd on November 29, 2016 10:15

I have taken a look on the User Portal and don't see a method for a user to update destination address for multiple subscriptions, other than opening each subscription and modifying the destination manually.
This is very non user friendly since it is possible (expected) to have scores of subscriptions.
Hopefully we can make sure this is possible in OpenWIS v4.x

Copied from original issue: OpenWIS/openwis#237

187. v4 ACC - Data access policies

LM will look at whether we can just port the OpenWIS 3.14 Data Services policies to OpenWIS4 and integrate them with CGNv3.

49. v4 SYS - ADMIN6 Browse subscriptions

This task was part of the OpenWISv4/GeoNetworksv3 work package assigned to GeoCat in 2015. The documentation produced as part of that work describes the new functionality delivered.

The outcome for this feature was:

2015 status: MO/GeoCat Review; Percent complete: 80; Priority: Normal; Questions outstanding: Is this 100% Done or 80% Done? No bulk edit actions?;

Extra info:

GeoNetwork shall be amended to provide a User Administrator with a web-based mechanism to:

browse the subscriptions for all users or users within a specific Group; and
suspend and reactivate specific subscriptions.
The SubscriptionService SOAP web-service provides for retrieval and persistence of the subscription objects.

Cannot create metadata: OpenWIS derives an empty string as the value for "GTS Category"

From @ywang-bom on November 18, 2014 2:11

NOTE: This bug concerns metadata records NOT describing data for global exchange, i.e. the data do NOT have a corresponding WMO Data License Code (e.g. WMOEssential).

OpenWIS refuses to create a metadata record because the system derives an empty string as the "GTS Category" from the given XML file.
The system reports error "Unable to extract OpenWIS information" shown as follows:

Steps to reproduce

1. Login to the user portal with an account that has editor right.
2. Navigate to "My Account" -> "Metadata Service" -> "Insert Metadata"
3. Choose the given test metadata for upload ( the content of the test metadata can found in the first comment to this ticket, simply copy and paste to create the test file)
4. Select "XSD only" from the "Validate" drop-down menu
5. Click "Insert"
6. The error report should be generated as shown above.

Analysis
From our investigation, OpenWIS has a set of logic to derive the value of "GTS Category" for a given XML file. The relevant method is extractGTSCategoryGTSPriorityAndDataPolicy in ProductMetadataISO19139Extractor.java. (The "GTS Category" corresponds to the "gts_category" column of "openwis_product_metadata" table in database)

The logic to derive "GTS Category" from a given file is list as follows:

1. Search for the existence of following two elements
   * //gmd:identificationInfo//gmd:resourceConstraints//gmd:useLimitation
   * //gmd:identificationInfo//gmd:resourceConstraints//gmd:otherConstraints
2. If none of the above two XML elements are found, set the "GTS Category" to Unknown and return.
3. If elements are found, loop through all elements and try to find match for WMO Data License Code (e.g. WMOEssential).
4. If WMO Data License Code is found, use it as the "GTS Category".
5. If WMO Data License Code is NOT found (source of the error), set "GTS Category" to the content of the last element in the loop.

Relevant config file is openwis-metadataportal.properties, which instructs OpenWIS to use the above two elements to derive "GTS Category".

The error is generated when:

1. The given metadata is NOT for global exchanged data (hence no WMO Data License code can be found)
2. Both elements,gmd:useLimitation and gmd:otherConstraints, can be found in the file.
3. gmd:useLimitation is an empty element (i.e. <gmd:useLimitation gco:nilReason="missing"> <gco:CharacterString /> </gmd:useLimitation>)
4. gmd:useLimitation is the last element in the loop.

The method extractGTSCategoryGTSPriorityAndDataPolicy will then set "GTS Category" to be an empty string, which is exactly the content of gmd:useLimitation. This in turn triggers the reported error generated by ensureProductMetadataValid in MetadataAligner.java.

Possible Fix
A possible fix is to check the "GTS Category" value at the end of the function when the given XML is NOT for global exchanged data. If the value is an empty string, simply set it to "Unknown" similar to step 2 in above logic.

Copied from original issue: OpenWIS/openwis#32

Metadata synchronisation: Partial Update v.s. ACID

From @ywang-bom on September 1, 2015 4:38

In the process of an OAI-PMH sychronisation task, the local metadata records are updated page by page (the page size is controlled by the server's OAI-PMH configuration). If the process fails in the middle, we are left with a partially updated local metadata set (and how partially is somewhat unknown).

If the whole process is ACID, which means any failure would cause any changes to roll back, we will then guaranteed to have an unchanged metadata set.

One may argue that the partially update could possibly reduce the running time of the next try of the same task. However if sever communication is the most time consuming part, the partially update does not help save much time because the OAI query has to be re-issued with the same --from value. Additionally, since the current metadata update logic is to update everything from server without any timestamp checking, this means no time can be saved for updating local records either. So there is little or no benefit to have a partial update. An ACID approach is pobably better as it is more predictable.

Copied from original issue: OpenWIS/openwis#90

43. v4 SYS - ADMIN3 Browse OpenWIS cache content

This task was part of the OpenWISv4/GeoNetworksv3 work package assigned to GeoCat in 2015. The documentation produced as part of that work describes the new functionality delivered.

The outcome for this feature was:

2015 status: MO/GeoCat Review; Percent complete: 100; Priority: Normal; Questions outstanding: Can we extract specific data products from the cache?;

Extra info:

GeoNetwork shall be amended to provide an Administrator with a web-based mechanism to:

browse the content of the global cache; and
extract specific data products from the cache.
The CacheIndexService SOAP web-service provides a mechanism to interact with the cache.

References to RMDCN should only apply if available

From @woollattd on March 18, 2016 10:38

references to RMDCN on the ‘Favorites’ tab in User Information section on User Portal should only appear if enabled in the Data Policy settings and the User belongs to that group.
also what is RMDCN Email? maybe that option should be removed from here and the Data Policy settings.

Copied from original issue: OpenWIS/openwis#160

42. v4 SYS - ADMIN2 Browse product metadata table

This task was part of the OpenWISv4/GeoNetworksv3 work package assigned to GeoCat in 2015. The documentation produced as part of that work describes the new functionality delivered.

The outcome for this feature was:

2015 status: MO/GeoCat Review; Percent complete: 70; Priority: High; Questions outstanding: 100% or 70% Done?;

Extra info:

Geonetwork shall be amended to provide an Administrator with a web based mechanism to:

browse the content of the product metadata table; and
for specific records, override attributes (e.g. dissemination priority) that have been extracted from the associated metadata record in the catalogue.
The ProductMetadataService SOAP web-service provides a mechanism to interact with the product metadata table.

53. v4 RET - DDS2 New subscription

The migrated version only shows the ‘subscribe’ action in the sample entries of the Search screen. It is possible that the metadata itself needs to have specific characteristics in order for the ‘Deliver’ and ‘Download’ actions to become available. Further investigation is required.

Furthermore, it is very important to point out that GN v3.2.0 appears to feature its own download mechanism. This shows a Download button in the UI but not in the same place as OpenWIS. It is possible that OpenWIS would like to integrate this functionality into its own delivery methods.

This task was part of the OpenWISv4/GeoNetworksv3 work package assigned to GeoCat in 2015. The documentation produced as part of that work describes the new functionality delivered.

The outcome for this feature was:

2015 status: MO Test; Percent complete: 100; Priority: Normal; Questions outstanding: nil;

Extra info:

GeoNetwork shall be amended such that a user with the appropriate permissions may set up a new subscription requesting routine delivery of data products from the OpenWIS Data Service.

The portal will provide a web-based user interface enabling a user to set up their subscription1; allowing them to specify details of their primary and secondary dissemination channels and how individual data product instances are packaged for delivery. On completion of the subscription details, the portal will package the user supplied information and the relevant product metadata record (retrieved via the ProductMetadataService SOAP web-service) into a subscription request object and send it for validation and processing to the OpenWIS Data Service using the createSubscription operation from the SubscriptionService SOAP web-service.

OpenWIS shall notify the relevant Groups that a new subscription has been created according to the notify permission defined for the metadata record associated with the dataset that has been subscribed to.

54. v4 RET - UI1 Subscription management

This task was part of the OpenWISv4/GeoNetworksv3 work package assigned to GeoCat in 2015. The documentation produced as part of that work describes the new functionality delivered.

The outcome for this feature was:

2015 status: MO/GeoCat Review; Percent complete: 100; Priority: Normal; Questions outstanding: In the manual, does ‘redirect the user to the Users and Groups section of the admin console’ really mean that, or just that identical functionality exists in the User Portal and Admin Portal?;

Extra info:

GeoNetwork shall be amended to provide users with a web-based mechanism to review and manage (create, update and delete) their subscriptions for routine delivery of data. The SubscriptionService SOAP web-service provides for retrieval and persistence of the subscription objects.

Feature: Retrieval

add ability to export all metadata from a category as a single MEF file

From @woollattd on April 30, 2015 9:59

It would be extremely useful to backup a whole catageory of metadata in a single MEF.
For Support it would be enormously helpful should an update need a full re-install of the system

currently it is only possible to export metadata as a MEF from the Admin Portal for a single page of search results (20 files) where there could be hundreds of pages covering all the results for a category

From the example above - This would lead to creating over 857 MEF files holding only 20 files each

Copied from original issue: OpenWIS/openwis#69

59. v4 XXX - Adoption of GeoNetwork 3 functions in OpenWIS 4.

This assessment will include assessing the role of categories in CGN and OpenWIS, including Kanban item 97 (role of category in metadata sync). This assessment will also include Kanban item 62 (delegation of management of Users and Groups to GeoNetwork - an alternative to using OpenAM).

We will not publish the demo system urls more widely until we see if the GeoNetwork project releases 3.2.1 on 26th Jan, as their milestone on GitHub suggests they are aiming to do. If that release is on time and includes many bug fixes, we will upgrade the demo instances before publishing them more widely.

62. v4 ACC - Delegation of management of Users and Groups to Geonetwork

This assessment will include assessing the role of categories in CGN and OpenWIS, including Kanban item 97 (role of category in metadata sync). This assessment will also include Kanban item 62 (delegation of management of Users and Groups to GeoNetwork - an alternative to using OpenAM).

This is related to Kanban item 55 v4 - AAC1 SAML2, in that these are alternatives. Some organisations will want SSO whereas some will want a local OpenWIS authN and authZ. Once they have decided on a method then they will likely stick with it.

v3.14.2 - Admin Portal reports fail when exporting

From @woollattd on March 1, 2016 8:43

The export function on the Admin Portal pages that allow this seem to have trouble accessing the relevant xml file and causes an error.

Data Service - Requests Statistics page
for Volume of data disseminated per day and per user:
/usr/share/tomcat/webapps/openwis-admin-portal/../../data/tmp/disseminated-users-statistics.xml (No such file or directory)

for Overall volume of data disseminated and extracted per day:
/usr/share/tomcat/webapps/openwis-admin-portal/../../data/tmp/disseminated-statistics.xml (No such file or directory)

Data Service - Cache statistics page
for Volume of data ingested per day:
/usr/share/tomcat/webapps/openwis-admin-portal/../../data/tmp/ingestion-statistics.xml (No such file or directory)

Alarms - Global Reports page
for Volume of data ingested per day:
/usr/share/tomcat/webapps/openwis-admin-portal/../../data/tmp/ingestion-statistics.xml (No such file or directory)

Metadata - Catalogue Statistics page
/usr/share/tomcat/webapps/openwis-admin-portal/../../data/tmp/catalog-statistics.xml (No such file or directory)

Some reports do export ok though
Alarms - Global Reports - Volume of data disseminated per day
Alarms -Global Reports - Overall volume of data disseminated and extracted per day
Backup - Availability Statistics of local system - both exports work ok.

so some xml are being produced and can be found
Has something broken along the way?

Copied from original issue: OpenWIS/openwis#152

Harvesting issue - v3.14.5 not downloading metadata from DWD - error - cannot find namespace to set for element gmd:MD_Metadata

From @woollattd on March 23, 2016 11:56

OpenWIS is having trouble ingesting metadata from DWD.
is OpenWIS unable to accept the German metadata with this namespace or is the German metadata incorrect?
I'm not sure that the metadata is incorrectly formatted as other GISCs seem to download the correct number of metadata from their catalogue. (seen here: http://oai.dwd.de/oaimonitorgui/#complete)

I also notice that there are discrepancies between the versions of OpenWIS I have available to harvest to. So I suspect it is OpenWIS that has the problem.

The harvest from DWD on v3.14.4 in our Dev environment harvests 34,139 records
The harvest from DWD on v3.10 in our Production environment harvests 34,756 records
The harvest from DWD on v3.12 in our Test environment harvests 34,806 records

What they should have is 34,802 records (as per DWD page)

why is there a difference?
from what I can see - v3.14.4 has errors on more records than v3.10 and v3.12 - why?
they all should have the same number of errors shouldn't they?
Has something changed in validation since v3.10 and v3.12?

there seems to be two types of errors:
No Prefix ..... (see image below)
and
Unable to extract an URN ...
see image

Copied from original issue: OpenWIS/openwis#166

127. v4 RET - Extend the Dissemination Harness.

PN is currently assessing the state of the UKMO Harness on 3.14 and DW will do some testing. It is likely that we will need to tidy up the Harness code and fix a few minor bugs in preparation for any major changes we decide to introduce for v4. We should also review where the Harness code is kept (which repo); even though each association member tends to run their own Harness(es), we should share a repo so we can learn from each other’s code and perhaps develop some common Harness components.

PR - Let’s review the position when we have the new modular OpenWIS v4 .

188. v4 RET - Sample Local Data Source Harness

Create a Sample Local Data Source Harness.

69. v4 SYS - ADMIN5 Configure dissemination channels for Groups

This task was part of the OpenWISv4/GeoNetworksv3 work package assigned to GeoCat in 2015. The documentation produced as part of that work describes the new functionality delivered.

The outcome for this feature was:

2015 status: MO Test; Percent complete: 100; Priority: Normal; Questions outstanding: nil;

Extra info:

An OpenWIS deployment will provide a number of channels for disseminating data products to users via subscription or ad-hoc delivery request (e.g. email, FTP etc.). The available dissemination channels shall be specified in as configuration items.

GeoNetwork shall be amended such that a User Administrator can assign to a Group the permission to use a dissemination channel. A Group may be permitted to use multiple dissemination channels. At a minimum, all Groups are permitted to disseminate data products via Staging Post- in which case an email notification is generated indicating the URL of the data product that has been generated.

186. v4 DIS - Metadata lifecycle call-backs

Metadata lifecycle call-backs (Spring) - add/delete/remove metadata.

80. v4 RET - UI2 My-downloads page

This task was part of the OpenWISv4/GeoNetworksv3 work package assigned to GeoCat in 2015. The documentation produced as part of that work describes the new functionality delivered.

The outcome for this feature was:

2015 status: MO/GeoCat Review; Percent complete: 0; Priority: Normal; Questions outstanding: 100% or 0% Done? In the manual, does ‘redirect the user to the Users and Groups section of the admin console’ really mean that, or just that identical functionality exists in the User Portal and Admin Portal?;

Extra info:

GeoNetwork shall be amended to provide users with a web-based mechanism to:

monitor the status of their active downloads;
discard requests that are no longer required;
browse their download history; and
retrieve packaged data products from the Staging Post if they are still available.
The RequestService and ProcessedRequestService SOAP web-services provide for retrieval and persistence of the request and process-request objects. Details of active downloads should automatically be refreshed.

61. v4 DIS - Fix v4.0 harvesting

Analyse, design, develop and implement solutions to current issues that v4.0 has with metadata harvesting and indexing scalability.

NM is documenting his performance investigations here

41. v4 SYS - ADMIN1 Blacklisting

This task was part of the OpenWISv4/GeoNetworksv3 work package assigned to GeoCat in 2015. The documentation produced as part of that work describes the new functionality delivered.

The outcome for this feature was:

2015 status: MO/GeoCat Review; Percent complete: 90; Priority: Normal; Questions outstanding: 100% or 90% Done? Pagination?;

Extra info:

A black-listed user is able to browse the catalogue but is not able to download data from OpenWIS nor are their subscriptions fulfilled. Black-listing occurs when a user’s data download thresholds have been exceeded within a given period (e.g. 24-hours), or when an administrator deems that a given user has breached the fair-usage policy. A second threshold is defined for each user that is used to trigger delivery of warning to that user indicating their imminent breach of the data download threshold.

GeoNetwork shall be amended to provide a User Administrator with a web-based mechanism to:

blacklist a specific user;
blacklist all users within a specific Group;
revoke blacklisting for a specific user;
revoke blacklisting for all users within a specific Group;
amend the download thresholds for a specific user or for all users within a specific Group; and
review blacklist thresholds and download volumes for the current period for all users or for all users within a specific Group.
The BlacklistService SOAP web-service provides for retrieval and persistence of black-list information.

Update Puppet scripts of Vagrant deployment in order to avoid overwriting application files

Some application configuration files are being overwritten by others auto-generated by Puppet scripts during a Vagrant deployment.

This should not be allowed. The deployed files/configuration should be the ones that the user has built.

40. v4 DIS - MM1 Use of custom metadata file identifier

This task was part of the OpenWISv4/GeoNetworksv3 work package assigned to GeoCat in 2015. The documentation produced as part of that work describes the new functionality delivered.

The outcome for this feature was:

2015 status: MO Test; Percent complete: 100; Priority: Normal; Questions outstanding: nil;

Extra info: nil.

question about how subscriptions are handled when metadata is removed

From @woollattd on October 12, 2016 9:53

small question

I notice that there has been some activity with metadata in recent months around the GISCs... so
What happens when a user has subscribed to some harvested metadata that is later removed by the originating provider?
I don't seem to be able to remove metadata manually on OpenWIS when there are subscriptions attached.
Won't this cause catalogue discrepancies between the GISCs?

Copied from original issue: OpenWIS/openwis#211

Enforcement of password complexity rules

From @lmika-bom on July 4, 2016 4:25

It looks like the Bureau will need to implement password complexity and validation rules shortly to comply with operational requirements. I could imagine that we will not be the only one. Examples of some of the rules the Bureau will need to implement:

• Minimum password lengths (this may be dependent on what makes up the password - a password of a certain length does not need to include special characters, for example).
• Minimum number of characters from specific sets (lower-case, upper-case, numbers, symbols).

OpenWIS will need to be change to support setting rules governing password complexity and checking that new passwords supplied by the user comply with these rules. The particular rules need to be displayed to the user, along with any validation errors. Since this is something that will change from implementer to implementer, it is necessary that these rules are exposed as configuration somewhere.

Copied from original issue: OpenWIS/openwis#197

50. v4 RET - DDS3 Ad-hoc request for delivery

This task was part of the OpenWISv4/GeoNetworksv3 [work package][WP] assigned to GeoCat in 2015. The [documentation][doc] produced as part of that work describes the new functionality delivered.

[WP]: {{ site.baseurl | prepend: site.url }}/assets/OpenWISv4-GeoNetwork-2015-Workpackage.pdf
[doc]: {{ site.baseurl | prepend: site.url }}/assets/OpenWISv4-GeoNetwork-2015-Documentation.pdf

The outcome for this feature was:

2015 status: MO Test; Percent complete: 100; Priority: Normal; Questions outstanding: nil;

Extra info:

GeoNetwork shall be amended such that a user with the appropriate permissions may request the immediate delivery of data product instances that are currently available from the OpenWIS Data Service; so-called 'ad-hoc' delivery.

The portal will provide a web-based user interface enabling a user to configure their ad-hoc request for the specified dataset; allowing them to specify details of their chosen dissemination channel, how individual data product instances are packaged for delivery and which specific data product instances are required. On completion of the ad-hoc request details, the portal will package the user supplied information and the relevant product metadata record (retrieved via the ProductMetadataService SOAP web-service) into an ad-hoc request object and send it for validation and processing to the OpenWIS Data Service using the createRequest operation from the RequestService SOAP web-service.

OpenWIS shall notify the relevant Groups that the dataset has been downloaded according to the notify permission defined for the metadata record associated with the dataset that has been requested.

97. v4 DIS - The role of Category for metadata sync

See also, OpenWIS/openwis issue#122

This assessment will include assessing the role of categories in CGN and OpenWIS, including Kanban item 97 (role of category in metadata sync). This assessment will also include Kanban item 62 (delegation of management of Users and Groups to GeoNetwork - an alternative to using OpenAM).

The role of Category for metadata sync.

LM - Are categories in GN conceptually different than they were in OpenWIS 3.14? Please check.

33. v4 ACC - OWASP

MG - OWASP - An automatic report on library vulnerabilities is now included in the routine build. Reporting some high and medium priority issues at present, which will need to be investigated. TC to review - may need to remove some libs. LM - Outcome of Owasp might need a review.