geo-data / medin-portal Goto Github PK

Replace the '+' symbol for the layer changing functionality with a 'Change layers' button floating in the top right corner.

As the EPSG site is often down, download the entire database and establish an automated daily update.

The abstract in the 'light' version of the site is not XML escaped, leading to invalid XML and rendering errors as exemplified by the following url:

http://portal.oceannet.org/search/light/catalogue/grid.bodc.nerc.ac.uk__MEDIN_2.3__EDMED176.xml

This issue was brought to light by Kay Thorne in an email on Tue, Jan 15, 2013 at 05:48:44PM +0000

Allow for multiple bounding box display on the map on the records page in line with the updated GEMINI2 standard. Helen to check with STFC whether the WSDL caters for this. It should also be implemented in the RSS downloads etc. Note from STFC - The current service already supports multiple spatial extents if multiple extents are present in the originating metadata record- the issue here would be updating the portal to use this functionality.

Search title to be changed to 'Data Holder' with the target of the search being both 'Custodian' and 'Distributor'. It will be a drop-down list from EDMO, with only those organisations that exist in the target fields being shown. For EDMO visit http://seadatanet.maris2.nl/edmo/ but for the actual webservice documentation go to: http://seadatanet.maris2.nl/ws/ws_edmo.asmx

WSDL available at http://seadatanet.maris2.nl/ws/ws_edmo.asmx?wsdl.

Helen to ask STFC to make a filter on data in the Custodian and Distributor fields possible. If this can't be done, then it will have to be a free text search.

The following page returns a 500 Portal Error:

http://portal.oceannet.org/search/full/catalogue/medin.ac.uk__MEDIN_2.3__MEDIN-266.xml

Under geographic search - remove the ‘globe’ image under geographic search, and replace with the geographical search map always showing.

Refer to ‘You may like related search terms (show)’ on the NERC DDS results page. The user is given choice to display or not. Olly has sent the code to Homme. This link provides an example of the NERC functionality.

Add a green light to indicate that an online resource (resource locator) is available. This will initially need to be added to the WSDL by STFC. Note from STFC - This is a portal issue – how the portal renders the returned XML is what needs to be altered here.

Example: http://portal.oceannet.org/search/full/catalogue/grid.bodc.nerc.ac.uk__MEDIN_2.3__EDMED196.xml at 2012-08-01 18:54:44,764

Example traceback is as follows:

The application encountered an unhandled exception
File /data/www/medin_portal/python/errata.py, line 58, in __call__
return self.app(environ, start_response)
File /data/www/medin_portal/python/selector.py, line 161, in __call__
return app(environ, start_response)
File /data/www/medin_portal/python/medin/__init__.py, line 341, in __call__
return app(environ, start_response)
File /data/www/medin_portal/python/medin/__init__.py, line 257, in __call__
return self.app(environ, start_response)
File /data/www/medin_portal/python/mediator.py, line 81, in __call__
return handler(environ, check_response)
File /data/www/medin_portal/python/medin/__init__.py, line 309, in wrapper
return app(environ, wrapped_response)
File /data/www/medin_portal/python/medin/views.py, line 1230, in __call__
return self.app(environ, start_response)
File /data/www/medin_portal/python/medin/templates.py, line 122, in __call__
output = template.render(**kwargs)
File /data/www/medin_portal/python/mako/template.py, line 133, in render
return runtime._render(self, self.callable_, args, data)
File /data/www/medin_portal/python/mako/runtime.py, line 364, in _render
_render_context(template, callable_, context, *args, **_kwargs_for_callable(callable_, data))
File /data/www/medin_portal/python/mako/runtime.py, line 381, in _render_context
_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
File /data/www/medin_portal/python/mako/runtime.py, line 414, in _exec_template
callable_(context, *args, **kwargs)
File /data/www/medin_portal/templates/full/base.html, line 133, in render_body
          ${next.body()}
File /data/www/medin_portal/python/mako/runtime.py, line 255, in <lambda>
return lambda *args, **kwargs:callable_(self.context, *args, **kwargs)
File /data/www/medin_portal/templates/full/base-ok.html, line 37, in render_body
${next.body()}
File /data/www/medin_portal/python/mako/runtime.py, line 255, in <lambda>
return lambda *args, **kwargs:callable_(self.context, *args, **kwargs)
File /data/www/medin_portal/templates/full/metadata.html, line 811, in render_body
        <%self:output_element element="${metadata.temporal_reference}" number="${16}">
File /data/www/medin_portal/python/mako/runtime.py, line 255, in <lambda>
return lambda *args, **kwargs:callable_(self.context, *args, **kwargs)
File /data/www/medin_portal/templates/full/metadata.html, line 107, in render_output_element
  % endif
File /data/www/medin_portal/templates/full/metadata.html, line 820, in body
        <div class="sibling">
File /data/www/medin_portal/templates/full/metadata.html, line 804, in format_date
        <p>It is recommended that all known temporal references of the
File /data/www/medin_portal/templates/full/metadata.html, line 66, in render_format_date
${date.strftime('%Y')}\
ERROR: ValueError: year=1890 is before 1900; the datetime strftime() methods require year >= 1900

Web addresses provided under ‘additional information’ should be active links.

Add a map showing all bounding boxes for results. Implemented in exactly the same way as on the NERC DDS. Olly has sent code to Homme.

Once a parameter has been selected, the Parameter Discovery Vocabulary will be populated with related keywords from P021 Vocabulary. A future enhancement will be to add a count to the P021 terms, so that people know how many metadata records are associated with that term. Helen will contact STFC to ask for an interface for this (as with issue #4).

In response to user feedback, add words 'Then click the search button.' to the ends of both paragraphs "Create a new box by dragging the mouse whilst holding down the CTRL key. Click the box to finish."
and "Click on an existing box to resize or drag it. Click the box again to finish. (help)"

Always remove 'You can start a new search' on the right hand side of portal homepage; this appears once ‘Edit this search’ is selected on results page.

The following email from Kay on Mon, 30 Apr 2012 11:37:54 details the issue:

A user of the NERC Data Catalogue Service (formerly the NERC DDS) got very confused when entering the co-ordinates in the Geographic search box manually.

They wanted data in the area

15-17 degrees North;
61 to 63 degrees West

And correctly established that the co-ordinates box needed the values:

-63.0,15.0,-61.0,17.0

Where -63.0 (63W) is the most westerly, 15.0 (15N) is the most southerly, -61.0 (61W) is the most easterly and 17.0 (17N) is the most northerly. The search correctly returned all records which intersect this region including global data sets, but the user became confused as the results page display was wrong, i.e.

You are searching for...
everything in 17.00N 15.00S -61.00E -63.00W

It appears that this display bug is also presented in the MEDIN Discovery Portal.

Reorganise the ‘summary area’ above the main metadata table to show title, abstract, Resource locator green light icon (if online resources exist), data holder (custodian), conditions for access and use constraints and ParentID. All metadata (including those in the summary area, and abstract and geographic extent) also to be shown in table below.

Use polygons instead of bounding boxes for geographic search, but continue the ability for a user to draw their own area of interest. Steps:

1. Homme to improve ability to resize rectangles after they have been drawn.
2. Helen to ask STFC about possibility of changing WSDL to enable polygon searching. If this is possible:
3. Helen send shapefiles listed in item 12 to Homme.
4. Helen and Homme to decide which shapefiles need resolution lowering to avoid slow searching problems.
5. Geodata to add ability for user to define area of interest by drawing (and resizing) a polygon

As discussed during the MEDIN teleconference on 20th December 2012 the polygons will also need to be shown on the user interface map, not just be used to represent a backend query.

An example traceback is as follows:

The application encountered an unhandled exception
File /data/www/medin_portal/python/errata.py, line 58, in __call__
return self.app(environ, start_response)
File /data/www/medin_portal/python/selector.py, line 161, in __call__
return app(environ, start_response)
File /data/www/medin_portal/python/medin/views.py, line 1230, in __call__
return self.app(environ, start_response)
File /data/www/medin_portal/python/medin/views.py, line 1026, in __call__
parser = self.request()
File /data/www/medin_portal/python/medin/dws.py, line 554, in __call__
response = super(MedinMetadataRequest, self).__call__()
File /data/www/medin_portal/python/medin/dws.py, line 538, in __call__
logger.error(msg + ': %s' % response.message)
ERROR: NameError: global name 'logger' is not defined

Example: http://portal.oceannet.org/search/full/catalogue/ukho.ac.uk__MEDIN_2.3__ukho20077079.xml at 2012-08-30 15:39:38,414

An example traceback is as follows:

The application encountered an unhandled exception
File /data/www/medin_portal/python/errata.py, line 58, in __call__
return self.app(environ, start_response)
File /data/www/medin_portal/python/selector.py, line 161, in __call__
return app(environ, start_response)
File /data/www/medin_portal/python/medin/__init__.py, line 341, in __call__
return app(environ, start_response)
File /data/www/medin_portal/python/medin/__init__.py, line 257, in __call__
return self.app(environ, start_response)
File /data/www/medin_portal/python/mediator.py, line 81, in __call__
return handler(environ, check_response)
File /data/www/medin_portal/python/medin/__init__.py, line 309, in wrapper
return app(environ, wrapped_response)
File /data/www/medin_portal/python/medin/views.py, line 1232, in __call__
return self.app(environ, start_response)
File /data/www/medin_portal/python/medin/templates.py, line 116, in __call__
ctxt = self.setup(environ)
File /data/www/medin_portal/python/medin/views.py, line 927, in setup
metadata = parser.parse()
File /data/www/medin_portal/python/medin/metadata.py, line 502, in parse
m.reference_system = self.referenceSystem()    # element 15
File /data/www/medin_portal/python/medin/metadata.py, line 878, in referenceSystem
return resolve(code)
File /data/www/medin_portal/python/medin/sr.py, line 37, in resolve
return _registry[code]
File /data/www/medin_portal/python/epsg/__init__.py, line 242, in __getitem__
raise KeyError(key)
ERROR: KeyError: 'urn:ogc:def:crs:EPSG::6326'

Change the Originator column to contain data from the custodian field, call it 'Data Holder', and only show the organisation name (to be able to be sorted alphabetically)

Use existing drop-down lists for Resource Type, Public Access Limits and Data format. See issue #2, #3 and #4 for others...

This thread suggests that the problem is in the use of the TemplateLookup class: it should only be instantiated once and used from a globally referenced variable which is currently not the case.

Note that the problem only manifests itself in a multithreaded persistent environment that is under load.

An example stack trace is as follows:

http://portal.oceannet.org/search/spatial/tms/1.0.0/bathymetry/12/4119/3273.png at 2012-09-01 04:42:11,217:
The application encountered an unhandled exception
File /data/www/medin_portal/python/errata.py, line 58, in __call__
return self.app(environ, start_response)
File /data/www/medin_portal/python/selector.py, line 161, in __call__
return app(environ, start_response)
File /data/www/medin_portal/python/medin/spatial.py, line 184, in tilecache
svc = get_tileservice(environ)
File /data/www/medin_portal/python/medin/spatial.py, line 168, in get_tileservice
template = lookup.get_template(mappath)
File /data/www/medin_portal/python/mako/lookup.py, line 85, in get_template
return self.__load(srcfile, uri)
File /data/www/medin_portal/python/mako/lookup.py, line 127, in __load
self.__collection[uri] = Template(uri=uri, filename=posixpath.normpath(filename), lookup=self, module_filename=(self.modulename_callable is not None and self.modulename_callable(filename, uri) or None), **self.template_args)
File /data/www/medin_portal/python/mako/template.py, line 84, in __init__
del sys.modules[self.module_id]
KeyError: 'config_tilecache_cfg'

The text 'unknown' or 'inapplicable' is currently incorrectly displayed as a clickable email address in Responsible Party information. An example is here.

@SteveDonegan clicking on any of the column headings in the results page should affect the ordering of results, allowing ascending and descending ordering. This works in the live portal but not in the new DWS. This is a DWS issue: the following DWS request should order the results by title (ascending) but fails to do so:

<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:ns0="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns1="http://discovery.ejb.revitalization.services.ndg/" xmlns:ns2="http://schemas.services.ndg/discovery" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
  <SOAP-ENV:Header/>
  <ns0:Body>
    <ns1:DoSearch>
      <request>
        <ns2:SearchCriteria>
          <ns2:TermSearch>
            <ns2:TermTarget>FullText</ns2:TermTarget>
          </ns2:TermSearch>
          <ns2:ResourceLocator/>
        </ns2:SearchCriteria>
        <ns2:RetrieveCriteria>
          <ns2:OrderBy>
            <ns2:OrderByField>DiscoveryTitle</ns2:OrderByField>
            <ns2:OrderByDirection>ascending</ns2:OrderByDirection>
          </ns2:OrderBy>
          <ns2:RecordDetail>DocumentBrief</ns2:RecordDetail>
        </ns2:RetrieveCriteria>
        <ns2:Start>1</ns2:Start>
        <ns2:HowMany>20</ns2:HowMany>
      </request>
    </ns1:DoSearch>
  </ns0:Body>
</SOAP-ENV:Envelope>

Drop ISO 19115 format (encoded as ISO 19139 XML) from download option as not working and arguably not required.

This is an issue affecting the new DWS: If the DWS does not respond to a portal query within 10 seconds, the portal times out and returns an error to the user. This is happening for certain data holder queries. The following query triggers the timeout whilst looking for all records held by 'British Geological Survey (BGS)':

/search/full/catalogue?dh=1827534157

The SOAP sent to the DWS is as follows (generated by /search/full/catalogue?dh=1827534157&soap=request):

<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:ns0="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns1="http://discovery.ejb.revitalization.services.ndg/" xmlns:ns2="http://schemas.services.ndg/discovery" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
  <SOAP-ENV:Header/>
  <ns0:Body>
    <ns1:DoSearch>
      <request>
        <ns2:SearchCriteria>
          <ns2:TermSearch id="1">
            <ns2:Term>'''British Geological Survey (BGS)'''</ns2:Term>
            <ns2:TermTarget>DataCustodian</ns2:TermTarget>
          </ns2:TermSearch>
          <ns2:TermSearch id="2" operator="OR">
            <ns2:Term>'''British Geological Survey (BGS)'''</ns2:Term>
            <ns2:TermTarget>DataDistributor</ns2:TermTarget>
          </ns2:TermSearch>
          <ns2:ResourceLocator/>
        </ns2:SearchCriteria>
        <ns2:RetrieveCriteria>
          <ns2:OrderBy>
            <ns2:OrderByField>DatasetMetadataUpdateDate</ns2:OrderByField>
            <ns2:OrderByDirection>descending</ns2:OrderByDirection>
          </ns2:OrderBy>
          <ns2:RecordDetail>DocumentSimple</ns2:RecordDetail>
        </ns2:RetrieveCriteria>
        <ns2:Start>1</ns2:Start>
        <ns2:HowMany>1</ns2:HowMany>
      </request>
    </ns1:DoSearch>
  </ns0:Body>
</SOAP-ENV:Envelope>

@SteveDonegan says this is a database issue that is being investigated.

Hi Homme,

Please let me know if this is not the correct way to submit potential issues to you!

I will email you the screen shot just in case since I cannot copy it here.

This could be a data error and I was hoping your logs would give an indication of what it could be.

{user reported following but I tried a variety and all gave error}

1. Go to MEDIN portal (http://portal.oceannet.org/search/full), enter BODC in search box and search
2. Select any title in the form “ship name cruise cruise ID CTD dataset” (eg: “RV Prince Madog cruise PD02/10 CTD dataset” or “RRS Charles Darwin cruise CD58 CTD dataset”)

cheers
Lise

Map search layers and background maps – produce a ‘how to’ guidance document to allow MEDIN core team to add further data layers to the geographical search, and to the background layers on the map.

As discussed in the MEDIN teleconference on 20th December 2012 this will be an outline document providing an overview of the process.

Bot proof email addresses using obfuscation techniques as on BODC site, so that email addresses still appear like proper email addresses.

Extend the metadata download facility to:

• also enable a csv download in addition to current RSS and Atom downloads,
• add the option to download all metadata in csv format to be on one page with fields as column headers. Fields with more than one occurrence will be concatenated,
• add the ability to download all the bounding boxes as KML as well as GeoRSS

‘Keywords’ to be removed from the summary area to the main metadata table, reference to dictionary source to be removed, leaving just the keywords without definition.

Replace text under the 'info' part of 'Responsible Party (info)' with the following:

Provides a description of an organisation or person who has a role for the dataset or resource. 'Originator' (a mandatory field) is the person or organisation who created the resource. 'Custodian' (a mandatory field) is the person or organisation that accepts responsibility for the data and ensures appropriate care and maintenance. If the data has been lodged with a MEDIN approved Data Archive Centre then the DAC should be specified as the Custodian. 'Distributor' (a conditional field) is the person or organisation that distributes the resource. 'pointOfContact' is the party who can be contacted for acquiring knowledge about or acquisition of the resource. There are further designations listed at 'Annex H. ISO Responsible party codelist' in 'Guidance notes for the production of discovery metadata for the Marine Environmental Data and Information Network (MEDIN)'.

At the start of the second paragraph, it should read 'If the data have..' and not 'If the data has....'.

BODC records do not always have complete dates only years under Temporal Reference (example ‘NERC North Sea Project Data Set (1988-1990)’ unique identifier ‘EDMED145’). The displayed metadata field forces a complete date to be displayed. Can this be changed so that if the metadata state only year then this gets displayed. Also, the portal element should read ‘The data span the period’ and not ‘spans’.

This involves using the new WSDL located at http://csw1-dev.cems.rl.ac.uk:8180/discovery/dws?wsdl. This provides a changed interface to the previous WSDL. Dealing with these changes has resulted in a SOAP error the details of which I have copied to Steve Donegan in the hope that he can help: it looks like a failure at the DWS back end.

Only the web addresses of online resources should be shown in the Resource Locator field. For some BODC records, there are multiple resource locators which need not all be displayed as some are just organisation URLs - for example, for ‘NERC North Sea Project Data Set (1988-1990)’ unique identifier ‘EDMED145’, URLs 1-12 should not be there as they are URLs from the participating organisations. Only the URLs in the resource locator field should be shown in this field. If the online resource has a name, then this should be displayed as a link. If only the URL is given, then this should be shown, rather than using the 'External link' wording.

Implement upgrade to the current US map at http://www.marine-geo.org/tools/web_services.php. Homme to look through for changes and recommend updates required.

In response to GEMINI2, the MEDIN standard will be changed so that the Date of Publication will go from mandatory to conditional and it will state that one of either of the ‘Date of Publication’, ‘Date of last revision’ and ‘Date of creation’ must be given which is specified in the GEMINI2 standard. This may require changes to the portal. Homme will check this and make any necessary changes.

Under date search – have the date text box permanently displayed instead of appearing only when the ‘the earliest record’ and ‘the latest record’ are clicked. Let the pop-up calendar appear when the cursor is placed in the text box.

Remove from target search ‘Topic Category’, ‘Author’, ‘Lineage’. Keep ‘Parameter’, ‘Resource Type’, ‘Data Holder’ (‘Custodian’ in the metadata), ‘Public Access Limits’, and ‘Data format’.