ckan / ckanext-spatial

Geospatial extension for CKAN

ckanext-spatial - Geo related plugins for CKAN

This extension contains plugins that add geospatial capabilities to CKAN, including:

• Geospatial dataset search powered by Solr, providing a bounding box via a UI map widget or the API.
• Harvesters to import geospatial metadata into CKAN from other sources in ISO 19139 format and others.
• Commands to support the CSW standard using pycsw.

Note: The view plugins for rendering spatial formats like GeoJSON have been moved to ckanext-geoview.

Full documentation, including installation instructions, can be found at:

https://docs.ckan.org/projects/ckanext-spatial/en/latest/

Supported Versions

ckanext-spatial >= 2.0.0 supports CKAN 2.9 and CKAN 2.10. Check the [tested enviroments](https://github.com/ckan/ckanext-spatial/blob/master/.github/workflows/test.yml) for more details.

For previous CKAN versions please use the v1.x tags.

Community

• Developer mailing list
• Gitter channel
• Issue tracker

Contributing

For contributing to ckanext-spatial or its documentation, follow the same guidelines that apply to CKAN core, described in CONTRIBUTING.

Copying and License

This material is copyright (c) 2011-2023 Open Knowledge Foundation and contributors.

It is open and licensed under the GNU Affero General Public License (AGPL) v3.0 whose full text may be found at:

http://www.fsf.org/licensing/licenses/agpl-3.0.html

ckanext-spatial's Issues

Allow CSW harvesters to define CQL filters

When defining a CSW harvester, we may want to only harvest a subset of the records from the remote source.
This issue may be solved by adding a cql filter in the harvest source configuration, e.g.:

  {"cql": "title LIKE '%Suitability%'"}

Update WMS previewer to new pluggable previews

Usecase: widget initiates search (not just filter on results)

Hi guys,

We have followed the docs and popped the Spatial Search Widget onto the ../package/search.html template.

It seems like the intended use for this map widget is to support search results filtering. Ie, the user must already start a search before they get a chance to filter on extent.

In our case 90% of the users want to start their search with the bounding box...then they can refine and filter the results based on topic etc.

So, I want to mount the search widget on the front page so that 'extent' is the first search term submitted.

Can this be supported?

Cheers,
JB

Support more fields in the ISO doc parser

Add some more fields from the seemly endless pool of available metadata fields for ISO 19139 docs, eg:

• alternate title
• presentation form
• purpose
• usage
• aggregation info
• ...

These would get added to the output of ISODocument.get_values() but probably not to the dataset dict that gets created from the harvesters. Extensions can add them customizing the get_package_dict hook.

Insert correct MimeTypes for WMS & WFS harvested layers

I manually set the mime-types of some of the resources that our Geonode is publishing and it has a real great effect: all recognised types now get a preview mode (and .csv can be imported in the datastore)

This seems like a very small thing with a lot of wins. I understand I could overwrite classes and things myself, but its so obvious, it would be great to have in the standard install.

Is that a lot of work?

Support for multipolygon geometries

Multipolygons are largelly used for acipelag and country and regional borders but unfortunatelly are not supported by the extension. It seems SOLR have sypport for it already when using JTS

Do the widgets work in ckan 2.0?

I think I've got the extension installed correctly (actually I know I do -- at least the geospatial search API is working), but I don't see any map widgets showing up. I am using a CKAN installation from source code. I installed from the master branch on Oct. 16th.

Sorry if this is trivial -- just getting started understanding CKAN. Its clear that the UI is entirely revamped between 1.7 and master, so I'm just wondering if this extension is compatible with 2.0 yet.

Thanks!

Dataset harvest object reference gets out of date if there isn't an update

When performing a harvest, if the remote object hasn't been updated we delete the previous harvest object and don't update the dataset. This means that the indexed data_dict for the dataset, the one used on package_show still has the old harvest object id. We need to reindex the dataset to update it.

Update dataset search widget for 2.0

Pure snippet, use fanstatic resources and move to Leaflet

Improve harvested metadata API

• Nicer and version independent URLs (with redirects for the old ones)
• Allow defining alternative XSLT for html view
• Support original documents

UomSymbol regex pattern rejects good values

Here, the UomSymbol validation regex pattern [^: \n\r\t]+ in theory is supposed to reject all chars of colon, space, newline, carriage return, and tab. But we found out it actually rejects all n, r, t chars. Values such as decimalDegrees and feet will fail the validation.

If we rewrite the pattern to an equivalent [^:\s]+, then all looks ok.

Troubleshoot preview map on read.html

Hi guys,
We are up to the stage of trying to get the dataset locator map to display on the read.html template.
I have edited a dataset and added in a geojson value with key=spatial in one of the custom fields. GeoJSON is:
{
"type":"Polygon",
"coordinates":[[[141.802,-13.2412],[141.802, -11.9863], [143.364, -11.9863], [143.364, -13.2412], [141.802,-13.2412]]]
}

The custom read.html template is being picked up (according to the debug), but the page appears to load same as always...no preview map shown.
How would I try to troubleshoot if it's the geojson or if it's the templating that is the trouble?
Thanks!
JB

Troubleshooting PostGIS connection issues

G'Day all,
I have been following your install docs and have come up against a permissions issue which you flag in the Troubleshooting section.
Problem is, what to do about it?
In our case we already had the PostGIS extensions installed as the DB is used by other applications (GeoServer and Drupal) so we did no need to go through the PostGIS installation portion of the fguide.
Next point is that I have dropped the '_default' term off of the default CKAN user and database...so my user=ckan and the main database=ckan
The core CKAN installation appears to be running fine...so I have apparently been successful to this stage.
Now, after installing the spatial search extension, here's the error generated while trying to reload the .ini file:

And here's my PostgreSQL privileges table, showing that the database is fully owned by ckan.

How am I able to get a permission denied error in this situation?
For testing I have also added the postgres user and a couple of others to the database with full privileges...still no good.
How can I check the spatial extension's configurations to see if "ckan_default" is hardwired in someplace as the assumed PostGIS user?
Thanks indeed,
JB

Check that sort parameter exists on Postgis spatial query

Auth problems with site_user

See ckan/ckan#1184 for details

Add pyparsing as requirement

Needed for the WAF harvester

Consolidate new spatial harvesters for 2.0

• Move new harvesters from geodatagov extension
• Move existing harvesters to own gemini namespace
• Adapt harvested document model for generic ISO support
• Adapt tests

Add geojson mimetype as part of the preview extension

See ckan/ckan#1534

Update dataset map to be a pure snippet

No need to register a plugin, just add the snippets provided or tweak the js module to load other base layers, etc

Use fanstatic resources.

Submitting valid geojson not containing polygon causes a crash

Steps to reproduce:

1. edit a dataset
2. set the spatial field to: { "type": "Point", "coordinates": [-3.145,53.078] }
3. save

CKAN fails with an (IntegrityError) new row for relation "package_extent" violates check constraint from postgis

Common function for initializing maps

To get consistent backgrounds, avoid duplication and make easier to fix things.

Open proxy

The proxy controller resolves any URL passed. i.e. it's an open proxy which has plenty of security issues.

In DGU we only allow URLs that appear in resources. However a WMS can specify a different base URL in GetCapabilities for further requests. Which we're still debating how to resolve. See http://redmine.dguteam.org.uk/issues/1322

individualName element needed in ISOResponsible Party class

I'm working with this extension and noticed that the class ISOResponsibleParty should also have an element for individualName according to the ISO specs. USGIN_ISO_MetadataV1.2

Fix broken markers images in geojson preview in debug mode

We need to apply the same fix we used on the recline preview on core

Exception with latest OWSlib

After cf202ba, the lxml version used by ckanext-spatial causes an exception:

    from owslib.etree import etree
  File "/home/adria/dev/pyenvs/ckan_geo/src/owslib/owslib/etree.py", line 19, in <module>
    patch_well_known_namespaces(etree)
  File "/home/adria/dev/pyenvs/ckan_geo/src/owslib/owslib/etree.py", line 14, in patch_well_known_namespaces
    etree_module.register_namespace(k, v)
AttributeError: 'module' object has no attribute 'register_namespace'

register_namespace was introduced in lxml 2.3 and we are using <=2.2.99.
We need to investigate a bit, but we probably need to require a higher version.

Spatial search widget does not work on 2.1

Becasue the form id has changed

Allow extensions to register custom validators for harvesters

Right now they need to extend the whole harvester

Move README to proper docs

Exception on deleting records in the CSW harvester

The object id is not properly set (because we haven't called obj.save() yet) when creating the list of ids returned from the gather_stage, so a value of None is added.

This causes an exception when getting values from the Redis queue:

Traceback (most recent call last):
  File "/usr/lib/ckan/default/bin/paster", line 9, in <module>
    load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 104, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke
    exit_code = runner.run(args)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 238, in run
    result = self.command()
  File "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/commands/harvester.py", line 126, in command
    for method, header, body in consumer.consume(queue='ckan.harvest.fetch'):
  File "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/queue.py", line 160, in consume
    self.redis.set(self.persistance_key(body),
  File "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/queue.py", line 165, in persistance_key
    return self.routing_key + ':' + message[self.routing_key]
TypeError: cannot concatenate 'str' and 'NoneType' objects

Search filter failure

G'Day all,

My apologies for this issue: it is surely something to do with PostGIS, but perhaps others may find similar problems.

We have the extension installed and it is displaying bbox outlines on datasets. This should indicate that the geoJSON is being stored with the record, and can be displayed on the preview map. Indeed, I can see the data rows in the package_extra table.

However, when we try to filter datasets based on bbox input, no records are returned.
After double checking the bbox extents and making sure that the filter should run, I decided to look in the database itself.

Investigating the package_extent table, I can see three rows there that look like good candidates for spatial data...looks like this:
6b99eac7-c849-4fb8-ac35-7d3b1568f38b 0103000020E610000001000000050000008B6CE7FBA9B961402497FF907E7B2AC08B6CE7FBA9B96140F0164850FCF827C0CFF753E3A5EB6140F0164850FCF827C0CFF753E3A5EB61402497FF907E7B2AC08B6CE7FBA9B961402497FF907E7B2AC0

So this should mean that our record entry process is sound since we're getting rows written to the db. But why can't we see a filtered result? We simply get "No records found" after applying the filter.

Looking in the logs I can see we have a stack of PostgreSQL errors like these:
2014-06-20 01:39:56 UTC LOG: could not receive data from client: Connection reset by peer
2014-06-20 01:39:56 UTC LOG: unexpected EOF on client connection

At this point I am out of clues oor ideas on how to further troubleshoot.
Any assistance much appreciated!

Validation crashes if ISO doc does not provide gmd:hierarchyLevel

It is an optional tag, according to the spec we should assume dataset if missing.

  File "/usr/local/ckan/pyenv/src/ckanext-spatial/ckanext/spatial/validation/validation.py", line 335, in is_valid
    is_valid, error_message_list = validator.is_valid(xml)
  File "/usr/local/ckan/pyenv/src/ckanext-spatial/ckanext/spatial/validation/validation.py", line 87, in is_valid
    metadata_type = cls.get_record_type(xml)
  File "/usr/local/ckan/pyenv/src/ckanext-spatial/ckanext/spatial/validation/validation.py", line 123, in get_record_type
    return iso_parser.read_value('resource-type')[0]
IndexError: list index out of range

Add default_tags and default_extras handling

CKAN harvester allow to define default_tags and default_extras in configuration, so that tags and extras are automatically attached to all dataset harvested from a given source.
This behaviour is currently missing on CSW harvester.

Support JSON when syncing between CKAN and pycsw

Currently when an "arcgis" record is found, it is skipped from the sync process.

https://github.com/okfn/ckanext-spatial/blob/master/bin/ckan_pycsw.py#L172

It would be nice to transform this JSON record to a valid ISO XML to import to pycsw.

Add command to manage integration with pycsw

Based on the work by @kindly on https://github.com/kindly/ckan_pycsw_import, add a command to the extension that allows to export easily the spatial datasets to pycsw. This will only work for harvested ISO datasets for the time being.

The command will be run both via paster (in the context of a CKAN install) or independently (eg a separate server), as it uses the API to get the datasets.

Contributing to docs?

G'Day all,
If I were to provide some recommendations for sharpening up the installation docs that are published at CKAN (eg: http://docs.ckan.org/projects/ckanext-spatial/en/latest/install.html), how would I go about it?
Can I pull a doc format of some sort, make eidts, and push them back for review and commit?
Cheers,
JB

Flag datasets created via the spatial harvesters via a generic extra

We use to have a UKLP=true, add something similar

Support Solr 4 based spatial search on 2.0

Solr 4 introduces new geospatial capabilities which will boost significantly the performance of the CKAN's spatial query.

Right now, the spatial query is performed in a previous step via PostGIS and the resulting dataset ids are sent to Solr as part of the query provided by the user. This was implemented in this way for a number of reasons, but it is obviously not very efficient and will create problems if the catalogue has a big number of datasets.

On Solr 4, you can define a "spatial" field on which you can index geometries (from simple lat/lon pairs to WKT polygons) and that can be later used to perform spatial queries combined with the rest of Solr parameters.

As Solr 4 is relatively recent (so still not present in the Ubuntu repos) and installation of the required libraries requires a bit of fiddling, this Solr 4 powered search will be not the default one for the time being, but it will strongly recommended for instances with large number of datasets.

In terms of implementation, it will be just a matter of indexing the right geometry in before_index and add the necessary syntax to the before_search extension points, as well as document the necessary changes on the Solr schema file.

PostgresSQL 9.1, CKAN 2.2 Ubuntu installation

When I try to run

paster --plugin=ckanext-spatial spatial initdb 4326 --config=/etc/ckan/default/development.ini

I get a exception trace with the error:

Traceback (most recent call last):
File "/usr/lib/ckan/default/bin/paster", line 9, in
load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 104, in run
invoke(command, command_name, options, args[1:])
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke
exit_code = runner.run(args)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 238, in run
result = self.command()
File "/usr/lib/ckan/default/src/ckanext-spatial/ckanext/spatial/commands/spatial.py", line 38, in command
self._load_config()
File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 96, in _load_config
load_environment(conf.global_conf, conf.local_conf)
File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 232, in load_environment
p.load_all(config)
File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 124, in load_all
unload_all()
File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 182, in unload_all
unload(_reversed(_PLUGINS))
File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 210, in unload
plugins_update()
File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 116, in plugins_update
environment.update_config()
File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 351, in update_config
engine = sqlalchemy.engine_from_config(config, 'sqlalchemy.', *_extras)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/init.py", line 292, in engine_from_config
return create_engine(url, *_opts)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/init.py", line 274, in create_engine
return strategy.create(_args, **kwargs)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 141, in create
engineclass.name))
TypeError: Invalid argument(s) 'client_encoding' sent to create_engine(), using configuration PGDialect_psycopg2/QueuePool/Engine. Please check that the keyword arguments are appropriate for this combination of components.

The fix is in /usr/lib/ckan/default/src/ckan/ckan/config/environment.py line 347.

    #extras = {'client_encoding': 'utf8'}
    extras = {'encoding': 'utf8'}

The client_encoding name will cause errors in PostgresSQL.

Extension does not work with PostGIS 2.0

Because geoalchemy does not support it. See [1]. Apparently geoalchemy2 is the way to go.

[1] geoalchemy/geoalchemy#22

plugin.py has an error with solr-spatial-field searching

Line 212 should be this:

search_params = self._params_for_solr_spatial_field_search(bbox, search_params)

because self._params_for_solr_spatial_search() does not exist.

Tweaks before 2.2 release

Minor frontend and docs tweaks

Problem in gemini csw harvester

When I try to harvest a csw server I get the following error:

2012-10-23 17:24:19,740 DEBUG [ckanext.spatial.harvesters.CSW.gather] Starting gathering for http://ims7.bkg.bund.de/geonetwork/srv/en/csw
2012-10-23 17:24:19,740 INFO [ckanext.spatial.lib.csw_client] Making CSW request: getrecords {'outputschema': 'http://www.isotc211.org/2005/gmd', 'startposition': 0, 'typenames': 'csw:Record', 'maxrecords': 10, 'keywords': [], 'esn': 'brief', 'qtype': None}
2012-10-23 17:24:20,050 INFO [ckanext.spatial.harvesters.CSW.gather] Got identifier 1fad8543-6552-4e7b-a723-ad36d17eb8be from the CSW
2012-10-23 17:24:20,102 INFO [ckanext.spatial.harvesters.CSW.gather] Got identifier 14fcbaa2-9e3b-4555-b84a-8404488c7015 from the CSW
2012-10-23 17:24:20,110 INFO [ckanext.spatial.harvesters.CSW.gather] Got identifier 1c3507ce-6631-4175-b8dd-6444fa3e9472 from the CSW
2012-10-23 17:24:20,211 INFO [ckanext.spatial.harvesters.CSW.gather] Got identifier 4b0bb1ed-29f6-471b-aef5-f74891c30d50 from the CSW
2012-10-23 17:24:20,313 INFO [ckanext.spatial.harvesters.CSW.gather] Got identifier e36df891-d58f-407b-a7fb-e3796935b3c6 from the CSW
2012-10-23 17:24:20,322 INFO [ckanext.spatial.harvesters.CSW.gather] Got identifier 49734de9-f84d-49b3-a4f3-4971c7429968 from the CSW
2012-10-23 17:24:20,328 INFO [ckanext.spatial.harvesters.CSW.gather] Got identifier b8434736-ebf1-4c75-83ea-17d0ca83352d from the CSW
2012-10-23 17:24:20,334 INFO [ckanext.spatial.harvesters.CSW.gather] Got identifier 30fb41aa-0a45-409b-a72b-a108c9e1919d from the CSW
2012-10-23 17:24:20,340 INFO [ckanext.spatial.harvesters.CSW.gather] Got identifier d71d0242-417b-4995-925c-7e4542dea8d1 from the CSW
2012-10-23 17:24:20,445 INFO [ckanext.spatial.harvesters.CSW.gather] Got identifier 0bef9252-7b3e-406c-8af8-730c7926f802 from the CSW
2012-10-23 17:24:20,553 INFO [ckanext.spatial.lib.csw_client] Making CSW request: getrecords {'outputschema': 'http://www.isotc211.org/2005/gmd', 'startposition': 10, 'typenames': 'csw:Record', 'maxrecords': 10, 'keywords': [], 'esn': 'brief', 'qtype': None}
2012-10-23 17:24:20,924 ERROR [ckanext.spatial.harvesters] Error gathering the identifiers from the CSW server ['NoneType' object has no attribute 'find']
2012-10-23 17:24:21,031 DEBUG [ckanext.harvest.queue] Received from plugins gather_stage: None

I tried with two different servers and I got the same error. We know that it worked with the old ckanext-inspire, at least with the server listed in the output above.

The other address is http://gateway.hamburg.de/OGCFassade/HH_CSW.aspx

We use latest CKAN from master.

Any help would be appreciated
Simon

Improve extensibility of the Spatial Harvesters

Right now you can extend the base SpatialHarvester class in order to customize things like the dataset dict that gets created, plug into a transformation mechanism, etc.

This is a bit painful because you need to create your own harvester class and register it as harvester plugin. We will add a new ISpatialHarvester interface which will provide proper extension points into parts of the import stage.

The get_package_dict will be more flexible, as apart from the default generated package dict it will get the parsed ISO values and the harvest object (which gives access to the original XML document and the source).

SOLR too many boolean clauses in spatial searches using "postgis" as search backend

It seems when selecting a bbox that include large number of datasets the resulting SOLR query exceed the LUCENE boolean clause limits.

In our CKAN production.ini we use this setting:
ckanext.spatial.search_backend = postgis
the issue is not present when using the solr and solr-spatial-field search backends

We got the following on the SOLR log and as you can see the expression placed after the AND clause containing all the dataset IDs is very large.

 org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:171)
    at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:187)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
    at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:721)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:417)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.search.SyntaxError: Cannot parse '"" AND (id:7c794f73-9fe0-49e1-9fd7-c92d700afe26 OR id:97e8ef6b-b717-4120-81f0-e4b76b7e81de OR id:56fb5f6e-cb7b-41aa-af13-069b8caf9fb9 OR id:53b4c108-08ea-4cd2-aa7f-237d56b185e6 OR id:0042f702-ddae-46d4-9565-244a30b489c1 OR id:3299a40c-0121-4fa8-bb70-67adaf35b9a9 OR id:8e2e335e-3f09-43e3-a5b7-7904530f00db OR id:04a5eaba-f794-4629-9e5c-3be741f5e3ec OR id:35a88ae6-4a28-49ab-a628-7d9fc6b192aa OR id:9bb3c7f1-ab03-4210-9369-5e2f70cc578d OR id:1f93d19f-515f-4f61-b46a-c4751f28ed7a OR id:b202b0e4-e217-448e-96f1-8c4168915fe8 OR id:c81e4d28-9a56-4696-a51a-4e6f0721c0c6 OR id:5dd447a1-1a83-43c2-b74e-fdaa52effe7f OR id:2356dc7c-947a-423f-b4f9-e8acfbd5f141 OR id:2a31274c-fc46-4680-91f7-f422d36ce9fa OR id:5f9be208-1984-459e-943f-090e3de7f70f OR id:a23a19a7-9fcb-4cc1-a281-260efc80e09c OR id:256397be-22cc-4824-9129-729b943ca62c OR id:45789a85-7a38-4664-af93-15f4008db3cc OR id:85df485b-56a2-488b-b497-592deead0ff9 OR id:3e1a1250-8bd6-4304-8075-6c766198f34d OR id:ae3cf945-91d3-43cb-af79-983faadc237b OR id:ad997698-3da8-4bdf-86a0-6e0ae3ce8fc6 OR id:0717ec63-5c04-446d-98fd-fe67de4120e8 OR id:915759ec-4737-47a1-be07-d780aeb582b7 OR id:dcc06d38-f6bf-4bce-9109-25c12a110803 OR id:2a777284-bcf4-4370-b339-2bcec10983d6 OR id:8dcfb4ce-a1a2-4853-8bdd-f7ca9fb1bdf1 OR id:000d27fa-d94f-4467-bf34-1a3940a8d977 OR id:23f16142-c3bc-4f9d-8911-e16617eedd28 OR id:c75009f3-c0cc-4bad-9127-77d661d4ccc3 OR id:f60f7410-44bc-435e-81a0-42fcb1f1f835  [...]': too many boolean clauses
    at org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:163)
    at org.apache.solr.search.LuceneQParser.parse(LuceneQParser.java:50)
    at org.apache.solr.search.QParser.getQuery(QParser.java:142)
    at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:142)
    ... 21 more
Caused by: org.apache.lucene.search.BooleanQuery$TooManyClauses: maxClauseCount is set to 1024
    at org.apache.lucene.search.BooleanQuery.add(BooleanQuery.java:142)
    at org.apache.solr.parser.SolrQueryParserBase.getBooleanQuery(SolrQueryParserBase.java:513)
    at org.apache.solr.parser.SolrQueryParserBase.getBooleanQuery(SolrQueryParserBase.java:489)
    at org.apache.solr.parser.QueryParser.Query(QueryParser.java:146)
    at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:190)
    at org.apache.solr.parser.QueryParser.Query(QueryParser.java:140)
    at org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:97)
    at org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:153)

Extract thumbnail from ISO documents

gmd:identificationInfo/gmd:MD_DataIdentification/gmd:graphicOverview/gmd:MD_BrowseGraphic

Store as extras so it can be displayed on the frontend.

Method for Geolocating Datasets

Howdy all,
I think we're making some serious progress here, very exciting. We have a search map displaying on the search.html template and the solr schema is updated. Next stop: defining the geo extent of each database.
The Docs are uncharacteristically thin on this point. I believe this is all we get:

Regardless of the backend that you are using, in order to make a dataset queryable by location, an special extra must be defined, with its key named ‘spatial’. The value must be a valid GeoJSON geometry, for example:

So I am assuming that "special extra" means "custom field"?
Ie, we are just going to use the standard input form, and take over one of the custom fields?
I will try this now, but can I ask if anyone has coded out a solution to draw a bbox or point on a map to derive this input?
Thanks!
JB

Mark get actions as side_effect_free

This will allow accessing them via GET requests

I've activated all plugins but nothing have changed

I'm using CKAN branch master

Also is not clear in the documentation what this plugins have to do with the "map" preview already included in trunk when previewing a resource.

In this preview I've tried to render some points but the marker images requests returns 404, the URL generated is: "/fanstatic/ckanext-reclinepreview/:version:2013-02-13T16:50:30.65/:bundle:vendor/flot/jquery.flot.js;vendor/leaflet/0.4.4/images/marker-icon.png"

Remove spatial_query_widget mention from README

CSW Harvester: comparing naive to aware dates

... on branch release-v2.0

If I use the harvester to bring in an ISO document where /gmd:MD_Metadata/gmd:dateStamp/gco:DateTime is time-zone aware, it comes in fine. However when I re-harvest it breaks.

ckanext.spatial.harvesters.base line 528:

if not self.force_import and harvest_object.metadata_modified_date <= previous_object.metadata_modified_date:

previous_object.metadata_modified_date comes out of the database is appears to be time-zone naive, while harvest_object.metadata_modified_date is time-zone aware, and the comparison can't be made.

I was able to fix it by just changing line 470:

metadata_modified_date = dateutil.parser.parse(iso_values['metadata-date']).replace(tzinfo=None)

... make the harvest_object.metadata_modified_date time-zone naive...