geopython / pygeoapi

pygeoapi is a Python server implementation of the OGC API suite of standards. The project emerged as part of the next generation OGC API efforts in 2018 and provides the capability for organizations to deploy a RESTful OGC API endpoint using OpenAPI, GeoJSON, and HTML. pygeoapi is open source and released under an MIT license.

geopython

Vanity package for geopython projects

pip install geopython

>>> import geopython

pygeoapi's Issues

Elasticsearch: fix 500 error on index not found 404

[Thu May 24 18:35:11.004322 2018] [:error] [pid 22573:tid 140706876249856] NotFoundError: TransportError(404, u'index_not_found_exception', u'no such index')[Thu May 24 18:35:11.004322 2018] [:error] [pid 22573:tid 140706876249856] NotFoundError: TransportError(404, u'index_not_found_exception', u'no such index')

support open ended range time queries

cc @boshek

Users should be able to query collections for data from or until a certain timestamp (see discussion in OGC WFS 3 issue tracker.

As a first step we can implement per opengeospatial/ogcapi-features#155 (comment). Note that this will have to be supported at the provider level (no changes to the API items handler) given the semantics.

Write provider for Google Big Query

BigQuery from google now supports SQL/MM Spatial standard:

https://cloud.google.com/blog/big-data/2018/07/bridging-the-gap-between-data-and-insights

Using data from BigQuery should be more or less like the PostGIS provider

support OGC API - Coverages

Support the OGC API - Coverages specification.

sqlite depends on generic geojson package - package name collition

Sqlite provider uses package geojson, same folder contains module geojson and in python2.7 this module is imported causing problems (Python3 has no problems)

Drop dependence of geojson use only json python package

Dockerfiles (docker-compose) for pygeoapi/

Prepare docker-compose and dockerfiles to run pygeoapi togehter with elasticsearch

OS: Alpine 3.7

link to metadata

Would be good to add and/or link to metadata in/from collections.

Considering opengeospatial/ogcapi-features#160 (comment) core currently does not provide a detailed mechanism to link to metadata. They only mention that a link to additional metadata could be provided. Oapi-cov is a bit more explicit in req 2.9.

Where to store metadata:

• as part of config (like title/desc currently)
• some providers (such as geopackage/geotiff) may have embedded metadata
• config could have a link to external metadata (file/database/online)

Ingest elements from embedded/external metadata?

• In order to include some metadata on the collection response, it would be nice to ingest some external/embedded metadata, but we'll run into the issue of supporting multiple or requiring specific metadata schema's: DC, DCAT, Iso19115(-3), QGIS, schema.org, ...

implement file paths

Base provider class uses self.url to indicate where the provider is locate, this is problematic when we have for example spatialite that is a file.

Current we used file:// with asbolute paths. Need to create a specific file attriubte in the base class and support relative paths
e.g
self.file = "./data/"

allow for properties queries with wildcards

Example: property=valu*.

roll back code to python 3.5

parts of code uses PEP-0498 as string interpolator, this is very specific to python3.6 and reducing compatilibity

Should this be supported (min python version 3.6) or rolled out and mim python version 3.5

Elasticsearch: fix 500s on connection refused errors

add multilingual capability

Following on from opengeospatial/ogcapi-features#134, add capability to support multiple languages:

• support requests with Accept-Language
• support responses with Content-Language
• multilingual metadata in configuration
• handling hreflang in links in config
• other?

refactor sqlite and geopackage code

Both SQLite and geopackage data provider plugin contains repeated code. This should be refactor into a single class or base structure

Update pygeoapi official image

Use official python image from here

https://hub.docker.com/_/python/

For this dockerfile:

pygeoapi/docker/Dockerfile

Line 1 in fc4794f

FROM frolvlad/alpine-python3

make header content not uppercase

reported by @alexandreleroux cc @pvgenuchten

pygeoapi config service title is currently displayed as uppercase in HTML mode. This is rooted in mini.css default stying of <header> elements. Suggest we override to represent the actual content.

activate sorting from html represention

#73 added sorting functionality
sorting should be activated by clicking a column header in the html representation
consider that for example pagination links should consider a sorting option applied

using startindex gives an error: Can't convert 'int' object to str implicitly

Using the following link: (sqlite provider and countries)
http://localhost:5000/collections/countries/items/?startindex=10

Gives error
File "/home/jesus/tmp/pygeoapi/pygeoapi/pygeoapi/views.py", line 260, in get_features next_ = startindex + settings['server']['limit'] TypeError: Can't convert 'int' object to str implicitly

Implement gunicorn integrated into pygeoapi

Check and try to implement gunicorn server from the pygeoapi command line.

fix SQLite provider feature responses

Following on from OSGeo/gdal#626 (comment), fix sqlite provider to add 'type': 'Feature'. The below seems to work but assigning to @jorgejesus as the lead on this provider.

index b344b37..d922eb6 100644
--- a/pygeoapi/provider/sqlite.py
+++ b/pygeoapi/provider/sqlite.py
@@ -73,7 +73,9 @@ class SQLiteProvider(BaseProvider):
         feature_list = list()
         for row_data in self.dataDB:
             row_data = dict(row_data)  # sqlite3.Row is doesnt support pop
-            feature = {}
+            feature = {
+                'type': 'Feature'
+            }
             feature["geometry"] = json.loads(
                 row_data.pop('AsGeoJSON(geometry)')
                 )

Feature representation is expected to have ID (capitalized) field

This concerns in particular the templates for HTML rendering of items like here.

Problem

Some of the code in PyGeoAPIrelies on the fact that a Feature has an id-field named 'ID'. IMHO the GeoJSON standard specifies 'id' (lowercase) as Id-field:

  If a Feature has a commonly used identifier, that identifier
  SHOULD be included as a member of the Feature object with the name
  "id", and the value of this member is either a JSON string or number

Proposed Changes

Ok, we are dealing with an internal representation, but since some formatting is automated I propose to use the name id all over or at least some indirection i.s.o. of hard-coding ID..

travis fails with unavailable ES

The connection to ES is rejected in the travis built

The problem is caused by a typo on the travis build file

provide CSV output capability

support OGC API - Processing

Support the OGC API - Processing specification. Note we have initial partial support based on what we see on GitHub.

Docker image doesnt build

Docker image is broken with package problem:

OK: 13151 distinct packages available
fetch http://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
fetch http://nl.alpinelinux.org/alpine/edge/testing/x86_64/APKINDEX.tar.gz
ERROR: unsatisfiable constraints:
  libcrypto1.1-1.1.1-r3:
    masked in: @edge-testing
    satisfies:
               openssl-dev-1.1.1-r3[libcrypto1.1=1.1.1-r3]
               libssl1.1-1.1.1-r3[so:libcrypto.so.1.1]
  libssl1.1-1.1.1-r3:
    masked in: @edge-testing
    satisfies:
               openssl-dev-1.1.1-r3[libssl1.1=1.1.1-r3]
  openssl-dev-1.1.1-r3:
    masked in: @edge-testing
    satisfies: world[openssl-dev]
The command '/bin/sh -c apk update && apk add --no-cache    git    python3-dev    libffi    libffi-dev    musl-dev    gcc   

Upgrading to alpine 3.8 fixes the problem

write pytest for query range

Sqlite, Geopackage and Postgresql provider need to be tested for query range e.g

http://localhost:5000/collections/countries/items/?startindex=10&endindex=25

HTML Table layout problem with many-column features

In some cases where a Feature has many attributes (columns), the HTML Table display within the items.html template gets "too compact". See image below (feature 'RD Point' has around 20 attributes/columns):

How to solve? Ideas:

• limit number of columns displayed?
• "horizontal layout" as supported by mini.css ?
• explicitly define columns in dataset config?

implement PostGIS provider

Support PostGIS provider, better with native SQL instead or ORM

implement GeoPackage provider

Several Providers return FeatureCollection on get single Feature

This applies to the Providers GeoPackage, PostgreSQL and SQLite.

The WFS3 operation to request a single Feature, like e.g. http://localhost:5000/collections/obs/items/371?f=json
should return a single Feature GeoJSON construct like:

{"type": "Feature", 
 "ID": "371", 
 "geometry": {"type": "Point", "coordinates": [-75.0, 45.0]}, 
 "properties": {
     "stn_id": "35", 
     "datetime": "2001-10-30T14:24:55Z", "value": "89.9"
        .
   }, 
  "links": [...
 }

Instead these Providers return a GeoJSON FeatureCollection with a single Feature.
Other providers like CSV, ES and GeoJSON return a single Feature as expected.

Applies to Provider implementations: geopackage.py, postgresql.py and sqlite.py and their tests.

json is provided for /features/api?f=html instead of html

As discussed with @tomkralidis , html formatting is correctly provided for the features page /features/?f=html, however, the OpenAPI definition /features/api?f=html generates a response in JSON, not html.

This is true at least for the following public pygeoapi implementation:

• http://geo.weather.gc.ca/geomet/features/?f=html is html
• http://geo.weather.gc.ca/geomet/features/api?f=html is JSON and should be html

implement query sorting

Add support for clients specifying sorting of query results as per sortby=p1[:A|D][,p2[:A|D].

add crumbs path to html representation

As part of #26 a suggestion would be to add a crumbs-path to each of the pages
It indicates clearly to the user where he landed (if arriving from external) and navigation options to parent levels

Home > Collections > Lakes > Items > Ijsselmeer

py conventions for trailing slash management

I noticed sometimes users arrive in collections/obs/items and sometimes at collections/obs/items/. Both url's provide a valid answer however browsers treat both url's different when linking relatively. From these locations a '..' link will guide users to /collections in case 1 and to /collections/obs in case 2

Some 'solutions'

• do not use relative links, always link to a full url
• check all links for consistency on adding the trailing slash (or not)
• auto-remove/add a trailing slash if detected it is not added (rewrite rule)

pass OGC CITE tests

OGC has released testing for OGC API - Features in beta mode. Services can be tested directly from the CITE beta facility or a local Docker setup. More information can be found in the recent CITE announcement from OGC.

add support for items propertyname parameter

propertyname=foo,bar should return the response with just foo and bar properties.

support for transactions

Ability to delete, insert, update or patch features from the api
OGC-API-Features currently does not offer a specification for transactions yet
However Openapi typically uses DELETE, POST and PUT verbs to update content in an API, so I don't expect much deviation to this in the upcoming ogc api's. It could actually support the discussion there to have an initial implementation available.

Transactions should be disabled by default, and only activated if proper authorisation is in place

Create feature

POST /items/{id} [obj]

Update feature

PUT /items/{id} [obj]

Delete feature

DELETE /items/{id}

Patch feature (partial update)

PATCH /items/{id} [partial obj]

support text/html encoding request in headers

Currently the default output of pygeoapi is a json format encoding with html only being supported when using f=html on the requests

Sofwares that are not geospatial aware e.g browser should automatically display the html formating. This could be achieved by by reading text/hml encoding on the HTTP header request

https://github.com/opengeospatial/WFS_FES/blob/75d7c7839aca8c27ad9f8590727b37e110cfa0e7/placeholder-additional-conformance-class/parameters/f.yaml

discussion:
opengeospatial/ogcapi-features#179

possibilities:

• a visitor opening a feature in a browser (browser will send html header)
• a search engine crawling the service (crawler will send html header)
• leaflet/openlayers requesting a set of features (it will send the json header)
• a desktop gis following a link from a feature to another feature (does it expect a webpage or a feature?)

'Invalid format' error

Running plain pygeoapi in virtualenv Python 3.7.1 Ubuntu 18.04 and requesting any URL via the browser all requests return:

                InvalidParameterValue
                Invalid format

We expect collections etc.

Inspecting the code it appears to come out of the function pygeoapi.py: check_format(), in particular this line will return a list of accept formats so the check on this line will always fail.

Have tested and will make a PR, replacing the check_format() function with something like:

def check_format(args, headers):
    """
    check format requested from arguments or headers

    :param args: dict of request keyword value pairs
    :param headers: dict of request headers

    :returns: format value
    """

    # Optional f=html or f=json query param
    # overrides accept
    format_ = args.get('f')
    if format_:
        return format_

    # Format not specified: get from accept headers
    format_ = 'text/html'

    if 'accept' in headers.keys():
        format_ = headers['accept']
    elif 'Accept' in headers.keys():
        format_ = headers['Accept']

    format_ = format_.split(',')

    if 'text/html' in format_:
        format_ = 'html'
    elif 'application/json' in format_:
        format_ = 'json'

    return format_

This works for us, both with query parm, and even default (neither f= nor accept headers).

revise Docker images status and DockerHub

Revise the current docker images for updates, check if they are funcional and status on Dockerhub

sqlite/spatialite provider

Provider based on sqlite/spatialite

• pytest
• no sqlalchemy (for now?)

create presentation

Create presentation using reveal.js as a baseline for general purpose outreach.

Add filter options from html representation

pyGeoApi seems to have filter options, but not sure if available for all providers

pygeoapi/pygeoapi/api.py

Line 350 in d2b7693

for k, v in args.items():

Idea of this issue would be to add an option to html representation to create a filter

enhance HTML representations

The f=html responses need a better layout. The goal here should be to enhance layout (especially the maps) without introducing too many external dependencies.

implement ordered properties

Implement server side property order in configuration so, by default, users get result properties in a specified order.

Note that this differs from #51 which will allow the user to explicitly specify properties.

feature: facilitate a wfs 2 as a backend

so pygeoapi can be used on the many wfs 2 implementations out there

create documentation

Create documentation using Sphinx or MkDocs.

setup demo server

Currently a modest pygeoapi demo operates at http://geo.kralidis.ca/pygeoapi. Setup a proper demo area per below:

• setup demo at https://demo.pygeoapi.io
• ensure demo is auto-updated
• link demo server from https://pygeoapi.io/

HTML UI issues from #92 (mini.css)

Several issues found, some were there already before #92:

1. Stale links when navigating with Next|Prev through Collections

• navigate to http://localhost:5000/collections/lakes/items
• click in table on id single Item, Ok, shows single Item
• back to http://localhost:5000/collections/lakes/items
• click Next: ok, navigates to http://localhost:5000/collections/lakes/items/?startindex=10
• click single Item Id in table. e.g. id 10
• navigates to stale link: http://localhost:5000/collections/lakes/items/items/10

NB this was already seen before #92

2. Ditto for single Item

• navigate to http://localhost:5000/collections/obs/items/238
• HTML+Map single Item shows
• click Next or Prev button
• Tries to show: http://localhost:5000/collections/obs/items/238&f=html which fails

3. JSON OpenAPI doc link

• go to http://localhost:5000/
• click the link the OpenAPI definition as JSON
• opens http://localhost:5000/api showing HTML i.s.o. JSON (when #101 as well integrated)

Not sure what the standard requires, or if http://localhost:5000/api?f=json should be explicitly rendered?

fix CSV provider geometry types

Following on from OSGeo/gdal#626 (comment), fix CSV geometries to return float.

implement schema.org in responses

http://schema.org/