qwikgeo / qwikgeo-api

QwikGeo is a enterprise scale api for a GIS portal.

QwikGeo API

QwikGeo API is a enterprise scale api for a GIS portal. QwikGeo API is written in Python using the FastAPI web framework.

---

Source Code: https://github.com/qwikgeo/qwikgeo-api

---

Tech Docs

Docs available at this link.

Requirements

QwikGeo API requires PostGIS >= 2.4.0.

Configuration

In order for the api to work you will need to edit the .env with your database to host the API.

DB_HOST=localhost
DB_DATABASE=qwikgeo
DB_USERNAME=postgres
DB_PASSWORD=postgres
DB_PORT=5432
CACHE_AGE_IN_SECONDS=0
MAX_FEATURES_PER_TILE=100000
SECRET_KEY=asdasasfakjh324fds876921vdas7tfv1uqw76fasd87g2q
GOOGLE_CLIENT_ID=asdasdas745-cj472811c26nu77fm5m98dasdasdasda1vkvk2pscfasad.apps.googleusercontent.com
JWT_TOKEN_EXPIRE_IN_MIUNTES=60000

Usage

Running Locally

To run the app locally uvicorn qwikgeo_api.main:app --reload

To run the docs locally mkdocs serve

Production

Build Dockerfile into a docker image to deploy to the cloud.

Aerich Commmands

aerich init -t qwikgeo_api.main.DB_CONFIG --location migrations -s .

aerich init-db

qwikgeo-api's Issues

Transfer application into qwikgeo_api folder

Description

For better file structure management, have api nested under a folder called qwikgeo_api.

Tasks

• Create folder qwikgeo_api
• Move files/folders

Typo in .env.sample and in docs

In env.sample and in the readme.md change
JWT_TOKEN_EXPIRE_IN_MIUNTES=xxx to JWT_TOKEN_EXPIRE_IN_MINUTES=xxx

Update all endpoints to use table_id vs table

Description

For consistency, we should be using table_id vs table on all endpoints.

Endpoints to update

• Analysis
• Collections
• Imports

Imports API - WFS

Description

Allow users to import a dataset via a WFS service.

Endpoint

api/v1/imports/wfs_service

Add next and previous links for /collections/{collection}/items

Description

Currently we are missing the next and previous links for items. We need to add them to be compliant. The href should respect any additional url parameters the user already submitted.

Example

{
    "type": "application/geo+json",
    "rel": "prev",
    "title": "items (prev)",
    "href": "/collections/{collection}/items?offset=0"
},
{
    "type": "application/geo+json",
    "rel": "next",
    "title": "items (next)",
    "href": "/collections/{collection}/items?offset=20"
}

Implement part 2 of OGC Features API

https://docs.ogc.org/DRAFTS/18-058r2.html

• Update api endpoints
• Update docs

Add photo_url to users/search endpoint

Description

Return the photo_url with the search endpoint.

Endpoint

/api/v1/users/search

Expected Response

{
    "users": [
        {
            "username": "username",
            "photo_url": "https://example.com/image/username"
        }
    ]
}

Add google token auth endpoint to docs

Description

Add google authentication endpoint into authentication endpoint docs

Endpoint

• Google

Allow API Token Authentication via Url Parameter

Description

Vendors like ArcGIS do not provide more secure methods for authentication via http headers or cookies. In order to load collections from QwikGeo into ArcGIS we need to have authencation within a url parameter.

Tasks

• Create an overall auth method that checks for a JWT token in the headers or url parameters.

Remove pages from docs

Description

Remove endpoints from docs that are not used

Pages

• Services
• Tiles
• Getting Started

Collections API - Find Closest Feature/Features

Description

Create an endpoint that returns the closest feature to a specified latitude and longitude

Method

GET

Inputs

Latitude
Longitude
filter:optional
limit:optional
offset:optional

Output

Geojson FeatureCollection

Help

https://www.crunchydata.com/blog/a-deep-dive-into-postgis-nearest-neighbor-search

Update docs for imports api

Add example files for download.

Update create_table endpoint update to create metadata in database

Endpoint: /api/v1/items/tables/create_table

After the table is created and geometry column is added, call utilities.create_table to create metadata in database.

Add linting for imports route using pylint

Currently there is no linting validation on the imports route.

Delete all items in portal when user delete's their username

Description

If the /users/user DELETE method is called, delete all items in portals.

Items to delete

• Tables
• Items
• user_data tables

Create docs for users endpoints

Description

Update docs for all users endpoints

Endpoints

• Create User
• Get User
• Update User
• Delete User
• User Search

username vs user_name

Description

There is currently a discrepancy in the code where some parts of the api use username and others use user_name. Convert all code to use username.

Add qwikgeo image back

Description

Qwikgeo image got removed from the assets folder.

Tasks

• Add image

Create admins for groups

Description

Only allow users who are admins of a group the ability to update/delete a group.

Database Model

• Add GroupAdmin

class GroupAdmin(models.Model):
    """Model for group_admin in database"""

    id = fields.IntField(pk=True)
    group_id: fields.ForeignKeyRelation[Group] = fields.ForeignKeyField(
        "models.Group", related_name="group_admins", to_field="group_id"
    )
    username = fields.CharField(500)

Update model to include a new field admins which is a ReverseRelation to GroupAdmin

Groups Endpoints

• Check if user is in admin group for POST request
• Check if user is in admin group for DELETE request

Analysis API - Find Nearest Point for each Polygon/ Find Nearest Polygon to Each Point

Description

Allow user to perform analysis to find nearest point for each polygon or vice versa

Example:

SELECT points.name, polygons.name, points.dist*100 as distance_in_kilometers, polygons.geom
FROM counties as polygons
CROSS JOIN LATERAL (
  SELECT points.name, polygons.geom, points.geom <-> polygons.geom AS dist
  FROM starbucks AS points
  ORDER BY dist
  LIMIT 1
) points

Tasks

• Create methods in analysis_queries
• Create endpoints in router for analysis
• Create docs

Add response examples for items route.

Create docs for groups endpoints

Description

Create docs for all groups endpoints

Endpoints

• Groups
• Group Search
• Create Group
• Get Group
• Update Group
• Delete Group

Items - Table Autocomplete

Description

Search for distinct values in table using column.

Endpoint

api/v1/items/tables/{table}/autocomplete/?column={column}&q={q}&limit={limit}

Table Autocomplete Model

table: str
column: str
q: str
limit: int=10

Response

["value_1","value_2",...]

Imports API - Enhance Validation/Cleanup

Description

Currently there is not much validation that occurs for the imports api. It assumes use will input the correct file types and have clean data. This is not the case in reality. In order to deal with this, we need to add further validation to help produce appropriate errors and clean data automatically for users.

Tasks

• Validate file types
• Validate header columns for leading and trailing white spaces
• Validate lat lng columns with no data in some columns
• Validate a csv with lat lng data with first column with no lat lng data
• Validate joining to a zip code polygon dataset from a csv with a zip code starting with a zero

Analysis Api - Spatial Join

Description

Allow user to perform spatial join analysis

Example:

WITH join_points AS(
	SELECT points.gid, polygons.name, polygons.state_name, polygons.population
	FROM starbucks AS points
	JOIN counties AS polygons
	ON ST_INTERSECTS(polygons.geom, points.geom)
)
SELECT points.gid, join_points.*
FROM starbucks AS points
LEFT JOIN join_points
ON points.gid = join_points.gid
ORDER BY state_name

Tasks

• Create spatial_join method in analysis_queries
• Create endpoint in router for analysis
• Create Docs

Tables API - Bad column response

Description

Currently if a user provides a incorrect column name, they receive a 500 error. Instead return a 400 error with a description that the column does not exist for that table.

Endpoints

• Table Autocomplete
• Custom Bin Values
• Numeric Bins
• Bins
• Statistics

Implement Part 4 of OGC Features API in collections router.

http://docs.ogc.org/DRAFTS/20-002.html

Endpoints to create:

• POST - adding an object URL: /api/v1/collections/{scheme}.{table}/items/
• PUT - object replacement URL: /api/v1/collections/{scheme}.{table}/items/{id}
• DELETE - deleting an object URL: /api/v1/collections/{scheme}.{table}/items/{id}
• PATCH - changing the object URL: /api/v1/collections/{scheme}.{table}/items/{id}

Endpoints to delete:

• /api/v1/items/tables/add_row
• /api/v1/items/tables/delete_row
• /api/v1/items/tables/edit_row_attributes
• /api/v1/items/tables/edit_row_geometry

Docs

• Remove endpoints from docs
• Update for new endpoints from docs

Conformance

• Update conformance endpoint for part 4

For each new endpoint use the method utilities.validate_table_access with write_access=True to ensure the user has the correct access to update a feature

Create example responses for main.py

Update docs for collections api

Description

Add missing endpoints into docs

Endpoints

• Queryables
• Tiles
• Tile
• Tiles Metadata
• Get Tile Cache Size
• Delete Tile Cache Size

Allow filtering of results for items/tables via title,tags and descriptions

Allow a filter parameter that searches the title, tags, and descriptions for any that contain the search text.

Convert geojson links to type of application/geo+json

Description

In the collections api, we currently return links that are geojson as application/json. They should be converted to application/geo+json.

Create min db and max db connection settings in config.py

Description

Currently the min and max connections are hardcoded. Allow user to pass in settings via env file for their database, have those loaded into the config.py file, and use those settings in db.py

Variable Names

• DB_MIN_NUMBER_OF_CONNECTIONS
• DB_MAX_NUMBER_OF_CONNECTIONS

Imports API - ArcGIS - Timestamp Conversion

Description

Currently if importing a service from ArcGIS the timestamp will stay in epoch time 1672631874000. When converting data into geojson format, convert these columns to human readable timestamps

Tasks

• Convert timestamps to human readable

Imports API - OGC API Features

Description

Allows users to import data via a OGC API - Features collection using https://pypi.org/project/ogcFeatureApiCollectionDownloader/

Endpoint

api/v1/imports/ogc_api_features_collection

Analysis API - Create table metadata if succesful

Description

Currently if an analysis is ran it does not save table metadata.

Create doc examples for landing page and conformance

Clean up docs

Description

With many changes to api endpoints we need to adjust docs to have correct urls.

Tasks

• Update docs for correct urls

Add timestamp to response of collections/{collection}/items

Description

Currently we are missing the timeStamp property in our response. We need to add the timeStamp property.

Example

"timeStamp": "2022-10-11T17:33:08.219953Z"

Update docs for items api

Description

Add docs for missing endpoints

Endpoints

• Items
• Tables
• Table

Add security to imports api endpoint.

Currently all imports code was directly copied from the FastImporter project which has no table security validation. We can use the method utilities.validate_table_access to validate the user has access to the table for endpoints that join data to another table.

Add security to analysis endpoints

Currently all analysis code was directly copied from the FastGeofeature project which has no table security validation. We can use the method utilities.validate_table_access to validate the user has access to the table before performing each analysis.

Add linting for analysis route using pylint

Currently there is no linting validation on the analysis route.