pdok / ngr-services-spider

CLI app to retrieve list of services and datasets in a simple JSON format from nationaalgeoregister.nl

NGR SPIDER

CLI (command line interface) application to retrieve list of services and datasets in a simple JSON format from nationaalgeoregister.nl (NGR), leveraging the NGR CSW service.

To install from source run (from the root this repo):

python3 -m pip install -e .

This should install the cli tool ngr-spider:

$ ngr-spider
usage: ngr-spider [-h] subcommand ...

Generate list of PDOK services and/or service layers

positional arguments:
  subcommand
    services  Generate list of all PDOK services
    layers    Generate list of all PDOK layers

options:
  -h, --help  show this help message and exit

Examples

1. To generate a list of layers per service run the following:

ngr-spider layers -p 'OGC:WMS,OGC:WMTS' pdok-services.json
ngr-spider services -p 'OGC:WMS,OGC:WMTS' pdok-services.json

Development

Install dev dependencies and package from source:

python3 -m pip install -e ".[dev]"  

To format code install black en run from the root of this repo:

black .

Delete unused imports with autoflake:

autoflake --remove-all-unused-imports -i -r .

Organise and order imports with isort:

isort  -m 3 .

Run mypy for type checking (from root):

mypy ngr_spider 

Sorting Rules Example

When running the layers command in flat mode (--mode flat), it is possible to sort the layers in the output by passing a path to a JSON file containing, sorting rules. See below for an example a sorting rules JSON file (use for example with: ngr-spider layers --sort sorting-rules.json -n 20 -m flat --pretty -p "OGC:WMS,OGC:WMTS" output.json):

[
  { "index": 0, "names": ["opentopo+"], "types": ["OGC:WMTS"] },
  { "index": 10, "names": ["^actueel_orthohr$"], "types": ["OGC:WMTS"] },
  { "index": 11, "names": ["^actueel_ortho25$"], "types": ["OGC:WMTS"] },
  { "index": 12, "names": ["^actueel_ortho25ir$"], "types": ["OGC:WMTS"] },
  { "index": 12, "names": ["lufolabels"], "types": ["OGC:WMTS"] },
  {
    "index": 20,
    "names": ["landgebied", "provinciegebied", "gemeentegebied"],
    "types": ["OGC:WFS"]
  },
  { "index": 30, "names": ["top+"], "types": ["OGC:WMTS"] },
  {
    "index": 32,
    "names": ["^standaard$", "^grijs$", "^pastel$", "^water$"],
    "types": ["OGC:WMTS"]
  },
  {
    "index": 34,
    "names": ["bgtstandaardv2", "bgtachtergrond"],
    "types": ["OGC:WMTS"]
  },
  { "index": 60, "names": ["ahn3+"], "types": ["OGC:WMTS"] }
]

ngr-services-spider's Issues

Functionaliteit OGC:API features correct maken & testen

Doel:

De ngr-spider (https://github.com/PDOK/ngr-services-spider) aanpassen om ook voor OGC API features de juist info van een records via de CSW (Catalogue Service for the Web)

Beschrijving:

De implementatie voor het ophalen van ogcapi features moet uitgebreid worden. Er zijn op dit moment (22-08-2023) nog geen features records in het NGR om de implementatie te testen: Er is slechts een layout gemaakt met 'lege'-functies en TODO's die in dit ticket aangevuld moeten worden. Deze uitbreiding kan gedaan worden zodra PDOK een features service gaat ontsluiten conform OGCAPI features.

Add error handling ATOM HTTP get requests

Add error handling ATOM HTTP get requests, see:

• ngr-services-spider/ngr_spider/util.py

  Line 264 in 99276bc

  r = requests.get(service_record.service_url)

• ngr-services-spider/ngr_spider/models.py

  Line 222 in 99276bc

  r = requests.get(ds_feed_url)

Add exceptionhandler:

except ConnectionError as e:

Add docker image and image build pipeline

Add docker image and image build pipeline