knmi / covjson-pydantic

Pydantic models for CoverageJSON

CoverageJSON Pydantic

This repository contains the coveragejson-pydantic Python package. It provides Pydantic models for CoverageJSON. This can, for example, be used to develop an API using FastAPI serving or receiving CoverageJSON.

Install

pip install covjson-pydantic

Or you can install directly from source:

pip install git+https://github.com/KNMI/covjson-pydantic.git

Usage

from datetime import datetime, timezone
from pydantic import AwareDatetime
from covjson_pydantic.coverage import Coverage
from covjson_pydantic.domain import Domain, Axes, ValuesAxis, DomainType
from covjson_pydantic.ndarray import NdArray

c = Coverage(
    domain=Domain(
        domainType=DomainType.point_series,
        axes=Axes(
            x=ValuesAxis[float](values=[1.23]),
            y=ValuesAxis[float](values=[4.56]),
            t=ValuesAxis[AwareDatetime](values=[datetime.now(tz=timezone.utc)])
        )
    ),
    ranges={
        "temperature": NdArray(axisNames=["x", "y", "t"], shape=[1, 1, 1], values=[42.0])
    }
)

print(c.model_dump_json(exclude_none=True, indent=4))

Will print

{
    "type": "Coverage",
    "domain": {
        "type": "Domain",
        "domainType": "PointSeries",
        "axes": {
            "x": {
                "values": [
                    1.23
                ]
            },
            "y": {
                "values": [
                    4.56
                ]
            },
            "t": {
                "values": [
                    "2023-09-14T11:54:02.151493Z"
                ]
            }
        }
    },
    "ranges": {
        "temperature": {
            "type": "NdArray",
            "dataType": "float",
            "axisNames": [
                "x",
                "y",
                "t"
            ],
            "shape": [
                1,
                1,
                1
            ],
            "values": [
                42.0
            ]
        }
    }
}

Contributing

Make an editable installation from within the repository root

pip install -e '.[test]'

Running tests

pytest tests/

Linting and typing

Linting and typing (mypy) is done using pre-commit hooks.

pip install pre-commit
pre-commit install
pre-commit run

Related packages

• edr-pydantic - Pydantic data models for the Environmental Data Retrieval (EDR) API
• geojson-pydantic - Pydantic data models for the GeoJSON spec

Real world usage

This library is used to build an OGC Environmental Data Retrieval (EDR) API, serving automatic weather data station data from The Royal Netherlands Meteorological Institute (KNMI). See the KNMI Data Platform EDR API.

TODOs

Help is wanted in the following areas to fully implement the CovJSON spec:

• The polygon based domain types are not supported.
• The Trajectory and Section domain type are not supported.
• The NdArray only supports float data.
• Not all requirements in the spec relating different fields are implemented.

License

Apache License, Version 2.0

covjson-pydantic's Issues

Axes x and y coordinates cannot be integers, but standard doesn't prohibit this

I'm creating CoverageJSON objects with coordinates from a UTM coordinate system, where the coordinates are whole numbers. As far as I can tell from the standard section 9.6.1.1 it is not a requirement for these coordinates to be float, in fact the word int or float is never mentioned.

However the code for the Axes class states:

class Axes(CovJsonBaseModel):
    x: Optional[Union[ValuesAxis[float], CompactAxis]] = None
    y: Optional[Union[ValuesAxis[float], CompactAxis]] = None
    ...

Shouldn't it be:

class Axes(CovJsonBaseModel):
    x: Optional[Union[ValuesAxis[Union[float, int]], CompactAxis]] = None
    y: Optional[Union[ValuesAxis[Union[float, int]], CompactAxis]] = None
    ...

The same goes for CompactAxis:

class CompactAxis(CovJsonBaseModel):
    start: float
    stop: float
    ...

Key="null" in output

Some optional fields write out key="null" when not used. How can that be disabled?

Example showing ValuesAxis giving dataType and coordinates as null:

    c = Coverage(
        id="isobaric",
        type="Coverage",
        domain=covjson_pydantic.domain.Domain(
            domainType=covjson_pydantic.domain.DomainType.vertical_profile,
            axes=covjson_pydantic.domain.Axes(
                x=covjson_pydantic.domain.ValuesAxis[float](values=[point.y]),
                y=covjson_pydantic.domain.ValuesAxis[float](values=[point.x]),
                z=covjson_pydantic.domain.ValuesAxis[float](values=isobaric_values),
                t=covjson_pydantic.domain.ValuesAxis[AwareDatetime](values=[datetime.now(tz=timezone.utc)]),
            ),

{
  "id": "isobaric",
  "type": "Coverage",
  "domain": {
    "type": "Domain",
    "domainType": "VerticalProfile",
    "axes": {
      "x": {
        "dataType": null,
        "coordinates": null,
        "values": [
          55.45
        ],
        "bounds": null
      },
      "y": {
        "dataType": null,
        "coordinates": null,
        "values": [
          -1.35
        ],
        "bounds": null
      },

Missing cs in referencing -> coordinates -> system

Please look at https://docs.ogc.org/cs/21-069r2/21-069r2.html -> Annex A -> Coverage Examples -> A.1. Vertical Profile Coverage which looks like this:

    "referencing": [{
      "coordinates": ["z"],
      "system": {
        "type": "VerticalCRS",
        "cs": {
          "csAxes": [{
            "name": {
              "en": "Pressure"
            },
            "direction": "down",
            "unit": {
              "symbol": "Pa"
            }
          }]
        }
      }
    }

I can't find any field "cs" in class ReferenceSystem at https://github.com/KNMI/covjson-pydantic/blob/e74abaadfecd605ce1dea05643dfa3434620e211/src/covjson_pydantic/reference_system.py#L20C1-L20C1

The class allows for "extra", so one can add non-defined fields. How to do that in a correct manner?

Null values in Axes, properties, x, anyOf, type

Hi

Please se my openapi.json. The relevant code generating this is around this line https://github.com/metno/edrisobaric/blob/f344ca8a414e75b52726a2d727737144bddac8f4/app/routes/position_page.py#L111.

These null values cause trouble in EDR validation and API code generation (https://github.com/OpenAPITools/openapi-generator).

I'm unsure why the null appears. X should not be an optional value. Have you seen the problem in your use, or am I using the library wrong?

`i18n=Dict[LanguageTag, str]` throws warning

When using:

i18n = Dict[LanguageTag, str]

The following warning is thrown, when serializing to JSON

Expected `definition-ref` but got `LanguageTag` - serialized value may not be as expected

This seems to be related to pydantic/pydantic#6467 or pydantic/pydantic#6422.

For now, we are using:

i18n = Dict[str, str]

Enable strict modus for `domainType`

There is a bug/problem with FastAPI, where it doesn't use model_validate_json() when passing model data as POST data. This causes domainType not to be validated in strict mode, when it is passed as str in a JSON.

For now we applied the following workaround:

    @field_validator('domainType', mode='before')
    @classmethod
    def value_to_enum(cls, v):
        if isinstance(v, str):
            return DomainType(v)
        return v

See also: tiangolo/fastapi#9868

Enable `strict=True` for `ValuesAxis`

Pydantic v2 strict mode has been enabled everywhere, except for ValuesAxis.

When enabled the following test would fail:

>       assert object_type.model_validate_json(json_string).model_dump_json(exclude_none=True) == json_string
E       pydantic_core._pydantic_core.ValidationError: 4 validation errors for Axes
E       t.values.0
E         Input should be a valid datetime [type=datetime_type, input_value='2008-01-01T04:00:00Z', input_type=str]
E           For further information visit https://errors.pydantic.dev/2.3/v/datetime_type

This seems to be an issue in the combination between Generics (ValuesT) and datetime and strict mode when converting between JSON <-> Pydantic. Needs further investigation.