geonovum / sospilot

Sensor Observation Service (SOS) and data management for Air Quality data from RIVM The Netherlands

sospilot

This repository is archived.

Sensor Observation Service (SOS) and data management for Air Quality data from RIVM The Netherlands. In this project Air Quality data provided by Het Rijksinstituut voor Volksgezondheid en Milieu (RIVM) is harvested and converted for further dispatching via the OGC SOS INSPIRE-enabled web-service and other standards like WMS/WFS and E-reporting for Shared Environmental Information System (SEISs) like the EU Shared Environmental Information System for reporting ambient air quality (2011/850/EU).

The SOSPilot website is a landing page for docs, viewers, services and more. All documentation can be found at: sospilot.readthedocs.org

In september 2015 support for Air Quality, Meteo and Audio data for the Smart Emission project was added. Read also about this project in a dedicated chapter in the documentation: sospilot.readthedocs.org/en/latest/smartemission.html

The Smart Emission project now has a GitHub repo of its own at github.com/smartemission/smartemission. The SE-resources of sospilot have been moved there.

sospilot's Issues

Use sos-based RIVM LML Files

Currently the XML files at URL http://www.lml.rivm.nl/xml/ are harvested. But we should use
http://www.lml.rivm.nl/sos/

SOS database reset does not work via Admin GUI

Sometimes one wants to emtpy the DB. This does not work via the Admin GUI (400 or 500 error).

The procedure below works:

See data and scripts at https://github.com/Geonovum/sospilot/tree/master/data/sosdb. Using this procedure, no
reinstall of the .war file is requried or any other Admin reset (somehow an Admin reset did not work).

As root do

• Stop Tomcat (command /opt/bin/tcdown)
• Clean DB: reinstall PG schema using: https://github.com/Geonovum/sospilot/blob/master/data/sosdb/empty-dm-dump.sql
• Remove /var/www/sensors.geonovum.nl/webapps/sos/cache.tmp.
• Start Tomcat (command /opt/bin/tcup)

Install HTML5 52North sensorwebclient (js-sensorweb-client)

For demo purposes, install and configure 52North sensorwebclient. Request for latest version already done by Just.

SOS InsertObservation request duration increases as SOS database grows

This is observed in the timing data from the cron job that publishes rivm_lml measurements via InsertObservation to the SOS server. Each half hour max 1000 measurements are published. In the first rounds about 5 Observations per/sec could be published, see logging data:

2014-06-09 15:47:34,854 httpoutput INFO Response status: code=200 msg=OK
2014-06-09 15:47:35,123 httpoutput INFO Response status: code=200 msg=OK
2014-06-09 15:47:35,450 httpoutput INFO Response status: code=200 msg=OK
2014-06-09 15:47:35,668 httpoutput INFO Response status: code=200 msg=OK
2014-06-09 15:47:35,873 httpoutput INFO Response status: code=200 msg=OK
2014-06-09 15:47:36,088 httpoutput INFO Response status: code=200 msg=OK

After a few days with about 100000 observations in the SOS database, this has slowed down to one insert taking between 1 and 2 seconds:

2014-06-11 17:02:57,890 httpoutput INFO Response status: code=200 msg=OK
2014-06-11 17:02:59,328 httpoutput INFO Response status: code=200 msg=OK
2014-06-11 17:03:00,633 httpoutput INFO Response status: code=200 msg=OK
2014-06-11 17:03:01,900 httpoutput INFO Response status: code=200 msg=OK
2014-06-11 17:03:03,255 httpoutput INFO Response status: code=200 msg=OK

Smart Emission: locatie parameters (Lat, Lon) decodering

In ieder sample zit een lokatie, echter denk ik dat de omrekening vnl voor Latitude niet klopt. Dit staat bijv in sample data file bijv.

07/24/2015 07:25:41,P.UnitSerialnumber,1
07/24/2015 07:25:41,S.Longitude,5914103
07/24/2015 07:25:41,S.Latitude,53949942
07/24/2015 07:25:41,S.SatInfo,90889

Latitude zou *100 moeten zijn Longitude *10. Echter 53.949942 5.914103
is bijna bij Denemarken...Longitude zou idd bij Nijmegen kunnen zijn.

Get FIWARE running on SOSPilot platform

See http://sospilot.readthedocs.org/en/latest/fiware.html

Plan is to do a simple scenario with Orion Context Broker (OCB) and IoTAgent (IDAS) using UL protocol first:

• create IoT service via IoTAgent (IDAS)
• create IoT device via IoTAgent (IDAS)
• observe new Entity in OCB
• send temperature via IoTAgent
• observe changed Entity via OCB

Harmonization of SOS Semantics NL/BE .. EU

Use same codelists, pollutant naming, offering, procedures, etc.
SensorML, Sensor description etc.

Stations don't show in SensorWebClient 3.3.0 SNAPSHOT

See SWE mailing list/forum:
http://sensorweb.forum.52north.org/Stations-don-t-show-in-SensorWebClient-3-3-0-SNAPSHOT-tc4027389.html

I just built and deployed SWC 3.3.0 from GitHub
https://github.com/ridoo/SensorWebClient.git and tried to attach it to
our SOS 52N-SOS-INSPIRE-with-RestAPI_20140519.zip with some patches.

The SOS runs ok, we have inserted Sensors and Observations via SOS-T.
The built-in client works ok. And Simon was able to demo our SOS with
stats in Aalborg last week.

The SWC runs at http://sensors.geonovum.nl:8080/swc. Belgium stations
show up ok and give time series results. Our SOS server runs at
http://sensors.geonovum.nl/sos, in the same Tomcat instance as the SWC.

I have added our instance as follows:

 <instance> 
     <itemName>Air Quality The Netherlands</itemName> 
     <url>http://sensors.geonovum.nl/sos/sos</url> 
     <version>2.0.0</version> 
     <waterML>false</waterML> 
     <supportsFirstLatest>false</supportsFirstLatest> 

    <metadataHandler>org.n52.server.da.oxf.DefaultMetadataHandler</metadataHandler> 
     <adapter>org.n52.server.da.oxf.SOSAdapter_OXFExtension</adapter> 
     <requestChunk>300</requestChunk> 
     <autoZoom>true</autoZoom> 
     <llEasting>3.0</llEasting> 
     <llNorthing>50.7</llNorthing> 
     <urEasting>7.2</urEasting> 
     <urNorthing>53.8</urNorthing> 
     <defaultZoom>11</defaultZoom> 
 </instance> 

I have attached output from Tomcat, messages are both from SWC and SOS.
At some points I see something like:

17:51:53.088 [ajp-bio-8009-exec-886] DEBUG SosService - Outgoing 
response for request No. 116235 is committed = true 
17:51:53.092 [Timer-8418] WARN  DefaultMetadataHandler - No FOI 
references found for procedure 
 'http://sensors.geonovum.nl/rivm-lml/procedure/244'. 
 17:51:53.093 [Timer-8418] WARN  DefaultMetadataHandler - ==> Reference 
 all (0) available. 
 17:51:53.093 [Timer-8418] DEBUG DefaultMetadataHandler - Still #41 
 responses to go ... 

So why are the stations not showing up? It does not find FOIs?
Axis-ordering issue?

More docs via our homepage: http://sensors.geonovum.nl

Thanks and best regards,

--Just van den Broecke
www.justobjects.nl

Create OGC Web Coverage Service from RIVM RIO APS output files

Uit: http://www.rivm.nl/Documenten_en_publicaties/Wetenschappelijk/Rapporten/2014/juni/Verbeterde_actuele_luchtkwaliteitskaarten_Validatie_interpolatiemethode_RIO_Nederland :
"Het RIVM presenteert elk uur de actuele luchtkwaliteitskaarten voor stikstofdioxide, ozon en fijn stof in Nederland op de website van het Landelijk Meetnet Luchtkwaliteit (LML). Meetgegevens van representatieve LML-locaties worden gebruikt om te berekenen wat de concentraties voor de rest van Nederland, buiten de meetpunten om." De RIO (residual interpolation optimised for ozone) interpolatiemethode, wordt hierbij gebruikt voor luchtkwaliteitskaarten om het publiek informeren over de actuele luchtkwaliteit.

Het resultaat van de RIO-procedures zijn ASCII grid bestanden in het zgn APS-formaat. Het formaat van deze bestanden is bekend (verkregen van RIVM via D Mooibroek). In een .aps bestand staat op de eerste regel metadata, met gegevens als datum, uur, component-naam, cell-grootte en origin (RD coordinaten). In rest van het bestand volgt dan de data, per rij een N-aantal kolommen. In feite een 2 dimensionaal array. De APS proefdata en APS-format documentatie staat in GitHub onder https://github.com/Geonovum/sospilot/tree/master/data/rivm-rio

In het kader van het project is er de wens om deze data als een OGC Web Coverage Service te ontsluiten. Dit zal ook aansluiting geven bij INSPIRE. Daarna zal data uit bijv het Smart Emission project (RU Nijmegen) in het RIO model ingevoerd worden en kan de output direct als WCS (en WMS) ontsloten worden.

Daartoe zal in eerste instantie een aantal APS proefbestanden omgezet worden in GeoTIFF bestanden. Deze GeoTIFFs kunnen vervolgens geserveerd worden als WCS en WMS via de bestaande SOSPilot GeoServer. Via WMS kan ook styling via SLD aangewend worden zodat de output sterk gelijkend wordt op de PNGs door RIVM gegenereerd uit de APS bestanden als in plaatje hieronder.

Enable SOS in Heron Viewer

For demos and for downloading data in several formats

Create e-reporting (IPR) with RIVM data

SOS server is down

Hi there,
the SOS with the URL

http://sensors.geonovum.nl/sos

is not responding.

Kind regards!

Add timeseries/SOS data and web services for historic KNMI data

KNMI provides free datasets for various measurements. The rainradar provides WMS-Time, but it would be interesting to add all existing other data like temperature, pressure, wind direction/force, also to correlate with pollution measured values.

Via http://data.knmi.nl, https://data.knmi.nl/portal/KNMI-DataCentre.html

De volgende KNMI datasets lijken van belang.

• Waarneemstations
• Etmaalgegevens KNMI stations (alle historie sinds 1951)
• Actuele 10 min data KNMI stations (laatste metingen)

Soortgelijk als voor LML AQ kan een simpele ETL worden opgezet:

• eenmalig alle historie laden, dan incrementeel laatste metingen
• waarneemstations, evt coordinaten

verder zijn daarmee services te maken zoals WMS-Time, WFS en SOS.

Fileformaat: .nc (hierarchical?).

Create INSPIRE Metadata for services and data(sets)

For what resources should metadata be provided? WFS, WMS, SOS? And for what data? The observations & measurements, the aggregated datasets?

Do we need a sensor registry to describe the sensors?

ETL Step 3 alternative: value tables for SOS

It seems possible to have data outside of SOS-tables and provide a mapping file to SOS such that SOS-T publication is not required. In theor the Core AQ DB could serve as data-value tables to the SOS.

Publish coverages of LML with WMS and/or WCS

See www.lml.rivm.nl for some examples, but there are probably coverages available. Publish these through INSPIRE View and Download services

Idea: AQ maps using RIO and CORINE

See http://wwwdev.irceline.be/en . Very nicely done. RIVM has Rio-calculated data as well. We can integrate this with cross-border cells to compare.

Put Station Name into all Station/Sensor appearances

The Station EIONET data only has municipality but not for all stations.
In reality (GELUK-DB) each station has a real name. This needs to be intagrated into the ETL

InsertObservation: use of "identifier"

I insert something like:

 "observation": {{
   "identifier": {{
    "value": "107-NO-01/05/2014-20",
    "codespace": "http://www.opengis.net/def/nil/OGC/0/unknown"
  }},

and this translates in a GetObservation to gml:identifier:

<om:OM_Observation gml:id="o_58D6F37EAA4CD562A39045D92248E79DE6CE8A22">
  <gml:identifier codeSpace="http://www.opengis.net/def/nil/OGC/0/unknown">107-NO-01/05/2014-20</gml:identifier>

But now the GetCapabilities doc keeps growing, see http://sensors.geonovum.nl/sos/service?service=SOS&request=GetCapabilities, as each Observation identifier is published as a "parameter AllowedValue". E.g.

<ows:Parameter name="observation">
  <ows:AllowedValues>
  <ows:Value>107-NO-01/05/2014-20</ows:Value>
  <ows:Value>107-NO-01/05/2014-21</ows:Value>
  <ows:Value>107-NO-01/05/2014-22</ows:Value> 

InsertSensor: observableProperty always has to be unique in SOS-T

InsertSensor: it looks like the observableProperty always has to be unique. As these represent a chemical component, I assumed that I could use an identifier like: http://sensors.geonovum.nl/rivm-lml/observableProperty/NO2 and later look for an official code. But this is only accepted for the first inserted station. For the next stations I get like a "duplicate key error", at least that the prop already exists. So I now use the station id: http://sensors.geonovum.nl/rivm-lml/observableProperty/{station_id}/NO.

Make SOS INSPIRE compliant

Together with Simon

Connect Geonovum Davis weather station to SOS and other services

The title says it all.

Create GML for AQD Stations

Separate issue, part of #7.

For Stations, the XML schemas define a type AQD_StationType. This is an extension of EF's ef:EnvironmentalMonitoringFacilityType

See XSD.

So there is more than EF only to provide.

Setup ETL and OGC Services for Smart Emission Project (Nijmegen) data

"Het ‘Smart Emission’ (project) draait om het in kaart brengen van luchtkwaliteit in de stad op een fijnmazig schaalniveau, door inwoners met zogenoemde burger-sensor-netwerken. ", zie
http://www.ru.nl/gpm/onderzoek/research-projects/smart-emission

Citygis.nl heeft techniek geleverd op basis Josene Sensor van Intemo om metingen te doen en te verzamelen. Deze metingen, huidige componenten CO, NO2 en O3 dienen in de SOS en andere OGC services gebracht te worden op het sensors.geonovum.nl platform, gelijk de RIVM LML metingen.

De huidige metingen zijn via FTP beschikbaar als accumulerende bulk files, 1 file per sensor. Er wordt gewerkt aan een service om deze metingen gestructureerder aan te bieden, bijv dagfiles.

Dezelfde ETL-principes kunnen worden aangewend als eerder (2014) voor RIVM LML:

• Stap 1: verzamelen ruwe data files en stations metadata
• Stap 2: inlezen ruwe data en stations in reguliere (PostGIS) database tabellen
• Stap 3: publiceren stations en observaties naar SOS vanuit deze tabellen

Daarnaast moeten OGC services voor WMS-Time worden opgezet om in de reguliere viewer te tonen:
http://sensors.geonovum.nl/heronviewer. Hierdoor is snel te vergelijken met LML RIVM waarden.