brtle / sl2influxdb

Fetch seedlink data and store them into InfluxDB

Seedlink to InfluxDB

Dump seedlink (seismological) time series into InfluxDB. Use Grafana to plot waveforms, real time latency delay, etc. Maps uses the grafana worldmap-panel plugin.

Dockerfile, docker-compse.yml are available here.

Install

pip install .

Usage

~$ seedlink2influxdb -help
Usage: seedlink2influxdb [options]

Options:
  -h, --help            show this help message and exit
  --dbserver=DBSERVER   InfluxDB server name
  --dbport=DBPORT       InfluxDB server port
  --slserver=SLSERVER   seedlink server name
  --slport=SLPORT       seedlink server port
  --fdsnserver=FDSN_SERVER[:PORT]
                        fdsn station server name
  --streams=STREAMS     streams to fetch (regexp): [('.*','.*','.*Z','.*')]
  --flushtime=NUMBER    when to force the data flush to influxdb
  --db=DBNAME           InfluxDB name to use
  --dropdb              [WARNING] drop previous database !
  --keep=NUMBER         how many days to keep data
  --recover             use seedlink statefile to save/get streams from last

Example :

seedlink2influxdb
    --dbserver localhost \
    --dbport 8086 \
    --slserver rtserve.resif.fr \
    --fdsnserver http://ws.resif.fr \
    --db eost2 \
    --keep 1

Note

Fdsnserver request (--fdsnserver option) is optional. It is only used and performed at the begining and could be slow (if too much stations info are requested) ! Data collected are only used to get station coordinates and are converted to geohash, needed to plot measurements on a map.

Screenshot

Delay/Latency Map

Latency & traces for multiple stations

Waveform, RMS, latency plots for a given station

InfluxDB

InluxDB data representation (measurements, tags, fields, timestamps).

Measurements:

• queue: internal messages producer queue (seedlink thread) and consumer queue (influxdb exporter thread)
  • tags
    • type=(consumer|producer)
  • field
    • size=queue size
  • timestamp
• count : amplitude in count (waveforms)
  • tags
    • channel = channel name (eg. FR.WLS.00.HHZ)
  • field
    • value = amplitude
  • timestamp
• latency : seedlink packet propagation time from station to datacenter
  • tags
    • channel = channel name
  • field
    • value = latency value
  • timestamp
• delay : time since last seedlink packet was received
  • tags
    • channel = channel name (eg. FR.WLS.00.HHZ)
    • geohash = station coordinates geohash formated
  • field
    • value = latency value
  • timestamp

Dependencies

• obspy
• python InfluxDB
• geohash
• grafana (with worldmap-panel plugin)

Using docker-compose

A docker-compose is available to quickly setup influxdb and grafana.

Data storage

If you are running this project for the first time, you have to create a influxdb data docker volume in order to keep your measurements between restarts:

docker volume create --name=sl2influxdb_influxdb_data

Start services

For RaspberryShake

This configuration is ready to be run, assuming your raspeberryshake is in you local network and reachable using raspberryshake.local address.

To start all the containers (influxdb, seedlink fetcher and grafana):

docker-compose up -d rshakegrafana

Check the logs to see if seedlink data is fetched without problem:

docker-compose logs -f sl2raspberryshake

For Generic Seedlink Server

You need to customize the docker-compose.yml file to set properly this environement variables:

• SEEDLINK_SERVER
• FDSN_WS_STATION_SERVER
• SEEDLINK_STREAMS (without space !!)

Then, starts the container:

docker-compose up -d grafana

To check the logs if seedlink data is fetched well:

docker-compose logs -f sl2generic

Acces to grafana interface

Then launch you preferred browser and go to http://localhost:3000, with:

• user: admin
• passwd : admin

Stop services

docker-compose down -v