bertrik / ttnhabbridge

Bridge between TheThingsNetwork/Helium and the habitat network, to receive amateur balloon telemetry using LoRaWAN

ttnhabbridge

Bridge between TheThingsNetwork/Helium and the habitat network, to receive telemetry from amateur balloons using LoRaWAN

See https://revspace.nl/TTNHABBridge for a more detailed description of this project. See https://travis-ci.org/bertrik/ttnhabbridge for the Travis CI status

Development environment

I develop this on in Java 11 in Eclipse.

Steps to install everything:

• download a Java 11 JDK and put it in tools/jdk
• download Eclipse and put in in the tools/eclipse directory (for example)

Steps to prepare the Eclipse environment:

• open a command line to the checked out project, enter the 'ttnhabbridge' directory
• type 'env.bat' to initialise the tool paths etc
• type 'gradlew eclipse' and watch dependencies being downloaded from the internet
• start Eclipse, using the 'workspace' directory as workspace
• import sub-projects (ttnhabbridge, cayenne) into the workspace

Steps to update source code from github:

• in the top-level 'ttnhabbridge' directory, enter 'git pull'

Steps to create the executable package:

• in the root directory of the project, type './gradlew assemble' (or just 'gradlew assemble' on Windows)
• the .tar (for Linux) and .zip (for Windows) file can be found under ttnhabbridge/build/distributions

Deployment

Steps to deploy the application:

• unzip the .zip or .tar file
• cd into the application directory
• start the .bat or .sh file, if no .yaml configuration file exists, a default one is generated
• edit and install the systemd service file, if desired (instructions inside the .service file)

Usage with Helium Network

The key difference between using The Things Network and the Helium Network MQTT integration is The Things Network hosts their packet broker, while the helium network does not. Therefore, we have to setup our own packet broker, and get Helium to and TTNhabbridge to connect to it. Here are a few steps on how to do it.

1. Setup a MQTT packet broker on machine with port 1883 open to the internet. Docker is the simplest way to do it. Run the following command, changing the <username> and <secret_password> password fields to something secret :

docker run \
    --name mosquitto \
    -d \
    -p 1883:1883 \
    -p 9883:9883 \
    -e MQTT_USERNAME=<username> \
    -e MQTT_PASSWORD=<secret_password> \
    cburki/mosquitto:latest

2. Now make Helium to speak with the MQTT broker in the earlier step. To do that, first add a label to your tracker device registration. In this case, it has been labeled icspace26

3. Next in your integrations tab, make a new MQTT integration. Your endpoint address should be in the format mqtt://<username>:<secret_password>@your-ip-or-domain.com:1883. Your <username> and <secret_password> fields should be the same as the ones your set in step 1.

4. Next, head over to the flows tab and connect the label box with the mqtt integrations box. You should immediately see some data appearing on your MQTT packet broker log. This will be log data from Helium servers connecting with your packet broker. The log entries would look like this:

mosquitto_1     | 1650374268: New connection from 52.8.80.146 on port 1883.
mosquitto_1     | 1650374268: New client connected from 52.8.80.146 as 8fda71a3-be09-44b0-b3b0-611fb5a54de2 (p2, c0, k30, u'medad').

5. Finally, update the ttnhabbridge.yaml with your MQTT packet broker's address and credentials. Helium will publish incoming messages to your MQTT broker. TTNhabbridge will subscribe to your broker, repackage messages and post it on Habhub. An example ttnhabridge.yaml file looks like this:

---
thethingsnetwork:
  url: "tcp://eu1.cloud.thethings.network"
  user: "icss-lora-tracker@ttn"
  pass: "NNSXS.JDUJCMUYUNDVPIOZ53TN6GBTT2P74NBH5FZXYXQ.OCWSRRQ4E46M5OMNATY35GJ6lkaBH6MOWZ5QFP32DMLP823S6M2A"
  topic: "v3/+/devices/+/up"
helium:
  url: "tcp://beta.medadnewman.co.uk:1887"
  user: "medad"
  pass: "secret_password"
  topic: "helium/+/rx"
habitat:
  url: "http://habitat.habhub.org"
  timeout: 60
gwCacheExpirationTime: 600
payloadEncoding: "cayenne"

ttnhabbridge's Issues

Is it time to extend this bridge to Helium Network?

In many ways, The Things Network and the Helium network are very similar; End-node hardware can be used for both networks.
I am planning a launch of a LoRaWAN tracker transmitting on The Things Network V3 and the Helium network on alternate transmissions. This should increase the coverage a bit, especially in America.

I will take a stab at it in a fork of this repo.

Here is an example packet from the Helium network:

{
  "id": "3e9d7a33-a306-4e25-9db2-5480e7e29967",
  "router_uuid": "48cb0514-4037-4ea1-b0ee-b1ee2078e20c",
  "category": "uplink",
  "sub_category": "uplink_unconfirmed",
  "description": "Unconfirmed data up received",
  "fcnt_up": 6,
  "payload": "IQAAJa6rHt3/qgEAAAAAAAC/rgAAAAAAAL+uAAAAAAAAv64AAAAAAAC/rgAAAAAAAL+uAAAAAAAAv64AAAAAAAC/rgAAAAAAAL+uAAAAAAAAv64AAAAAAAC/rgAAAAAAAL+uAAAAAAAAv64AAAAAAAC/rg==",
  "payload_size": 115,
  "port": 99,
  "reported_at": "1631642657741"
}

Below is the view from the helium console.

Add option to hardcode a callsign in config file

I would like to be able to hardcode a callsign in the config file, that will override the callsign from ttn/helium. This is useful because the ttnhabbridge may receive packets from helium or ttn with different call signs from the same balloon. They need to be treated as a single balloon by habhub.

If the callsign is not supplied, then use the callsign from ttn/helium.

LoRaWAN retry packets with wrong timestamp being forwarded to Habhub

When flying over areas with lots of LoRaWAN gateways, some TTN gateways seem to send the data later than other gateways to the MQTT broker. These late packets are have the attribute "is_retry":true. This is a problem because these have a later timestamp than the original message and they all get sent to habhub with the wrong timestamp.

For example in the image below, packet 301 is reported to arrive at 3 different times: 14:30:59,14:31:00 and 14:31:04. Only the first is correct.

Therefore, when calculating speed of the balloon, due to the wrong timestamps, the speed readings are incorrect.

I suggest 2 solutions.

1. filter out all MQTT packets that contain the "is_retry":true attribute. Dont use the data from these retry packets. It is likely that the data contained within already came in a packet earlier
   A packet with this retry attribute looks like this.

{
  "app_id": "icss_lora_tracker",
  "dev_id": "icspace22",
  "hardware_serial": "00B76CF36C7CFA8A",
  "port": 99,
  "counter": 286,
  "is_retry": true,
  "payload_raw": "AHMJZwFn/4UCiAe5pgDdPxA6TAMCJyYEARM=",
  "payload_fields": {
    "analog_in_3": 100.22,
    "barometric_pressure_0": 240.7,
    "digital_out_4": 19,
    "gps_2": {
      "altitude": 10635,
      "latitude": 50.6278,
      "longitude": 5.6639
    },
    "temperature_1": -12.3
  },
  "metadata": {
    "time": "2020-10-19T13:03:31.730922239Z",
    "frequency": 868.3,
    "modulation": "LORA",
    "data_rate": "SF8BW125",
    "airtime": 154112000,
    "coding_rate": "4/5",
    "gateways": [
      {
        "gtw_id": "eui-fcc23dfffe0f4c7b",
        "timestamp": 2089330388,
        "time": "2020-10-19T13:03:31.096209Z",
        "channel": 1,
        "rssi": -111,
        "snr": -4.8,
        "rf_chain": 0
      }
    ]
  }
}

Conversely, a packet that contains the correct timestamp and not the retry attribute looks like this:

{
  "app_id": "icss_lora_tracker",
  "dev_id": "icspace22",
  "hardware_serial": "00B76CF36C7CFA8A",
  "port": 99,
  "counter": 287,
  "payload_raw": "AHMJhwFn/3QCiAe4mwDfGhA0DAMCKCkEARI=",
  "payload_fields": {
    "analog_in_3": 102.81,
    "barometric_pressure_0": 243.9,
    "digital_out_4": 18,
    "gps_2": {
      "altitude": 10619,
      "latitude": 50.6011,
      "longitude": 5.7114
    },
    "temperature_1": -14
  },
  "metadata": {
    "time": "2020-10-19T13:05:18.016203852Z",
    "frequency": 868.5,
    "modulation": "LORA",
    "data_rate": "SF8BW125",
    "airtime": 154112000,
    "coding_rate": "4/5",
    "gateways": [
      {
        "gtw_id": "eui-689e19fffe8e615b",
        "timestamp": 2149128972,
        "time": "2020-08-23T11:40:30.367695Z",
        "channel": 2,
        "rssi": -108,
        "snr": -3.8,
        "rf_chain": 0
      }
    ]
  }
}

2. The second solution is to use a timestamp from the packet data from the balloon itself. Most balloon trackers have a RTC or a GPS that can tell it the real time. This can be send down in the packet to give an unambiguous timestamp. Relying on the gateway to give the correct timestamp is insufficient because I have seen rogue gateways give the wrong timestamp altogether.
   The ttnhabhub bridge already has options to parse the timestamp with the SodaqOne raw packet parser.

Custom LoRaWAN packet format to send archive position data.

On long range LoRaWAN flights, such as ICSPACE22 which travelled to Japan from the UK, TTN LoRaWAN coverage is very very poor along the way. With reference to the flight path picture below, the balloon was heard last in Europe, before disappearing for 5 days, and then reappearing in Japan where there is some coverage. Therefore, we have no idea where the balloon was in the 5 days it disappeared.

I am thinking of making a very compact packet structure that combines the current Longitude, Latitude, Altitude, Temperature, Pressure, Sats along with position data from 4 earlier points in time so that we know where the balloon was.

All this has to be done in 26 bytes, to comply with air time restrictions.

This will require me to do bit level optimisation of the packet structure, and possibly use some compression schemes.

In the ttnhabbridge, I would have to make a decoder that will decode this packet. I am planning to make a pull request with this structure if thats alright. This will be in addition to the cayenne format and sodaq one format.

Dockerize ttnhabbridge

Its quite fiddly to deploy TTNhabbridge on remote servers due to the installation requirements of Java etc. I have been using Docker to easily set things up much more recently. In a single command(docker-compose up) I can install all dependencies, including java, and start running ttnhabbridge. I think it will be a good addition to the repo.

My dockerfile looks like this at the moment:

FROM gradle:4.2.1-jdk8-alpine
WORKDIR /ttnhabbridge

# Copy over the files
ADD --chown=gradle:gradle . /ttnhabbridge

# add access rights to run stuff
USER root
RUN chown -R gradle /ttnhabbridge
USER gradle

# Get gradle to build
RUN ./gradlew assemble

# Extract the tar executable
WORKDIR /ttnhabbridge/ttnhabbridge/build/distributions
RUN tar -xvf ttnhabbridge.tar

# Copy our yaml settings file to the executable location. Ensure there is a `ttnhabbridge.yaml` config file in the root of the project.
WORKDIR /ttnhabbridge
COPY ttnhabbridge.yaml ttnhabbridge/build/distributions/ttnhabbridge/

# Enter the executable folder and run the program
WORKDIR /ttnhabbridge/ttnhabbridge/build/distributions/ttnhabbridge
CMD ./ttnhabbridge.sh

My docker-compose.yml looks like this:

version: '3.8'

services:
  web:
    build: .
    ports:
      - "5000:5000"

Things todo:

• Put in docker files into repo.
• Put in readme instructions how to deploy with docker.

Sondehub Amateur Upload

Hi!

We are setting up a new HAB database, based on the SondeHub architecture. It's still early days, but the initial map is here: https://amateur.sondehub.org/

It would be awesome if we could have telemetry from TTN flights being uploaded into this DB!

Telemetry is uploaded to the API at: https://api.v2.sondehub.org/amateur/telemetry
Information on this API is here: https://github.com/projecthorus/sondehub-infra/wiki/API-(Beta)#amateurtelemetry

Telemetry is uploaded in a common format described here: https://github.com/projecthorus/sondehub-infra/wiki/%5BDRAFT%5D-Amateur-Balloon-Telemetry-Format

Unlikely Habitat, the SondeHub DB does not accept UKHAS strings (yet), and instead accepts a JSON object with some mandatory, and some optional fields. Multiple packets are uploaded by uploading a list of objects, which can be GZIPed to reduce transfer size. An example of an uploaded packet is as follows:

[
{
  "software_name": "ttnhabbridge", # Receiving software name
  "software_version": "0.0.1", # Receiving software version
  "uploader_callsign": "foobar",                 # Mandatory - TTN station name?
  "uploader_position": [ -34.0, 138.0, 0 ],  # Optional - TTN station location, if available
  "uploader_radio": "???",  # Optional - Any other details 
  "uploader_antenna": "???", # Optional - other rx details
  "snr": 11.79, # Optional - Receiver metadata - SNR
  "frequency": 434.201003, # Optional - Receiver Metadata - RX Frequency
  "modulation": "LoRaWAN - TTNv3", # Optional, but recommended - Modulation type
  "time_received": "2022-04-18T04:36:59.899304Z", # Time the packet was received on the TTN network
  "datetime": "2022-04-18T04:36:58.000000Z", # Date/time reported by the payload itself. Use todays UTC date if no date available.
  "payload_callsign": "CALLSIGN_HERE", # Callsign of the payload
  "frame": 6, # Optional - Frame number 
  "lat": -34.1, # Mandatory - Position
  "lon": 138.1,
  "alt": 100.0,
  "temp": 30, # Some examples of optional fields 
  "sats": 0,
  "batt": 3.15,
},
{ more packets },
{ more packets} 
]

Some example Python code that uploads to the DB is here: https://github.com/projecthorus/horusdemodlib/blob/master/horusdemodlib/sondehubamateur.py#L253

Noting that each packet from a LoRaWAN payload may be received by a large number of receivers, currently the best way to handle this is to put multiple copies of the packet into the uploaded list, with different uploader information in each packet.