nebraltd / hm-gatewayrs

Light Hotspot (gateway-rs) Container for Helium

hm-gatewayrs: Helium gateway-rs container

This is containerized version of helium gateway-rs used by nebra hotspots.

We take the gateway-rs deb package created by Helium from their Github repo (built from the GitHub source)

Environment variables

REGION_OVERRIDE and I2C_DEVICE are used to load the correct settings for the gateway. All settings override will continue to work as documented by upstream gateway-rs.

PREVENT_START_GATEWAYRS can be set to 1 to prevent the miner from starting. This is helpful for debugging the miner manually.

GW_KEYPAIR and GW_ONBOARDING - testing use only

Whilst these environment variables can be used to override the swarm key and onboarding key respectively, it is usually best to use the SWARM_KEY_URI_OVERRIDE and ONBOARDING_KEY_URI_OVERRIDE variables that are parsed in hm-pyhelper instead as this also get picked up in diagnostics and other containers.

Mr Bump

Mr Bump is a GitHub bot we created to automate some tasks related to the miner software. This includes updating the miner to the latest GA (and tagging / releasing this update) as well as updating the necessary docker-compose.yml files.

Mr Bump is currently used in the following repos:

• hm-miner
• hm-pyhelper
• hm-gatewayrs
• helium-miner-software
• light-hotspot-software
• helium-5g-software

Pre built containers

This repo automatically builds docker containers and uploads them to two repositories for easy access:

• hm-gatewayrs on DockerHub
• hm-gatewayrs on GitHub Packages

The images are tagged using the docker long and short commit SHAs for that release. The current version deployed to miners can be found in the helium-miner-software repo.

hm-gatewayrs's Issues

Add linting / flake8

Add super linter.

Acceptance Criterion

• flake8 runs as github workflow

Complete image creation pipeline for manufacturing

Acceptance criteria

• Gateway container should use the same base image as hm-config and hm-diag.
• Output of hotspot-production-images starts and shows up in Balena.

Gateway_key.bin location

Even when the configuration directory is changed the gateway_key.bin is located in /etc/helium_gateway:
this is creating the getway_key.bin in the /etc/helium_gateway directory instead of /opt/helium_gateway

/usr/bin/helium_gateway -c /opt/helium_gateway key info

As a consequence, the following command terminating with an error

/usr/bin/helium_gateway -c /opt/helium_gateway add --owner xxx --payer xxx
Error: Decode(Json(Error("EOF while parsing a value", line: 1, column: 0)))

This is solved by creating a symbolic link

ln -s /etc/helium_gateway/gateway_key.bin /opt/helium_gateway/

but I think a better way to fix this is to create the key file in the given directory.

refactor to use parameters from hm-pyhelper

Currently have hard coded i2c bus for ECC whereas we should pull this from pyhelper

Also as this has not had many sets of eyes on it the whole repo could do with an audit and probably a refactor

Start using light miner

As a user, one should be able to get blockchain specific diagnostic information after miner swapping.

Acceptance Criterion

• Following parameters work with the new miner:

• blockchain height
• region

Start script should be robust to errors

The startup script is currently brittle because it is written in bash. Rewriting with python will make it more robust to failures.

Original notes:
convert the startup script to python and use the helper functions in hm-pyhelper for the functions there
There isn't much to do here.

• get and check ecc bus
• start helium_gateway binary

Acceptance criteria

• Script is converted into a python module that can be executed with python MODULE_NAME.
• Test directory not included in module src
• File deletion handled in Dockerfile instead (eg don't copy in the first place)
• i2dection/ecc uses a pyhelper method
• Errors sent to sentry
• Use Tenacity library for retry logic
• Script is successful if gateway starts as a result
• Update README accordingly
• Log failures and exit details when helium_gateway terminates
• Python src linter added to workflow
• remove 'v' from the version string. dpkg -s doesn't show that.

Split build and run stages in dockerfile

Split build

need ability to pull frequency plan from miner / gateway-rs

currently, we have an environment variable set for REGION_OVERRIDE which determines the frequency plan and by extension the region in the settings.toml file.

At some point, it should be possible to get the frequency plan from the asserted location of the miner from the gateway-rs application.

Once this is possible, we need to implement this functionality here too

Relates to helium/gateway-rs#43

need ability to grab public keys / ECC support

In the full miners we have this script which pulls in the public_keys of the miner to a file var/data/public_keys

PUBLIC_KEYS=$(/opt/miner/bin/miner print_keys)
[ $? -ne 0 ] && exit 1
echo $PUBLIC_KEYS > /var/data/public_keys

We need this implementing in hm-gatewayrs so that hm-config and hm-diag can have access to the keys for proper onboarding.

This will need implementing in https://github.com/helium/gateway-rs first - which I do not believe it has been yet.

spike: investigate gateway-rs build to use.

We are using rasp01.deb from gateway-rs releases. This will work only on raspberry pi boards but not on rock pi. On Rockpi boards, the binary just hangs up.

if we are using pi zero 2 W for light miners. I think, we can just pick up rasp234.deb build which should work on all our current arm boards and close this issue.

Acceptance criteria

• List of gateway-rs build to use for ROCK PI miner, radxa zero, and rpi light miner, rpi indoor miner.
• Point helium-miner-software update task to correct image
• List our products in upstream repo

devices with very old region parameters (in format region_868) cause miner to get in crash loop and eventually crap out

Miner - https://admin.cloud.nebra.com/#/device/29607/show
Dash - https://dashboard.nebra.com/admin/device/devices/3a431d434f1cc112/
Explorer - https://app.hotspotty.net/hotspots/112h4zwWHL36zEcczphpQaN8YRdSBSG2mDaCzuekxxy2qoAoDRyD/rewards
Discord - https://discord.com/channels/404106811252408320/404106811252408322/1116519788294979684

Weird issue, but at some point in the past helium-miner gave regions as format region_eu868 instead of just EU868. This has now been reverted, however, if a miner has been turned off for a long time and just turned on in last few months, since the solana update they will not work as the gateway-rs tries to start with this older region format and fails like the below output. It should then try to get the region from the region service but this never works and the container just gets stuck in a restart loop and eventually just gives up.

root@nebra-72E75B:~# balena ps -a
CONTAINER ID   IMAGE                                                            COMMAND                  CREATED       STATUS                  PORTS                               NAMES
9cc17f8b3ded   86879d57ab37                                                     "./start_pktfwd.sh"      8 days ago    Up 7 days                                                   packet-forwarder_32632_5140_8b4f2188bcbf785f63850c5f7e8ba573
808be0a3778b   c0874c0bc901                                                     "/root/start_multipl…"   8 days ago    Up 7 days               1681/tcp                            multiplexer_32636_5140_8b4f2188bcbf785f63850c5f7e8ba573
bd35d7d5cb0b   fa01a3f2bfa3                                                     "/opt/start-gateway-…"   8 days ago    Up 7 days                                                   gateway-config_32635_5140_8b4f2188bcbf785f63850c5f7e8ba573
1ca0f44cd02e   73e114e4dabb                                                     "/opt/nebra-gatewayr…"   8 days ago    Exited (1) 6 days ago                                       helium-miner_32633_5140_8b4f2188bcbf785f63850c5f7e8ba573
fe1b1c395ddd   2c8ab0f089b6                                                     "/opt/start_diagnost…"   8 days ago    Up 7 days               0.0.0.0:80->80/tcp, :::80->80/tcp   diagnostics_32634_5140_8b4f2188bcbf785f63850c5f7e8ba573
236ed9ac3de1   5ce1dc1fad6f                                                     "sh ./entry.sh"          8 days ago    Up 7 days                                                   dbus-session_32637_5140_8b4f2188bcbf785f63850c5f7e8ba573
a2bc3756581a   registry2.balena-cloud.com/v2/6e1bcf249fcc2f8202341d4641c13acf   "/usr/src/app/entry.…"   2 years ago   Up 7 days (healthy)                                         balena_supervisor
root@nebra-72E75B:~# balena logs 1ca0f44cd02e
root@nebra-72E75B:~# balena restart 1ca0f44cd02e
1ca0f44cd02e
root@nebra-72E75B:~# balena logs 1ca0f44cd02e -f
Keypair has been found. (GW_KEYPAIR = ecc://i2c-7:96?slot=0)
Onboarding key has been found. (GW_ONBOARDING = ecc://i2c-7:96?slot=0)
Starting gateway-rs...
crond[14]: crond (busybox 1.35.0) started, log level 8
Error: Config(unsupported region: region_eu868)
Keypair has been found. (GW_KEYPAIR = ecc://i2c-7:96?slot=0)
Onboarding key has been found. (GW_ONBOARDING = ecc://i2c-7:96?slot=0)
crond[13]: crond (busybox 1.35.0) started, log level 8
Starting gateway-rs...
Error: Config(unsupported region: region_eu868)
Keypair has been found. (GW_KEYPAIR = ecc://i2c-7:96?slot=0)
Onboarding key has been found. (GW_ONBOARDING = ecc://i2c-7:96?slot=0)
Starting gateway-rs...
crond[14]: crond (busybox 1.35.0) started, log level 8
Error: Config(unsupported region: region_eu868)

Should still work with DIY gateways

This container should still work for DIY gateways.

So we need to maintain ability to have file based key in persistent storage for that

This was removed in #25 / #27 but needs to be added back

Implement standardized build pipeline based on existing repos

As a user of this repository, any new push should result in workflow triggers that do linting and dockerfile builds.

• cleanup the dockerfile build and startup script.
• verify that gateway miner runs on the device with ecc.
• update workflow trigger similar to hm-miner

Acceptance Criterion

• a new release of upstream gateway-rs results in auto bumping the dockerfile and corresponding dispatch to helium-miner-software
• Linting workflow actions for dockerfile, code, spellcheck and security. Copy linting settings from hm-confing and hotspot-production-tool.
• docker builds on every push.
• Add CODEOWNERS and CONTRIBUTING. Copy from existing repos.
  • Device team should be code owners.

Add auto bumper to create PRs in light hotspot software repo

As per the full miner software, add repository dispatch action to auto add a pr to nation light hotspot software repo on update of this container

Audit and improve documentation / README

Improve docs

helium_gateway info region outputs settings.toml or environment parameter which causes issues when freq/region changes

In this file https://github.com/NebraLtd/hm-gatewayrs/blob/main/gen-region.sh we use helium_gateway info region which outputs the region from the region config oracle service.

However, we have inadvertently misunderstood the functionality of this command as being the same as it was in pre-gateway-rs helium miner. This was not a problem with the old hm-miner container, as the region check command on the old miner only ever returned the asserted region plan, and never something configured in a settings file. Hence why this only started happening after the move to hm-gatewayrs container.

If there is a region set in the settings.toml file or using a GW_REGION environment variable this command will output the contents of the file/envvar until it gets the region directly from the config service. As we are running this check before we run the helium_gateway server at startup (and only once), this means that the region file never gets correctly updated with the correct frequency.

This is a problem in certain circumstances:

• in situations like the change of frequency plan in Australia from AS923_1 to AU915 the gateway-rs will be running using the correct frequency plan, however the packet-forwarder will be using an incorrect config.
• if a customer from one region plan area sells their miner to someone from another region plan area, it will not work correctly until a factory reset is performed.

The only way to fix this currently is a factory reset (it will never resolve itself otherwise).

The way to solve this is to periodicially run the region-checker script (that populates the region file at /var/pktfwd/region) and if an updated one is found then restart the packet-forwarder container.

Connection refused

When running start-gatewayrs.sh

  if ! PUBLIC_KEYS=$(/usr/bin/helium_gateway -c /etc/helium_gateway key info)

I get

27.01.22 23:37:15 (+0100)  helium-miner  Error: Service(Service(tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })))))

What could be the issue?

start_gatewayrs.sh needs updating

helium/gateway-rs#119

A recent change in gateway-rs changed the settings name for listen_addr to simply listen. This was done because underscores in field names for settings conflicts with the usage of _ as a nesting seperator.

It is worth mentioning that you could also use the environmental variables to override settings.toml instead of constructing it the way you currently do. For example:

export GW_LISTEN=${LISTEN_ADDR}
export GW_REGION=${REGION_OVERRIDE}
/usr/bin/helium_gateway -c /etc/helium_gateway server

Remove pinned packages in dockerfile

Remove pinning to match other repos

Add sentry

Enable sentry

Add instructions for onboarding of DIY gateways

Need to add instructions for onboarding DIY gateways