Integrate NATS.io about simpleiot

[ColinSullivan1]

If you'll take some unsolicited advice, here are a few notes to help you get started.... :)

Embedding the NATS server is straightforward, some example code can be found here.

After embedding the NATS server, enabling it as a leaf node and connecting into a central NATS deployment or NGS may be of interest to you. More on that here: https://docs.nats.io/nats-server/configuration/leafnodes

Let us know how it goes, happy to answer any questions as you go.

[cbrake]

@ColinSullivan1 Advice is very welcome -- thanks! That is exactly the architecture I had in mind, so neat the NATS already has the leaf node feature that facilitates this.

Thinking a little more on the cloud side, responding to real time changes in the portal web application is a problem that needs solved. The current access model is devices and users both belong to groups, so user access to devices requires that users be in the same group as the device. As sensor data comes in, we want to update the view (web app) into the portal in real time for all users connected. It seems a subject for device changes could be made for every group, then web clients simply subscribe to the group changes they are a member of (perhaps over websockets, or SSE). I assume if there are no subscribers then a published message simply goes nowhere. I really like the idea of routing real time data at the application level rather doing something like using a database technology where you can watch changes (mongodb db.collection.watch() for example). It seems backwards to put it in the database, then get a notification that something changed and have to read it back out.

This will not scale to 10,000 devices, so perhaps another approach is to keep track of which devices the user is viewing at the time (pagination), and have the websocket code subscribe to changes for those devices.

[cbrake]

reference discussion: nats-io/nats-server#758

[cbrake]

binary size before embedded nats server and client: 13540452
after: 17675078
difference: 4134626

[cbrake]

view of a pub message:

overhead to publish once connected appears to be about 80 + 88 = 168 bytes.

Trace of PING/PONG:

A PING/PONG appears to take 94 + 94 + 88 = 276 bytes.

[cbrake]

1st pass at embedding server and client went well: a85443b -- server embeds nicely -- I like the simplicity.

I was originally thinking of using coap because it is more efficient, but then I need to do auth, retries, coap observe (to push from portal to GW), etc. Perhaps we'll still implement coap for microcontroller targets and super bandwidth constrained cellular connections, but leveraging NATS for Linux edge devices seems to make a lot of sense. With IoT cellular costs coming down, may not be an issue.

[cbrake]

recent changes:

• switch to protbuf encoding for sending samples over nats
• switch to async subscription on server
• store samples sent over nats in DB.

Thinking more about architecture -- I like the idea of the edge and cloud app being the same app, but the edge device is leaf node of the cloud instance. Simply set an "upstream" when running the edge instance that points to the cloud, and then all data received by the edge instance would be forwarded to the upstream. For this to work, each SIOT instance will need an ID, and then it will display as "Self" in the UI of the instance. The "Self" instance would contain samples of any sensors sent to it, as well as some monitoring metrics about about itself. If additional devices connect to the edge instance, they would be forwarded up to the cloud as additional devices.

Could also have "peer" instances, where they mirror each other's data -- not sure what practical problem this solves yet :-)

Anyway, need to get some NATS stuff implemented on a customer project with a full custom edge app and will then return to this.

[cbrake]

ideas on mechanism to send update files to device: #65

@ColinSullivan1 if interested, let me know if this sounds like good/bad/horrible idea :-)

[cbrake]

trace of publish using protobuf payload:

publish packet is 151 bytes vs 177 bytes for JSON. Sending multiple samples per publish would improve efficiency some.

[cbrake]

Ping pong may actually be 4 packets:

284 bytes total. assuming we execute ping/pong every 15m, that is 27,264 bytes/day, or ~800KB/mo -- that seems reasonable to keep a NATS connection live if you have a 20-100MB/mo budget.

however, not sure if every 15m will be enough to keep modems/networking infrastructure from dropping the connection -- experimentation needed.

[ColinSullivan1]

ideas on mechanism to send update files to device: #65

@ColinSullivan1 if interested, let me know if this sounds like good/bad/horrible idea :-)

Sounds like a good idea to me. I also left a note about using JetStream to hold configuration.

[cbrake]

Some numbers on package sizes. With xz compression, the siot binary is 5.9MB, and after adding nats server, 8.5MB.

[cbrake]

testing NATS over Cat-M modem connection -- mostly working. One thing I've done is set the ping/pong interval pretty long to not use bandwidth:

nc, err := nats.Connect(server,
		nats.Timeout(30*time.Second),
		nats.DrainTimeout(30*time.Second),
		nats.PingInterval(2*time.Minute),
		nats.MaxPingsOutstanding(5),
		nats.RetryOnFailedConnect(true),
		nats.ReconnectBufSize(5*1024*1024),
		nats.MaxReconnects(-1),
		nats.SetCustomDialer(&net.Dialer{
			KeepAlive: -1,
		}),
		nats.CustomReconnectDelay(func(attempts int) time.Duration {
			delay := ExpBackoff(attempts, 10*time.Minute)
			log.Printf("NATS reconnect attempts: %v, delay: %v", attempts, delay)
			return delay
		}),
		nats.Token(authToken),
	)

so if the Cat-M modem is not connected when NATS starts, it seems subscriptions don't work until the first ping/pong fails, and the NATS client re-connects. Is there any way to force a client re-connection -- say if I know the CAT-M modem just connected?

Ideally the *nats.Conn can live for the lifetime of my app (days/weeks/months) so I don't have to re-setup subscriptions in the device app. Latency is pretty good when sending messages to the device -- often a fraction of a second and occasionally up to 1s or so -- makes for a nice user experience. Will let it sit overnight and see how the connection holds up.

simulated connection problems by disconnecting the antenna from modem -- for shorter durations, where the modem manager only restarts PPP, NATS seemed to maintain connection. For longer durations where the manager resets the modem, NATS lost the connection (I could not publish or get subscription messages), and it took the client 10m to reconnect (as configured -- 5*2m). The good news is after 10m, it did indeed re-connect, and everything started working again -- NATS client handled all that in the background.

[cbrake]

With TLS enabled, a PING/PONG is little heavier:

116 + 88 + 116 + 88 = 408 bytes vs 284 bytes without TLS.

So without TLS, PING/PONG overhead is:

every 2m: ~6MB/mo
every 5m: ~2.4MB/mo
every 15m: ~800K/mo

With TLS, PING/PONG overhead is:

every 2m: ~8.8MB/mo
every 5m: ~3.5MB/mo
every 15m: ~1.2MB/mo

I think I recall hearing that if there is traffic, during the PING/PONG interval, then it is not used, but I have not verified this -- so perhaps if there is other traffic, then there won't be as much overhead to PING/PONG.

[cbrake]

initial proof of concept is a success! feature-nats has been merged to master.