simpleiot / simpleiot

Simple IoT cloud/edge application/framework

Simple Iot enables you to add remote sensor data, telemetry, configuration, and device management to your project or product.

Implementing IoT systems is hard. Most projects take way longer and cost more than they should. The fundamental problem is getting data from remote locations (edge) to a place where users can access it (cloud). We also need to update data and configuration at the edge in real time from any location. Simple IoT is an attempt to solve these problems by embracing the fact that IoT systems are inherently distributed and building on simple concepts that scale.

Simple IoT provides:

• a single application with no dependencies that can be run in both cloud and edge instances
• efficient synchronization of data in both directions
• a flexible UI to view configuration and current values
• a rules engine that runs on all instances that can trigger notifications or set data
• extensive support for Modbus -- both server and client
• support for the Linux 1-wire subsystem.
• flexible graph organization of instances, users, groups, rules, and configuration
• integration with other services like InfluxDB and Twilio
• a system that is easy to extend in any language using NATS
• a number of useful Go packages to use in your custom application

See vision, architecture, and integration for addition discussion on these points.

See detailed documentation for installation, usage, and development information.

Motivation

This project was developed while building real-world IoT applications and has been driven by the following requirements:

• Data (state or configuration) can be changed anywhere — at edge devices or in the cloud and this data needs to be synchronized seamlessly between instances. Sensors, users, rules, etc. can all change data. Some edge systems have a local display where users can modify the configuration locally as well as in the cloud. Rules can also run in the cloud or on edge devices and modify data.
• Data bandwidth is limited in some IoT systems — especially those connected with Cat-M modems (< 100kb/s). Additionally, connectivity is not always reliable, and systems need to continue operating if not connected.

Core ideas

The process of developing Simple IoT has been a path of reducing what started as a fairly complex IoT system to simpler ideas. This is what we discovered along the way:

1. treat configuration and state data the same for purposes of storage and synchronization.
2. represent this data using simple types (Nodes and Points).
3. organize this data in a graph.
4. all data flows through a message bus.
5. run the same application in the cloud and at the edge.
6. automatically sync common data between instances.

Design is the beauty of turning constraints into advantages. -- Ava Raskin

These constraints have resulted in Simple IoT becoming a flexible distributed graph database optimized for IoT datasets. We'll explore these ideas more in the documentation.

Support, Community, Contributing, etc.

Pull requests are welcome -- see development for more thoughts on architecture, tooling, etc. Issues are labelled with "help wanted" and "good first issue" if you would like to contribute to this project.

For support or to discuss this project, use one of the following options:

• Documentation
• Simple IoT community forum
• #simpleiot Slack channel is available on gophers.slack.com
• open a Github issue
• Simple IoT YouTube channel
• subscribe to our email newsletter for project updates.

License

Apache Version 2.0

Contributors

Thanks to contributors:

Made with contrib.rocks.

simpleiot's Issues

Device communication protocol and packet encoding

try to mirror HTTP API

-disableAuth command line option no longer works

This option is intended for internal setups where people don't want to mess with auth. Low priority.

Find way to dump database to file for debugging

perhaps dump types we store to JSON file or something.

add support for logging via syslog

v2 implementation of bbolt database

Lately I've been db questions -- if we should stick with bolt or move the sqlite, continue using bolthold, etc.

SQL seems to still be loved by many. However, Matt Ryer used firebase in his latest project (pace.dev). As I continue to implement functionality (like Rules), I really appreciate the ability to store complex/rich data types. To do the same thing in SQL would require a large number of tables and joins which seems like it could get just as complex. Additionally, keeping the project pure Go has a lot of value and probably worth a little extra pain. From the bolt readme:

Bolt is currently used in high-load production environments serving databases as large as 1TB.

so, it seems Bolt should be more than adequate for many IoT systems -- especially because write load is not super high -- data is trickling in, but you want decent read performance to query it.

So thinking it still makes sense to continue with bolt, but other opinions are welcome.

Thinking it is time to quit using bolthold and implement directly on bbolt. Bolthold has served us well in allowing functionality to be added quickly. Some reasons to continue with bolt and implement directly on bolt:

• Once NATS is integrated, more operations will be triggered by events in real time, so there will be less reliance on complex database operations.
• we've had some problems with UUID keys and have some horrible hacks in place to work around
• forEach does not allow us to do db operations in the callbacks.
• We're also not using the bolthold query functionality much, so that does not add a lot of value
• not encoding keys as gobs should be a lot more efficient
• would like to move to protobuf anyway for encoding (more space efficient and better handles schema changes than gob)
• we're to the point where we need to start doing more indexing, and bolthold support for indexing is weak, so thinking it makes sense to just implement directly on bolt.
• having direct access to seek, next, etc allows us to break long running operations (example scanning all devices) into smaller operations that reduces the db read-write transaction lock time.

Checklist:

• create interface for DB access so we can support multiple implementations -- need to do this anyway to support mongodb and others in the future
• switch current db implementation to use the interface
• create protobuf defs for all types stored in db
• store uuids in key as int64 (LE) instead of encoding as gob -- eliminates any confusion as to what is stored and should be more efficient.
• write v2 implementation on top of bbolt
• create a migration that reads v1 database format and writes to data2.db -- this eliminates any confusion as to what is what.
• revisit indexing

Consider using Makefiles to encode build dependencies

We currently use shell functions to encode build dependencies. While this is straightforward, it has two drawbacks:

• redundant dependencies can be satisfied multiple times, repeating terminal output and increasing build time
• only a shell can run shell functions, whereas executable files can be used by many other tools, such as entr.

Makefiles often poorly fit the task at hand, such as minimizing C/C++ builds. Make's more advanced features can quickly lead to unreadable code.

In our case, Make's dependency analyzer seems apt for such tasks as "run tests, then run the executable". We have little need of functionality beyond phony targets and basic recipes, like variables, functions, file inclusion, conditionals, pattern matching, or even regular file targets. Thus, we can avoid nearly all of the pitfalls of Makefiles and arrive at a clean description of our build dependencies. It might look like this:

all: test deps siot

deploy: deps
	gcloud app deploy cmd/portal

docs:
	. ./envsetup.sh; siot_build_docs

run: deps
	go run cmd/siot/main.go -sim

siot:
	go build -o siot cmd/siot/main.go

deps: frontend assets

frontend:
	. ./envsetup.sh; siot_build_frontend

assets:
	. ./envsetup.sh; siot_build_assets

test: 
	golint -set_exit_status ./...
	go vet ./...
	go test ./...

.PHONY: all deps frontend assets test

There may be other tools that provide dependency analysis without Make's baggage.

use coap observe to push data to devices

add DCO/signed-off

This project has a nice script for setting up git hooks:

https://github.com/envoyproxy/envoy/tree/master/support/hooks

Some information on kernel DCO:

https://www.kernel.org/doc/html/latest/process/submitting-patches.html

Would also be nice to rewrite old commits.

BLE connection failures with Android Phones

with latest firmware, BLE connection works fine on MAC, but fails on two different Android phones.

Add support for influxdb 2.0

influxdata has a cloud offering that is now seems very cost effective for time series data storage.

Support BLE sensors

Update rPI client to read data from Silabs Thunderboard 2

Have had good success with https://github.com/go-ble/ble on another project.

migrate old UI code into new elm-ui based UI

new UI code is in the frontend2 directory -- old UI code needs to be migrated from frontend to frontend2, as well as the build system fixed up. This is mostly just a list of devices that can be expanded to list sample values, and edit the name of the device.

Automated backups

One of the guiding principles of SIOT is to minimize the number of moving pieces and thus operations. So it seems it would be convenient if SIOT can simply back up its own data to an external location (like S3), or provide a utility that can pull the data out of SIOT onto a company server.

add user authentication

Add support for raspberry PI device

perhaps expose IO using gobot or periph.io

Investigate Sparkplug

Sparkplug is an encoding scheme that uses protobuf for encoding, and is intended as a very full featured, yet efficient data format. Example implementations are here. The Sparkplug encoding can encode data in a number of different formats and even has the capability to encode matrix data. However, it is efficient in that it does not use space in the packet for features that are not used (a feature of protobuf). Protobuf is also nice in that the field names are not included in every packet -- they are represented as ints.

create command line options to send custom samples

This will allow any computer to function as a simple iot device by doing something like:

siot -id=ab23124 -value=23.5 -index=1 -portal=https://portal.mysiotinstance.com -apikey=28341291sasdf2234 type=temp

Will allow the following:

• makes it easy to test/evaluate SIOT
• enable sending data to a SIOT from scripts (backups, cron jobs, etc)

modem: move setting APN out of PPP scripts

this will give us the flexibility to set the APN

Add SMS notification support

Using Twilio.

add configure state to modem code

we really need to configure the modem before trying to do anything with it in cases where modem needs configured.

update to latest elm-bootstrap

https://github.com/rundis/elm-bootstrap/releases/tag/5.2.0

don't display navbar menu entries if not logged in

display uptime on Configure page

data is already exported via BLE so may as well use it

create a reliable HTTP download over cell modems

with Go http client, we find large downloads (5M) over CAT-M modems sometimes stall (probably sites with low cell signal). Would like to find a more robust way to download, and report status back to server.

add device authentication

perhaps using https://jwt.io/ for HTTP -- not sure about coap ...

Improve NATS device authentication model

would eventually be nice to have different NATS accounts for every device. This would prevent someone using credentials from a hacked device from wreaking havoc on the system.

better handle errors/closing in respreader goroutine

• handle situations when read fails with something other than io.EOF
• is there some way to detect when the reader is garbage collected and close the goroutine?

Add user authentication

@cbrake My assumptions:

1. Use SSL to prevent password exposure upon initial authentication
2. Backend will serve a JWT to the frontend, which will have to send it along with every request

Switch frontend to elm-spa v5

Currently on v4.

It looks like author is happy with the v5 API, so hopefully this will be the last major change.

evaluate elm-ui, elm-spa.dev

started a new branch to explore this:

https://github.com/simpleiot/simpleiot/tree/feature-elm-spa-dev

login/startup failure

commit faec459

[cbrake@mars go]$ siot_run 2

elm-spa created 4 files:

  /scratch/simpleiot/go/frontend2/elm-stuff/.elm-spa/Generated/Pages.elm
  /scratch/simpleiot/go/frontend2/elm-stuff/.elm-spa/Generated/Params.elm
  /scratch/simpleiot/go/frontend2/elm-stuff/.elm-spa/Generated/Route.elm
  /scratch/simpleiot/go/frontend2/elm-stuff/.elm-spa/Generated/Routes.elm

Success! Compiled 4 modules.

    Main ───> output/elm.js

2020/04/20 10:58:24 found root org: {00000000-0000-0000-0000-000000000000 root}
2020/04/20 10:58:24 Error opening db:  error checking whether database is initialized: found a root org, but not an admin: No data found for this key
exit status 255

move REST API documentation to a server

The current method is not working very well:

https://htmlpreview.github.io/?https://github.com/simpleiot/simpleiot/blob/master/docs/api.html

I think if it was on its own server, it would work a lot better.

Integrate NATS.io

May be applicable for edge->cloud as well as within edge/cloud apps. Conversation on slack below:

Cliff Brake 8:33 AM
On IoT edge device projects we've done in the past, we typically write one application in Go that has a number of subsystems (modem/network management, cloud communication, reading IO/sensors, rules engine, etc). We typically implement each subsystem as a goroutine, and then pass messages around using raw Go types in buffered channels. Most messages are sent to the main goroutine, and then forwarded to interested goroutines. Overall, this architecture has worked well, is robust and flexible. However, you have to be a little careful with this approach as Go lists are a reference to underlying data, so are not a true copy when passed on a channel. So I was wondering if it would make sense to use a NATs server inside a Go app to communicate between subsystems. Then subsystems could subscribe to messages they are interested in rather than manually routing Go types at the top level. Additionally, it seems the system could be easily be broken up into multiple apps, or expand to apps on other devices at the site -- perhaps a central gateway with some remote IO. I imagine this would also require instantiating the NATS client in each subsystem. Serializing and de-serializing each message would be some overhead, but at least then you would be sure you had a true copy and not referencing shared data.
The reason we use a single Go app is for simplicity -- an edge device is basically a simple Embedded Linux rootfs with a single go app with embedded assets. Updates are typically just deploying a single file (the go app exe).

R.I. Pienaar 8:37 AM
yeah that works well, the nats server is easy to embed in go running in a go routine, so should work fine

ability to add users, roles, and orgs

For now, I think we can just have two roles:

• admin
• user

Admins can see all users, add users, assign users to devices, etc. Users can see and modify the sites they have access to.

Mechanism to send file to device

This initially will be used for sending update files to edge devices. In the past, we've had trouble with downloads of 3MB files stalling when using the Go http client. Proposing to do something like this:

• chunk file into say 50KB chunks and send as NATs message
• create API where you can give it an io.Reader, and device ID
• system verifies the device is connected, and then sends file to device/:id/pushfile
• the device subscribes to device/:id/file and processes any files coming in
• when file has been downloaded, and assembled, it sends a message to the local NATs server running on the device -- perhaps /device/:id/newfile
• the device would have an update handler that listens for newfile, and if the file coming in is an update file, process it.

This would give us a generic mechanism for pushing files to a device, and then handlers could be attached at either end to send or process the files.

Come up with a better logo

Since this project is building blocks, maybe have something with blocks in it?

branching model

interesting series of posts:

https://about.gitlab.com/blog/2020/03/05/what-is-gitlab-flow/
https://about.gitlab.com/blog/2016/07/27/the-11-rules-of-gitlab-flow/
https://docs.gitlab.com/ee/topics/gitlab_flow.html

rename this repo to go

renaming will give us a go import path of:

github.com/simpleiot/go

instead of:

github.com/simpleiot/simpleiot

it is a little shorter and fits in better with the other repo names. This repo is essentially a monorepo of go code that can be used to build both edge and cloud IoT apps, so it seems go is a fitting name for the repo.

Let me know any thoughts, or if this will greatly inconvenience anyone? Now is better than 2 years from now to make this change.

Enforce unique email addresses per user upon user creation

Web Bluetooth can't receive large payloads

It would be convenient to receive JSON data structures over web bluetooth, but on two different devices I am experiencing issues:

• Mot e4 Android phone (BT 4.2) -- seems to only be able to transfer around 20 bytes. With BlueFruit test app on same device, I can view > 100 bytes.
• Macbook Pro 2012 (BT 4.0) -- can transfer around 100 bytes in Chrome web BT app, but packet fails when larger. See the same thing with a BLE test app on this device, so likely a limitation of the radio.

Investigate SenML

Another standard for encoding sensor/IoT data. This standard seems a little slimmer than Sparkplug.

Error getting users for group 9b66268d-9046-4bbb-8205-b5b24a353e88: No data found for this key

I think we are neglecting to delete a user from Group:Users array when deleting a user.

Display some notification that we are disconnected from backend

modem: listen to URC codes

URC (Unsolicited result codes) are messages sent from the modem to provide information about an event. They may provide helpful diagnostic information about problems, etc.

handle large numbers of devices

One issue with SPA apps is how to handle large amounts of data as you can't download it all into the frontend at once.

This issue is to collect ideas on how this might be implemented in the future:

https://guide.meteor.com/data-loading.html#pagination
https://discourse.elm-lang.org/t/strategies-for-fetching-only-the-required-data-for-a-page-in-an-spa/1073/6

show some type of notification if login failure

correctly handle BLE concurrency issues

Occasionally save operations do not work, and have to be repeated.

Lets encrypt integration for NATS TLS certs

Right how we are just bundling self signed certs with the app to get encryption. Long term it would be nice to support certs from a service like letsencrypt, but if we do that, we want the renewal to happen automatically.

several possibilities:

• use something like:
  • https://github.com/caddyserver/certmagic
  • https://github.com/go-acme/lego
• use certs generated by Caddy as typically we will have caddy running on this server anyway

https://docs.nats.io/nats-server/configuration/securing_nats/tls

UI for restoring data

This follows the SIOT principle that some down time is fine as long as we have an easy way to rebuild the system.

choose license

Candidates:

• https://www.apache.org/licenses/LICENSE-2.0.html
• https://spdx.org/licenses/0BSD.html

Caddy uses apache-2