msgflo / msgflo-arduino

Arduino library for easy Msgflo support on MQTT

MsgFlo - Flow-Based Programming with Message Queues

Implementation of the Flow-Based Programming using message queues as the communications layer between different processes. Initial message queue transports targeted are AMQP and MQTT.

MsgFlo lets you build robust polyglot FBP systems spanning multiple computers/devices. A node can be implemented in any language, to reuse existing code, libraries and developer know-how.

In FBP each component is a black-box that processes and produces data, without knowledge about where the input data comes from, or where the output data goes. This ensures that a service is easy to change, and facilitates automated testing.

MsgFlo is designed to enable partial and gradual integration into existing systems; by using standard broker/transports, not placing restrictions on message payloads, allowing to use existing queue names, and integrating non-MsgFlo nodes seamlessly.

Status

In Production

• Used in production at TheGrid website builder, with AMQP/RabbitMQ. 20 roles, 1'000'000 jobs/weekly+
• Used in production in imgflo image processing server. 4 roles, 200'000 jobs/weekly+
• Used for IoT networks at hackerspaces c-base and Bitraf, using MQTT/Mosquitto.

Client support

• msgflo-nodejs makes it easy to make Node.js participants in JavaScript/CoffeeScript
• noflo-runtime-msgflo makes it super easy to use NoFlo in the participants
• Basic support for C++ participants with msgflo-cpp and MicroFlo
• Basic support for Python participants with msgflo-python
• Basic support for browser participants with msgflo-browser
• Basic support for Arduino participants with msgflo-arduino
• Experimental support for Rust participants with msgflo-rust

Tooling

• msgflo executable implements the FBP runtime protocol.
• Initial support for automated testing using fbp-spec
• Experimental support for visually building networks using Flowhub
• guv provides autoscaling of workers when using Heroku/AMQP.

Licence

MIT, see ./LICENSE

Documentation

Please refer to https://msgflo.org

Support

MsgFlo is a part of Flowhub, a platform for building robust IoT systems and web services.
We offer an Integrated Development Environment and consulting services.

Debugging

The msgflo executable, as well as the transport/participant library uses the debug NPM module. You can enable (all) logging using:

export DEBUG=msgflo*

msgflo-arduino's Issues

Register in Arduino IDE library manager

From Arduino docs:

If you want to add your own library open a new issue on github.

https://github.com/arduino/Arduino/issues

Automated tests

Should test that discovery messages is emitted correctly, and that send/receive works as expected.

Tricky because of all the Arduino dependencies, and possibly device-specific code.
But it looks like the PubsubClient which does the MQTT has a testing setup running on Linux, which might be a starting point. https://github.com/knolleary/pubsubclient/tree/master/tests

Support/test/document ESP32 use

Theoretically they should just-work with the Arduino core for ESP32, but practice is often different :)

Invalid discovery message when component has multiple in/outports

"outports": [{
			"id": "sensor1",
			"type": "boolean",
			"queue": "public/bitraf/windowsensor/workshop/sensor1"
		} {
			"id": "sensor2",
			"type": "boolean",
			"queue": "public/bitraf/windowsensor/workshop/sensor2"
		} {
			"id": "sensor3",
			"type": "boolean",
			"queue": "public/bitraf/windowsensor/workshop/sensor3"
		}],

(missing comma in between)

Support multiple participants

A participant should be able to model a particular "service", as as group of functionally-related ports. Independent pieces should be independent participants. A device should be able to "host" many of these.

Some preparation has been done in the code for this, though the Participant class - but Engine implementation needs to support more than one.

Allow usage without MQTT username/password

Example: how to add icon for component?

Currently not demonstrated in example

PubSubClient: Discovery fails due to 128 byte buffer limit