thin-edge / thin-edge.io

The open edge framework for lightweight IoT devices

The open edge framework for lightweight IoT devices

thin-edge.io

Table of Contents

1. About The Project
2. Reference Systems
3. Roadmap
4. Contributing
5. License
6. Contact

Jump right in:

• Explore the Docs
• Install thin-edge.io on your device
• Getting started with thin-edge.io on a Raspberry Pi
• Connect your device to Cumulocity
• Connect your device to Azure
• Connect your device to AWS
• Find some projects that work together with thin-edge.io

About The Project

thin-edge.io is the first open-source and cloud-agnostic edge framework designed for resource-constrained IoT edge devices.

We have started the project to solve the challenge of IoT device enablement in industrial IoT or more precisely to make the integration of IoT devices with cloud and IoT platforms easy and reliable.

With thin-edge.io we want to provide re-usable and modular components, which are not bound to a specific IoT platform, domain model or vendor. It runs on a wide variety of hardware, from small, embedded Linux (Debian, Yocto, etc) devices with very low RAM footprint to large, multi-core industrial servers (IPC). It comes with multi-language support and aims to provide out-of-the box connectivity, security and device management features on any device its deployed on.

Reference Systems

For an overview of the supported platforms please have a look at the documentation.

Roadmap

If you want to propose features to the maintainers or want to contribute a development of yours, it´s always a good idea to get started with a conversation based on an issue.

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. How you can contribute to thin-edge.io you can find in the Contribution Guideline

Any contributions you make are greatly appreciated.

License

Distributed under the Apache 2.0 License. See LICENSE for more information.

Contact

Discord - Join the thin-edge.io community now!

LinkedIn - @thin_edge_io

Email - [email protected]

Website - thin-edge.io

thin-edge.io's Issues

The links under 002_installation.md to register and connect are broken

Is your documentation request related to a problem? Please describe.
A clear and concise description of what the problem is. E.g. I'm always frustrated when [...]

Describe the improvement you'd like
A clear and concise description of what you want to happen.

Additional context
Add any other context or screenshots about the documenatation request here.

tedge.toml does not include device-id under [device]

Is your feature request related to a problem? Please describe.
Currently the device id is not stored within the tedge.toml which might be needed by other plugins or modules.
Describe the solution you'd like
Write the device id into the toml
Describe alternatives you've considered
I can get the device id also from tedge config list but there is no reason why only the device id is missing on the toml.

ARMv6 support

Describe the bug
A cannot create a tedge certificate on raspberry pi

pi@raspberrypi:~ $ tedge cert create -id thin_edge_kai_home
Speicherzugriffsfehler
pi@raspberrypi:~ $ sudo tedge cert create -id thin_edge_kai_home
Speicherzugriffsfehler
pi@raspberrypi:~ $ sudo tedge cert create --deivece-id thin_edge_kai_home
Speicherzugriffsfehler
pi@raspberrypi:~ $ tedge cert create --deivece-id thin_edge_kai_home
Speicherzugriffsfehler

To Reproduce

from bash_history
sudo apt install mosquitto
curl -LJO https://github.com/thin-edge/thin-edge.io/releases/download/0.1.0/tedge_0.1.0_armhf.deb
curl -LJO https://github.com/thin-edge/thin-edge.io/releases/download/0.1.0/tedge_mapper_0.1.0_armhf.deb
dpkg -i tedge_0.1.0_armhf.deb
sudo dpkg -i tedge_0.1.0_armhf.deb
sudo dpkg -i tedge_mapper_0.1.0_armhf.deb
sudo adduser pi tedge-users
sudo tedge cert create --device-id thin_edge_kai_home
sudo tedge cert create --device-id thin_edge_kai
sudo tedge cert create --device-id alpha
sudo tedge cert create
tedge cert create --device-id thin_edge_kai_home
tedge cert create -id thin_edge_kai_home
sudo tedge cert create -id thin_edge_kai_home
sudo tedge cert create --device-id thin_edge_kai
sudo reboot

Expected behavior
the cert is created

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian

Raspberry Pi 2

• thin-edge.io version [e.g. 0.1.0]

Additional context
Add any other context about the problem here.

c8y bridge misses "s/us/#" topic

Describe the bug
The configuration for the mosquitto bridge to c8y (/etc/tedge/mosquitto-conf/c8y-bridge.conf) does not bridge the s/us/# topic.

To Reproduce
Execute tedge mqtt pub 'c8y/s/us/test123' '211,25'.

Expected behavior
The above command should create a new temperature measurement for a child device with the external id test123. If the device does not already exist it will be created.

Screenshots

Environment (please complete the following information):

• Raspbian GNU/Linux 10 (buster)
• Raspberry Pi 4
• Linux raspberrypi 5.10.17-v7l+ #1414 SMP Fri Apr 30 13:20:47 BST 2021 armv7l GNU/Linux
• thin-edge.io version 0.2.0

Additional context
This can be easily fixed by adding topic s/us/# out 2 c8y/ "" at the bottom of the /etc/tedge/mosquitto-conf/c8y-bridge.conf file and restarting mosquitto.

Functionalities for Softwaremanegement of Docker Containers missing

Is your feature request related to a problem? Please describe.
Currently with 0.4 only the initial run or the deletion of an docker container is triggered. However there is more to the orchestration and management of containers. Currently no environmentals, port or volume mappings are provided. Same is true for the actual status of the container. That way the central distribution and management of containers can hardly be used in production.

Describe the solution you'd like
Parameters for the container such as -p, -v or -e could be added as additional fragment to the operation. However this would need an extension of the current ui. Same is true for the actual status of the container. Maybe collectd is capable of monitoring containers as well, however somehow the status should be displayed within the device management. Additionaly it should be possible to start, stop and restart containers.

As a device owner I want to push measurements to associated child-devices

As a thin-edge device owner I want to be able to push collected measurement data to the cloud to appear under associated child-devices.

Therefore thin-edge public measurement API (MQTT topic "tedge/measurements") shall be extended to support pushing data to child devices, as defined in specification and prototype of ticket [CIT-578|https://cumulocity.atlassian.net/browse/CIT-578].

Spec: [https://github.com/thin-edge/thin-edge.io-specs/pull/16/files?short_path=6368cb6#diff-6368cb6131e16ed812f4565433511f2cccb942a29ee31746da1d19700637e2ce|https://github.com/thin-edge/thin-edge.io-specs/pull/16/files?short_path=6368cb6#diff-6368cb6131e16ed812f4565433511f2cccb942a29ee31746da1d19700637e2ce]

Scope of that ticket is realization of child-device support for C8Y. Other clouds (e.g. Azure) not yet in scope.

Think about adding a version field to tedge config file.

This will simplify possible future incompatible config file updates.

Improve user experience for first-time thin-edge.io users

First time users, especially coming from thin-edge.io, do not have a nice and complete user experience.

I would expect that

• thin-edge.io links to a "Quick start guide" (?) under "Getting Started"
• The Quickstart guide demonstrates how to install and use most of the functionality of thin-edge. That includes measurement sending, device monitoring and device OTA software mgmt.
• We somehow would need to deal with C8Y and Azure differences. I would propose to have two quickstart guides, one for C8Y and one for Azure. Idea: Maybe another tuturial "Quickstart without cloud"

Problems with the approach today is:

• The script get-thin-edge_io.sh does not ensure that all components are installed.
• https://github.com/thin-edge/thin-edge.io/blob/main/docs/src/howto-guides/002_installation.md does not describe the installation of all components, at least not explicitly.
• The current tutorial "entry point" (https://github.com/thin-edge/thin-edge.io/blob/main/docs/src/tutorials/connect-c8y.md ) does not link to additional tutorials like monitoring and OTA software management.
• In https://github.com/thin-edge/thin-edge.io/blob/main/docs/src/tutorials/device-monitoring.md additional installation steps are needed. This makes the complete experience pretty bad. Why is that not done in the curl shell script?

Certificate can't be uploaded

Describe the bug
I want to connect based on this tutorial: https://github.com/thin-edge/thin-edge.io/blob/main/docs/src/tutorials/connect-c8y.md to a trial instance of cumulocity.
Certificate cant be uploaded / added due to 'Access denied / 404' error

To Reproduce
tried:
sudo tedge cert upload c8y --user xxx
Enter password:
Error: failed to upload root certificate

Caused by:
HTTP status client error (401 Unauthorized) for url (https://xyz.eu-latest.cumulocity.com/tenant/currentTenant)

Expected behavior
certificate added ...

Environment (please complete the following information):

• Raspberry Pi 4
• thin-edge.io version [ 0.2.0]

`tedge config set` fails on FreeBSD (Cross-device link)

Writing the config file does not work!

The problem is that NamedTempFile creates a file in /tmp and then tries to hard link that file into the final position ($HOME/.tedge/tedge.toml). This is not going to work in cases where /tmp and /home are on different file systems which generally is the case as you mount /tmp as tmpfs.

We cannot not use NamedTempFile in these cases. Instead, we would better create a $HOME/.tedge/tedge.toml.tmp.

Support OpenRC/procd/... in addition to systemd

• Not every Linux runs systemd, e.g. Alpine, OpenWRT, Gentoo. Also, any kind of BSD does not run systemd.

• There is no good reason why we want to only support systemd for starting / stopping services.

• We already have some generic code in that regards. Extending that to support other rc systems would be pretty easy.

What would I need to implement if I want to add another endpoint ?

I want to connect to another endpoint (generic MQTT Broker, IBM IoT Platform :-)).
What do I need to implement / change / add ?

Thanks
Joerg

Mapping of events and alarms

Is your feature request related to a problem? Please describe.
Currently only measurements are mapped via tedge/measurements. Gateways commonly send alarms or event information such that a mapping to e.g. tedge/events or tedge/alarms would be necessary.
Describe the solution you'd like
An event and alarm mapper on e.g. tedge/alarms and tedge/events that can be used to send alarms and events to Cumulocity. Since the measurement mapping also relays on json based payloads this can be adapted to events as well. For alarms this somehow needs to be extended for the lifecycle of alarms e.g. tedge/alarms/Alarm-ID or similar.

Describe alternatives you've considered
Currently thin-edge.io allows sending of alarms and events

Exalate test

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior. The more detail you add here, the better we can reproduce the bug.

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS [incl. version]
• Hardware [incl. revision]
• System-Architecture [e.g. result of "uname -a"]
• thin-edge.io version [e.g. 0.1.0]

Additional context
Add any other context about the problem here.

tedge config set mqtt.BROKERURL not provided

Is your feature request related to a problem? Please describe.
We tried an multi container setup where MQTT Broker and sedge-mapper are running in two different containers. Within the mapper the MQTT Broker address can not be configured.

Describe the solution you'd like
Use mqtt.url= as a new key value pair for config.

Cumulocity IoT child device support for thin-edge

Real-world device hierarchies often look the way, that the device thin-edge is running on acts as a communication gateway for multiple sub devices. These sub devices are typically assets I want to monitor and manage separately within Cumulocity IoT.

As currently there is a separation on a semantic level needed, it would be cool to implement Cumulocity´s sub device concept for thin-edge.

ThinEdge requires glibc 2.29, severely limiting OS compatibility

I'm playing around with thin-edge, installing it to a debian 10 VM using the script from github

curl -fsSL https://raw.githubusercontent.com/thin-edge/thin-edge.io/main/get-thin-edge_io.sh | sudo sh -s 0.2.0

It installs fine and I'm able to generate a cert and connect to c8y. However, the mapper fails to start. In the journalctl logs I see:

Jun 14 09:52:10 debian tedge_mapper[1464]: /usr/bin/tedge_mapper: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /usr/bin/tedge_mapper)
Jun 14 09:52:10 debian systemd[1]: tedge-mapper-c8y.service: Main process exited, code=exited, status=1/FAILURE

This implies ThinEdge is incompatible with many fairly recent Linux versions, such as the latest Debian release. Upgrading glibc is risky (see for example https://red.ht/3d6nWy0). This could be a show-stopper for many potential adopters that have devices out there in the field running incompatible OSes.

0: File Error. Check permissions for /etc/tedge/mosquitto-conf/tedge-mosquitto.conf

I'm trying to run thin-edge.io on a custom board following the instuction of , but get an error on connect comand.

$ tedge connect c8y
Checking if systemd is available.

Checking if configuration for requested bridge already exists.

Validating the bridge certificates.

Saving configuration for requested bridge.

Error: failed to create bridge to connect Cumulocity cloud.

Caused by:
    0: File Error. Check permissions for /etc/tedge/mosquitto-conf/tedge-mosquitto.conf.
    1: failed to persist temporary file: Invalid cross-device link (os error 18)
    2: Invalid cross-device link (os error 18)

I run tedge and mosquitto as root and also my directories ar owned by root:

$ ll /etc/tedge
drwxrwxrwx    5 root     root           440 Oct 13 06:55 .
drwxr-xr-x   16 root     root          3464 Oct 13 05:58 ..
drwxrwxrwx    3 root     root           232 Oct 13 05:44 contrib
drwxrwxrwx    2 root     root           320 Oct 13 05:59 device-certs
drwxrwxrwx    2 root     root           160 Oct 13 05:42 mosquitto-conf
-rw-rw-rw-    1 tedge    tedge          102 Oct 13 06:55 tedge.toml

Is my custom platform missing some features? Or is it really a permission error?

Best,
Artur

Investigate pros and cons of Cargo workspaces and if we need them.

• We are currently using a Cargo workspace within the thin-edge.io repository.

• To my knowledge, trait coherence rules are different for crates that live within a workspace.

• The risk is to introduce stronger coupling between individual crates, where this would not be allowed when the crates would not live in the same workspace.

• We do not really need Cargo workspaces for that purpose. We only use it as "build tool".

• Research the exact behaviour and pros and cons of using Cargo workspaces.

Make thin-edge.io capable for running on low-footprint Linux distros

It would be cool to have thin-edge.io running on thin linux distros like alpine, debian-slim, etc. Current limitations are the systemd dependency as well as apt dependency. Thin linux OS are often not based on systemd and use alternative package managers like apk or even no one. Also the installation on plc systems is not easy due to these issues.
As an improvement and to get rid at least of the apt dependency, an installation script installing all needed (sub)dependencies for tedge and moquitto would be helpful.

As a user who can't install from deb package I would like to know how to use tedge on my OS

As a user who doesn't run debian package management I would like to know how to install tedge on my OS.

It seems as our users / tasters community grows we are being more often asked how to install thin-edge without support for deb packages (eg I want thin-edge to run on alpine).

As we provide already binaries compatible with other OSes maybe it could be a matter of providing documentation on what needs to be where on the system or a script to install plain binaries and create appropriate directories with permissions?

GH: [https://github.com//issues/505|https://github.com//issues/505|smart-link]

AC:

• As a user I would like to install thin-edge on a Linux OS without deb package support (eg alpine)
• Enhance postinstall scripts with comments about how and why something is done. No need for all details but about what is important for the customer to deploy thin-edge on their system.
• Create doc page with some guidance how to use the posinstall scripts for manual installation.
• Point to extracting binaries and posinstall scripts from debian packages

Split `tedge` into multiple crates

• The tedge crate should only be the plumbing to produce the CLI binary.

• If we decide to put both tedge and mapper into a single binary ("busybox"), we can just plug in the tedge_cli crate and do some minor setup here and there.

• From a testing perspective, it favors unit tests (done in tedge_cli or the individual command crates) over tests where we execute the tedge binary (the latter belong in the tedge crate).

• Much clearer dependencies and better reuse. For instance, I would put the command implementations into separate crates. tedge_command_cert, tedge_command_connect, ... and put all CLI parsing and mapping of the TEdgeOpt to Box<dyn Command> (BuildCommand implementations) into tedge_cli. That means, each Command implementation has zero knowledge about the CLI itself unless these are injected through it's command struct. This favors unit tests where we only test the command implementation and NOT the CLI parsing.

• Simplifies refactoring, as you can make changes local to a crate (lets say migrating tedge_command_config to use a new data structure), without breaking other crates or the tedge_cli.

My proposal is to split them into:

• tedge: This is the driver, the plumbing, that creates the final tedge binary.
• tedge_cli: A library crate that does the CLI parsing and mapping of CLI TEdgeOpt into Commands. All structopt definitions are defined here.
• tedge_command_cert: This only provides the impl Command for CertCommand. It does not contains any structopt related code (that belongs to tedge_cli). Unit tests concerning CertCommand are local to that crate.
• tedge_command_*: Analog to tedge_command_cert
• utility crates....

For busybox we can have a separate binary that basically is a copy of the minimal plumbing done by tedge (10-20 lines of code).

Note, none of the tedge_command_* should depend on tedge_cli. But tedge_cli depends on tedge_command_*.

Don't nail me down on the precise naming of the individual crates. It's more about the conceptual structure.

Document special meaning of ':' character in client id for c8y

Is your documentation request related to a problem? Please describe.
According to the c8y documentation the column character has a special meaning within c8y and should not be used in the device identifier/clientId.

Describe the improvement you'd like
The Documentation should cover the possible issues when using the column character in the device identifier in combination with c8y.
We might also want to adjust the tedge cert create command to throw an error in case a column character is used or to at least output a warning message.
The tedge connect c8y command should then maybe also output a clear warning statement/throw an error if the column character is used.

Additional context
If you e.g. want to use the MAC address of your raspberry pis wifi interface as a unique identifier during the creation of your client certificate you wound have to replace the column characters by something different:
sudo tedge cert create --device-id tedge-$(cat /sys/class/net/wlan0/address | tr ':' '_')

Support c8y's remote access feature (ssh)

Is your feature request related to a problem? Please describe.
We have devices that are not accessible from public and are connected to c8y as children of thin-edge (so, thin-edge is responsible for fog computing in this scenario).
Sometimes we may want to SSH into the child device for debugging etc.

Describe the solution you'd like
One way to acheive above is to make tedge work as a SSH bastion host. I guess the change consists of two parts:

• Manage ssh server using SystemServiceManager
• Add module to handle Cloud Remote Access Connect (530) messages

If you do not mind, I can work on this.

Documentation about our logging

As a user i need to know what logs there are and where I can find these.

AC:

• Mention mosquitto logs
• Mention SM-logs
• Mention Collectd logs
• journalctl

Cloud communication without MQTT

I saw in the Architecture FAQs that:

"Using MQTT for cloud communication is not mandatory. You are free to add additional protocols beside MQTT: Because thin-edge.io has an internal bus, you can implement a bridge to another protocol (e.g. LWM2M or plain HTTPS). In that case, MQTT is used inside the edge devices, and another protocol is used for external communication."

As far as I understand anyone can implement a custom bridge that communicates with some cloud provider not via MQTT but via REST over HTTPS for example. However I couldn't find any documentation about how such a custom bridge can be implemented. I found only the documentation about the mapper. Are there examples of using some other protocol than MQTT for cloud communication?

Also while going through the documentation and examples it seems that the mapper also specifies the outbound MQTT topic that forwards the messages to the cloud, e.g. here for C8Y. Is that correct?

Wrong installation script order

Describe the bug
Installation script now fails for ARM with the following error:

#22 2.972 Unpacking tedge_agent (0.4.0) ...
#22 3.044 dpkg: dependency problems prevent configuration of tedge_agent:
#22 3.044 tedge_agent depends on tedge_apt_plugin; however:
#22 3.044 Package tedge_apt_plugin is not installed.

To Reproduce
Start installation with Version 3.0.0 or 4.0.0

Additional context
I guess it has something to do with the changes of the installation script in commit #9e02441

Frequent drop of operation messages from Cumulocity to Thin Edge

Observing frequent drop of software update operations sent from Cumulocity to thin-edge devices. Not all messages are getting dropped all the time. Sometimes, a couple of requests go through and gets delivered properly and the message delivery stops suddenly that with no external changes to the working system. This issue is only for cloud-to-device operation messages. Other device-to-cloud requests and responses like JWT token retrieval are working fine even when this operation message delivery is broken. But even then, requests to get all PENDING operations (SmartREST 500 message) remain unanswered even when there are PENDING requests queued on the server.

The messages are not even reaching the bridge mosquitto server as there's no trace of a message received from Cumulocity in its logs. But it can be confirmed that the connection is active as per the logs indicating hearbeat ping responses received from Cumulocity frequently:

{code:java}1634546169: Sending PINGREQ to Cumulocity
1634546169: Received PINGRESP from Cumulocity{code}

Timeboxed to 3 days

Fail to create bridge on WSL 2

Describe the bug
When installed and registered as described in the How To, sudo tedge connect c8y fails to create bridge.

To Reproduce
execute sudo tedge connect c8y on WSL

Expected behavior
Successful connection

Screenshots

sudo tedge connect c8y

Checking if systemd is available.

Checking if configuration for requested bridge already exists.

Validating the bridge certificates.

Saving configuration for requested bridge.

Restarting mosquitto service.

Error: failed to create bridge to connect Cumulocity cloud.

Caused by:
Systemd returned unspecific error for service mosquitto while performing restart it.
Hint: Lacking permissions or service's process exited with error code.

Environment (please complete the following information):

• OS System: Ubuntu 20.04.3 LTS WSL 2
• System-Architecture x86_64
• thin-edge.io version 0.4.0

Additional context
I think WSL has no systemd but it prints a Checking if systemd is available. without an error on this line.

Permissons should be alright:

ls -l /etc/tedge
total 16
drwxr-xr-x 3 root      root      4096 Oct 28 15:11 contrib
drwxr-xr-x 2 mosquitto mosquitto 4096 Oct 28 17:38 device-certs
drwxr-xr-x 2 tedge     tedge     4096 Oct 29 09:16 mosquitto-conf
-rw-rr-- 1 tedge     tedge      119 Oct 29 09:16 tedge.toml

and mosquitto:

ls -l /usr/sbin/mosquitto
-rwxr-xr-x 1 root root 297280 Oct 27 23:00 /usr/sbin/mosquitto

AC:

• If systemd is not available on the system create error at the end of the printout:
  ** The user has to start/restart components manually (With reference what actions are required)
  ** Configuration of mosquitto is created
  ** The UX of thin-edge is better with systemd

Test

Test of sync
yyyyaaaa

Software list upload fails for a new device

Describe the bug

When installing 0.4.1 on a fresh device the software list is not uploaded anymore to Cumulocity.

To Reproduce

sudo dpkg -i tedge_0.4.1_armhf.deb tedge_agent_0.4.1_armhf.deb tedge_apt_plugin_0.4.1_armhf.deb tedge_mapper_0.4.1_armhf.deb
sudo tedge cert create --device-id ...
sudo tedge config set c8y.url ....eu-latest.cumulocity.com
sudo tedge cert upload c8y --user ...
sudo tedge connect c8y
sudo tedge disconnect c8y
sudo tedge connect c8y

Expected behavior

A software list appears in Cumulocity.

Screenshots

|https://user-images.githubusercontent.com/75477722/139433973-d15b7efd-d5e2-4336-a0c3-b0c85e70f5ed.png)]

Environment (please complete the following information):

Rpi 3b+
Linux devraspberrypi 5.10.63-v7+ #1459 SMP Wed Oct 6 16:41:10 BST 2021 armv7l GNU/Linux
thin-edge 0.4.1

Additional context
Workaround: Manual restart of the software mapper while being connected:
sudo systemctl restart tedge-mapper-sm-c8y

Timeboxed to 1

Provide configuration via MQTT

Is your feature request related to a problem? Please describe.
For instance when I would like to know the BASE_URL of the cumulocity tenant (needed for remote connections) I have to query tedge commandline from my lib.

Describe the solution you'd like
It would be great to have a MQTT topic where after I publish some empty payload get the current configuration as a json string.

Typo in tedge command line hint

When I do:

sudo tedge cert create --device-id tedge
Error: failed to create a test certificate for the device tedge.

Caused by:
A certificate already exists and would be overwritten.
Existing file: "/etc/tedge/device-certs/thinedge.pem"
Run tegde cert remove first to generate a new certificate.

typo teged should be tedge

Strange output in certificate already exists message

Describe the bug
pi@raspberrypi:~/Downloads $ sudo tedge cert create --device-id chris-pi
Error: failed to create a test certificate for the device chris-pi.

Caused by:
A certificate already exists and would be overwritten.
Existing file: FilePath("/etc/tedge/device-certs/tedge-certificate.pem")
Run tedge cert remove first to generate a new certificate.

I would expect the "Existing file" to be reported without the FilePath() wrapper.

To Reproduce
Run this twice:
sudo tedge cert create --device-id chris-pi

Expected behavior
Existing file: "/etc/tedge/device-certs/tedge-certificate.pem"
with quotes or not - not sure.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS [incl. version]
• Hardware [incl. revision]
• System-Architecture [e.g. result of "uname -a"] Linux raspberrypi 5.4.72-v7l+ #1356 SMP Thu Oct 22 13:57:51 BST 2020 armv7l GNU/Linux
• thin-edge.io version [e.g. 0.1.0] 0.2.1

Additional context
Add any other context about the problem here.

As a user I want the file download to not fail if it is larger than available memory

If my device has limited amount of available system memory download of large software packages fails.

As a user I want packages of a size of available storage to be downloaded.

AC:

• Device can download large files which exceed size of available RAM, but not exceed size of available storage
• Availability of storage+buffer is checked before download
• Add configuration for a buffer size with a default of 5%
• Allocate storage before download

Tips:

Directly stream the content in a file rather than in memory

or

Download chunks and write to files

Exalate test from JIRA

test

Use unique spelling of either `TEdge...` or `Tedge...` but not both

Examples:

• TedgeMapperService
• TEdgeOpt

Provide Dockerfile for thin-edge.io to run within containers

Is your feature request related to a problem? Please describe.
Many gateways only provide the possibility to use containers since they hardly relay on standard libraries and do not want them to be changed. There is currently no provided official Dockerfile for running the thin-edge.io within an container.
Describe the solution you'd like
A Dockerfile that provided the possibility to build an image for thin-edge.io that directly brings all features and functionalities.
Maybe provide the container also on Docker hub as well.
Describe alternatives you've considered
Native installation is in several gateways and use cases not possible. Building up an own Dockerfile partly worked since some of the processes can not be automized.

Documentation tests are not running in tedge crate

Describe the bug
We have some documentation tests in tedge crate (e.g. commands.rs), but they are not executed by cargo test.

To Reproduce
Run cargo test.
The problem is tedge is binary project. Rustdoc's documentation tests can run only for library project.

Expected behavior
Do we want to execute documentation tests, don't we? I think the tests written in documentation in tedge should go to tests module.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS [incl. version] Ubuntu 18.04
• Hardware [incl. revision]
• System-Architecture [e.g. result of "uname -a"] Linux 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
• thin-edge.io version 0.1.0

Additional context
Add any other context about the problem here.

No events shown in Azure on publish

Describe the bug
When In follow the docs to publish to Azure. Connection to IoT Hub is fine. Publish works also. But I don't see the events when I subscribe in Azure like that:

az iot hub monitor-events --output table --hub-name msIoTHub

when I create a dummy device and use a simulator

az iot hub monitor-events --output table --hub-name msIoTHub

I see the dummy events .....

Expected behavior
see the thinedge events in az monitor-events

Environment (please complete the following information):
Linux UbuntuTest 5.4.0-1043-azure #45-Ubuntu SMP Fri Mar 19 17:33:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

• thin-edge.io version [e.g. 0.1.0]

Additional context
Add any other context about the problem here.

Convert {MqttMessageStream, MqttErrorStream} into futures::stream::Stream

Using the topic benefit of MQTT for Measurement Mappings

Is your feature request related to a problem? Please describe.
Currently the tedge/measurements awaits an json based payload. However this does not make benefit of the topic structure provided by MQTT. Many gateways, sensors, implementations, programs do send their data depending on their meaning e.g. sensor1/temperature/internal with a json or non json based payload.

Describe the solution you'd like
It would be great to provide next to tedge/measurements also the possibility to have tedge/measurements/x/y that creates the measurement mapping depending on the used topic structure for measurements and series. A good approach for e.g.

tedge/measurement/sensor1/temperature/internal and the payload: {value: 12, unit: "°"} could be:

{
"sensor1/temperature":
{"internal":
{"value": 12, "unit": "°"}}
}

or

{
"sensor1":
{"temperature/internal":
{"value": 12, "unit": "°"}}
}

So basically using the last or everything except the first as series and everything else as measurement. However the example is just an idea.
Describe alternatives you've considered
The current concept relays that the device already builds up a logic within the payload such that logic or implementation is needed. However many devices allow the configuration of blank payloads that are just send on various topics where currently implementation effort as additional module is required.

Persitence Service Addon (DB based buffering)

Some Devices are running in very network weak regions or even have only a cyclic transmission rate (e.g. once per day). For those devices it would be good to have an persistence service, which is storing data (not just in-memory) over a longer period and flush them at connection time.

It would be cool to have addons for connecting databases like influx, mongo or round-robin mechanisms based ones, which then could take care about data persistence during disconnected operation time.

Test for sync JIRA

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior. The more detail you add here, the better we can reproduce the bug.

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS [incl. version]
• Hardware [incl. revision]
• System-Architecture [e.g. result of "uname -a"]
• thin-edge.io version [e.g. 0.1.0]

Additional context
Add any other context about the problem here.

c8y bridge misses topics for JWT token retrieval

Describe the bug
The configuration for the mosquitto bridge to c8y (/etc/tedge/mosquitto-conf/c8y-bridge.conf) does not bridge the following topics:

• s/dat
• s/uat

To Reproduce
Follow steps described here for JWT token retrieval.

Expected behavior
Those topics should be bridged in case a custom implementation requires access to c8y via HTTP(S).

Environment (please complete the following information):

• Raspbian GNU/Linux 10 (buster)
• Raspberry Pi 4
• Linux raspberrypi 5.10.17-v7l+ #1414 SMP Fri Apr 30 13:20:47 BST 2021 armv7l GNU/Linux
• thin-edge.io version 0.2.0

Additional context
Similar to #300
Maybe someone should align with c8y R&D to find out all other possibly missing topics.

Installing new version of thin-edge.io isnt working when still connected

As a user of thin-edge i might not think about disconnecting before installing a new version

Reproduce:

E..g have thin-edge.io 0.3 installed

have thin-edge connected to c8y.

Try to over-installing 0.4

postrm of tedge-mapper 0.3 fails and therefore also install 0.4 fails

Resolution options:

Tell the user to disconnect thin-edge for installation of new version, and user should connect afterwards again.

Update installation document to disconnect tedge when you upgrate tedge-mapper.

prerm script should give a hint that user must stop tedge-mapper.

The device id should be derived from the certificate Common Name.

Currently the device id is stored in the configuration when a certificate is created.

• tedge cert create --device-id $MY_DEVICE_ID creates a certificate using the given id as the common name, and then updates the configuration.
• tedge config get device-id returns the identifier set by tedge cert create, if any.
• tedge config set device-id $MY_DEVICE_ID returns an error.
• tedge cert remove removes the certificate, and then unset the device id in the configuration.

Storing the same information in two places has several drawbacks.

• There is currently no way to use a certificate that has not been created with tedge cert create.
• If for some reason the configuration is out-of-sync, then the user is stuck. For instance, if the device id is not in the config while a certificate exists, then the user can not connect their device.
• The configuration can be out-of-sync, simply by setting device.cert.path so it points to a valid certificate.
• The configuration might be out-of-sync, because of a race condition (unlikely but possible).
• The UX model is not crystal clear, since most of the parameters can be set in the config but not all. Is this still a config?

There are several approaches to fix this.

1. Let the device id in the config file as today, and add a tedge cert use command to use a certificate that has been created elsewhere.
2. Remove the device id from the config file, but let the user query the config for the device.id parameters.
3. Remove the device id both from the config file and from the tedge config command. Add a --device-id option to the tedge cert show command.

Pros/Cons

• The option 2) let unchanged the current user experience, but complexifies the code a lot (the macro used underneath has to cope two different types of parameters (either Option<String> or akin to Fn<TEdgeConfig> -> Option<String>).
• The options 1) and 3) are easier to implement since there is no attempt to compute the device id on tedge config get.
• With option 3) the work recently done to add read-only config parameters has to be removed.
• The options 2) and 3) are DRY
• The option 3) has a user model that is simpler grasp, since the user provides and retrieves the device id from the very same command, i.e. tedge cert.

I prefer option 3, because both the UX and the code will be simpler.

Only keep in the config parameters that can be get and set.
Remove the device id from the config.
Add a --device-id option to the tedge cert show command.

Allow MQTT connection from external clients

Is your feature request related to a problem? Please describe.
As mentioned in #421, we would like to have children devices under thin-edge's local network. However, thin-edge prohibits mosquitto to establish a connection with external clients, see:

thin-edge.io/tedge/src/cli/connect/common_mosquitto_config.rs

Line 52 in c988c01

let listener = port.to_string() + " localhost";

Workaround for now is to edit /etc/mosquitto/mosquitto.conf manually.

Describe the solution you'd like
We can let users choose whether they want to limit external connections, through tedge config.
e.g. if mqtt.allow_remote is set to true, "0.0.0.0" suffix will be added instead of "localhost".

Test issue

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior. The more detail you add here, the better we can reproduce the bug.

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS [incl. version]
• Hardware [incl. revision]
• System-Architecture [e.g. result of "uname -a"]
• thin-edge.io version [e.g. 0.1.0]

Additional context
Add any other context about the problem here.

idling behavior of tedge-mapper-sm-c8y

Describe the bug
tedge-mapper-sm-c8y behaves unlike other mappers. If thin-edge is started in an unregistered state (public key not yet uploaded to tenant) to wait for registration the sm mapper will crash while everything else is able to stay in an idle state until public key is uploaded.

Expected behavior
thin-edge should be able to stay in an idle state until connection can be set up (public key uploaded, internet access, ...)

Environment (please complete the following information):

• thin-edge.io version 0.4.0

AC:

• SM Mapper should retry to initialise and establish connection.