freertos / coremqtt

Client implementation of the MQTT 3.1.1 specification for embedded devices

The FreeRTOS 202212.00 release updates FreeRTOS Kernel, FreeRTOS+TCP, coreMQTT, corePKCS11, coreHTTP, coreJSON, AWS IoT Over-the-air-Updates (OTA), AWS IoT Device Shadow, AWS IoT Jobs, AWS IoT Device Defender, Backoff Algorithm, AWS IoT Fleet Provisioning, coreSNTP, SigV4, and FreeRTOS Cellular Interface libraries to their LTS 2.0 versions. It also updates coreMQTT Agent to v1.2.0 to be compatible with coreMQTT v2.X.X, and updates MbedTLS to v3.2.1. This release also adds Visual Studio static library projects for the FreeRTOS Kernel, FreeRTOS+TCP, Logging, MbedTLS, coreHTTP, and corePKCS11. With the addition of the static library projects, all Visual Studio projects have been updated to use them. Additionally, all demos dependent on coreMQTT have been updated to work with coreMQTT v2.X.X.

Getting started

The FreeRTOS.org website contains a FreeRTOS Kernel Quick Start Guide, a list of supported devices and compilers, the API reference, and many other resources.

Getting help

You can use your Github login to get support from both the FreeRTOS community and directly from the primary FreeRTOS developers on our active support forum. The FAQ provides another support resource.

Cloning this repository

This repo uses Git Submodules to bring in dependent components.

Note: If you download the ZIP file provided by the GitHub UI, you will not get the contents of the submodules. (The ZIP file is also not a valid git repository)

If using Windows, because this repository and its submodules contain symbolic links, set core.symlinks to true with the following command:

git config --global core.symlinks true

In addition to this, either enable Developer Mode or, whenever using a git command that writes to the system (e.g. git pull, git clone, and git submodule update --init --recursive), use a console elevated as administrator so that git can properly create symbolic links for this repository. Otherwise, symbolic links will be written as normal files with the symbolic links' paths in them as text. This gives more explanation.

To clone using HTTPS:

git clone https://github.com/FreeRTOS/FreeRTOS.git --recurse-submodules

Using SSH:

git clone [email protected]:FreeRTOS/FreeRTOS.git --recurse-submodules

If you have downloaded the repo without using the --recurse-submodules argument, you need to run:

git submodule update --init --recursive

Repository structure

This repository contains the FreeRTOS Kernel, a number of supplementary libraries including the LTS ones, and a comprehensive set of example projects. Many libraries (including the FreeRTOS kernel) are included as Git submodules from their own Git repositories.

Kernel source code and example projects

FreeRTOS/Source contains the FreeRTOS kernel source code (submoduled from https://github.com/FreeRTOS/FreeRTOS-Kernel).

FreeRTOS/Demo contains pre-configured example projects that demonstrate the FreeRTOS kernel executing on different hardware platforms and using different compilers.

Supplementary library source code and example projects

FreeRTOS-Plus/Source contains source code for additional FreeRTOS component libraries, as well as select partner provided libraries. These subdirectories contain further readme files and links to documentation.

FreeRTOS-Plus/Demo contains pre-configured example projects that demonstrate the FreeRTOS kernel used with the additional FreeRTOS component libraries.

Previous releases

Releases contains older FreeRTOS releases.

FreeRTOS Lab Projects

FreeRTOS Lab projects are libraries and demos that are fully functional, but may be experimental or undergoing optimizations and refactorization to improve memory usage, modularity, documentation, demo usability, or test coverage.

Most FreeRTOS Lab libraries can be found in the FreeRTOS-Labs repository.

A number of FreeRTOS Lab Demos can be found in the FreeRTOS Github Organization by searching for "Lab" or following this link to the search results.

coreMQTT Agent Demos

The FreeRTOS/coreMQTT-Agent-Demos repository contains demos to showcase use of the coreMQTT-Agent library to share an MQTT connection between multiple application tasks.

The demos show a single MQTT connection usage between multiple application tasks for interacting with AWS services (including Over-the-air-Updates, Device Shadow, Device Defender) alongside performing simple Publish-Subscribe operations.

CBMC

The FreeRTOS/Test/CBMC/proofs directory contains CBMC proofs.

To learn more about CBMC and proofs specifically, review the training material here.

In order to run these proofs you will need to install CBMC and other tools by following the instructions here.

coremqtt's Issues

Log functions cause compiler warnings

Hi everybody,

First of all very good job with the new version, the quality seems to be improved massively. I hope this will be the last major version I will be porting :)

I am porting the SDK to Mbed OS, and realized that the log functions cause compiler warnings. My GCC ARM version is 9 2020-q2-update.

There might be some I missed, but here is the what I caught:

[Warning] core_mqtt.c@608,25: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@623,25: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@623,25: format '%d' expects argument of type 'int', but argument 6 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@635,21: format '%u' expects argument of type 'unsigned int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'} [-Wformat=]
[Warning] core_mqtt.c@717,25: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@734,25: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@734,25: format '%d' expects argument of type 'int', but argument 6 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@786,25: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@813,21: format '%d' expects argument of type 'int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'} [-Wformat=]
[Warning] core_mqtt.c@854,24: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@859,25: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@949,25: format '%d' expects argument of type 'int', but argument 5 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@1388,21: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@1406,29: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@1775,25: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@1866,25: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@2001,25: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@2061,25: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@2111,25: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]

Support for MQTT v5.x

The AWS MQTT broker supports MQTT v5.x. To make the most out of integration of devices to AWS MQTT broker using MQTT v5.x, the coreMQTT library needs an update as it currently is compliant to MQTT v3.x
Could this update be considered ?

Thank you.

TransportInterface_t.writev is no longer optional and must be provided?

Been trying to port my code to v2.1.1 and getting crashes. It looks like the cause is that TransportInterface_t.writev has to be provided to MQTT_init. This used to be optional on v1.2.0. Documentation does not mention this as now mandatory.

MQTT_Connect is a blocking call

Dear

I am using coreMQTT in an event based embedded architecture and I faced problems with the function MQTT_Connect as it's a blocking function, waiting on the connack message. Due to this blocking behaviour, I never get to the point of processing a TCP receive event which is preventing the connack message to be received.

Why is this function implemented to be blocking?

PS. I currently patched the coreMQTT library to avoid this blocking behaviour. I am willing to share this.

Handle QoS2 messages when the library loses state - store state in NVM

It is a very narow corner case but stil it breaks the client and renders it unusable.

1. coreMQTT client is running on an embedded device, it subscribed to a topic with QoS2.
2. The issue happens when a message is received and then the device reboots (for unrelated to coreMQTT reasons).
3. When it boots again and re-connects to the broker it immidiately receives MQTT_PACKET_TYPE_PUBREL but since the device was rebooted the library has no knowleadge of the packet it received before, outgoingPublishRecords is empty and MQTT_ProcessLoop returns MQTTBadResponse.

Q: How would you suggest to handle a situation like this? There are no public functions in the library to get the failed packet id and to get access to outgoingPublishRecords structure that one could simply inject a record with id and MQTT_PACKET_TYPE_PUBCOMP type to let go that record. In the corner case like this I guess it's ok to lose the record since not much you can do if device reboots.

What happens when the monotonic clock overflows?

The monotonic clock uses a 32-bit integer in milliseconds and so will overflow in just under 50 days. It does not have a per-connection epoch, so the integrator is required to use something like from boot or from first connection. After 50 days, it will overflow. Does the library handle this (in which case, can it be documented)? Is there some way of resetting the epoch? Failing that, is it possible to have an option to use 64-bit timestamps?

coreMQTT context does not allow user parameters to be passed to the Event Callback

Hello,
I have been using the library on a C++ project and would like to pass the this pointer to the Event Callback to be able to access my class instance.
Is it possible to add a void * pUserData field to the context and pass it to the callback?
Thank you!

undefined reference to `main' in building phase

When building coreMQTT in Ubuntu 23.10 I get this error:

raphy@raohy:~/libpeer/dependencies/coreMQTT$ gcc -I source/include/ -DMQTT_DO_NOT_USE_CUSTOM_CONFIG -I source/interface/ source/core_mqtt.c source/core_mqtt_state.c source/core_mqtt_serializer.c -o coreMQTT
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/13/../../../x86_64-linux-gnu/Scrt1.o: in function `_start':
(.text+0x1b): undefined reference to `main'
collect2: error: ld returned 1 exit status

What am I doing wrong? How make it work?

Log functions cause compiler warnings

Hi everybody,

First of all very good job with the new version, the quality seems to be improved massively. I hope this will be the last major version I will be porting :)

I am porting the SDK to Mbed OS, and realized that the log functions cause compiler warnings. My GCC ARM version is 9 2020-q2-update.

There might be some I missed, but here is the what I caught:

[Warning] core_mqtt.c@608,25: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@623,25: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@623,25: format '%d' expects argument of type 'int', but argument 6 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@635,21: format '%u' expects argument of type 'unsigned int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'} [-Wformat=]
[Warning] core_mqtt.c@717,25: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@734,25: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@734,25: format '%d' expects argument of type 'int', but argument 6 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@786,25: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@813,21: format '%d' expects argument of type 'int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'} [-Wformat=]
[Warning] core_mqtt.c@854,24: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@859,25: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@949,25: format '%d' expects argument of type 'int', but argument 5 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@1388,21: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@1406,29: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@1775,25: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@1866,25: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@2001,25: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@2061,25: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]
[Warning] core_mqtt.c@2111,25: format '%d' expects argument of type 'int', but argument 4 has type 'int32_t' {aka 'long int'} [-Wformat=]

Unsubscribe example calls MQTT_Subscribe

Hi all,

found a typo:

coreMQTT/source/include/core_mqtt.h

Line 563 in 4b8674a

* status = MQTT_Subscribe( pContext, &unsubscribeList[ 0 ], NUMBER_OF_SUBSCRIPTIONS, packetId );

Cheers

MQTTEventCallback_t doesn't have user data parameter

A user callback is supposed to have user data passed in, right?

typedef void (* MQTTEventCallback_t )( struct MQTTContext * pContext,
                                       struct MQTTPacketInfo * pPacketInfo,
                                       struct MQTTDeserializedInfo * pDeserializedInfo );

There is no way to use user context inside the callback in this form. How would one pass a class object or container of subscriptions? Global variables or static class members are a very lousy approach.

MQTT Subscribing for published data (Payload)

Hi,
I have a query , after using the AWS library can i get the data which i have sent to AWS IOT core using MQTT?

Lets say i sent {"Temp":80} now as soon as sent the data , i want to receive once it got published using MQTT.

CROSS COMPILATION

Can anyone tell me the cross compilation instructions for the Library?

Continuously send `PINGREQ` after `PACKET_T(R)X_TIMEOUT_MS`

Hi. I am developing an MQTT client based on coreMQTT-Agent v1.2.0 including coreMQTT v2.1.0. In the config file, I set PACKET_TX_TIMEOUT_MS and PACKET_RX_TIMEOUT_MS with the same value: 10000 (10 seconds). After starting my MQTT task, connecting to the broker and normal messaging, after 10 seconds with a polling interval of 200ms of the recv callback from the transport interface, continuous MQTT_Ping() spam begins. It never ends even if there is an exchange of messages with the broker

Roadmap to support MQTT 5.0?

Compile error for comparison of integer expressions of different signedness

../../../../lib/mqtt_client/coreMQTT/source/core_mqtt.c: In function 'sendConnectWithoutCopy':

../../../../lib/mqtt_client/coreMQTT/source/core_mqtt.c:2220:50: error: comparison of integer expressions of different signedness: 'int' and 'unsigned int' [-Werror=sign-compare]

     assert( ( pIndex - connectPacketHeader ) <= sizeof( connectPacketHeader ) );

Wifi Disconnection Handling

I currently have a task that handles all coreMQTT access for thread safety which I wrote before I learned of coreMQTT agent. I have a another task that establishes the TLS connection and passes requests for subs, pubs, unsubs, etc. It's been working well in almost all aspect except for when an unexpected disconnection occurs.

When the wifi disconnection happens, I end the connection establishing task. Before ending this task I clean everything up (free memory, disconnect calls, etc). I restart this same task, but the problem a successful connection seems to take a long time (very infrequently) or never at all (most of the time).

So is there a guide on how a reconnection should happen on unexpected network disconnection? I know this question isn't directly related to coreMQTT itself, but I think I've exhausted all the resources available to me. Thank you.

Assert fails on empty string

I'm trying to port my code from v1.2.0 to v2.1.0, which is integrated into the coreMQTT-Agent. However, I have come across an unexpected assertion firing which was not present in previous v1.2.0. Upon analyzing the code, I discovered that the problem also exists in v2.1.1 too.

coreMQTT/source/core_mqtt.c

Line 1869 in b0acf49

assert( ( length != 0U ) == ( string != NULL ) );

In my code I am trying to connect to a test broker that does not require a login and password. That's why I pass empty strings as "" and not NULL. strlen of the empty string returns 0, so this assertion fires: I have a valid string that is not NULL, but its length is 0.

Workaround of this problem can be achieved by passing a pointer to NULL instead of the empty string "". But still I think this case should not be handled as fatal error.

No ack found for packet id

After a running for a long time, these messages accumulate:

I (146451442) network: Attempting to connect to MQTT
I (146450881) hmqtt: Connected to MQTT.
I (146450882) hmqtt: Session present.
E (146450882) coreMQTT: No ack found for packet id 15788.
E (146450886) coreMQTT: No ack found for packet id 16425.
E (146450893) coreMQTT: No ack found for packet id 60272.
E (146450899) coreMQTT: No ack found for packet id 26707.
E (146450905) coreMQTT: No ack found for packet id 28396.
E (146450912) coreMQTT: No ack found for packet id 49384.
E (146450918) coreMQTT: No ack found for packet id 49987.
E (146450925) coreMQTT: No ack found for packet id 62338.
E (146450931) coreMQTT: No ack found for packet id 43880.
E (146450938) coreMQTT: No ack found for packet id 34763.
I (146450944) hmqtt: Session resumed.
I (146451992) network: ConnectToMQTT succeeded
I (146450954) hmqtt: Waiting for network to be ready...
I (146450960) network: Subscribing to MQTT topics ...

This only happens on some devices, some of the time.
What is the cause of this issue? Is it due to maintaining the session across reconnects?

I am using coreMQTT v2.1.1 and coreMQTT-Agent V1.2

MQTT_Status_strerror() desn't implement the MQTTNeedMoreBytes enum value.

Bug description:
MQTT_Status_strerror() for the MQTTNeedMoreBytes status returns "Invalid MQTT Status code" string.

---

Expectations:
MQTT_Status_strerror() for the MQTTNeedMoreBytes status should return "MQTTNeedMoreBytes" string.

Publish not checking for entire payload to have been sent

coreMQTT/source/core_mqtt.c

Line 632 in 4b8674a

if( totalBytesSent > 0 )

I think this should be comparing with bytesToSend instead of 0.

On a side note bytesRemaining is assigned twice at the beginning of the function.

Keep Alive is not treated according to MQTT standard

It was found that coreMQTT application that intensively sends messages to a cloud can't pass Device Advisor "Keep Alive" test.
Details are described in an attached doc.
RD-MQTTKeepAlivetestissue-211122-1343.pdf

Problem of unnecessary waiting time for reception

I use the FreeRTOS-LTS 2022210.01.
I realized that CoreMQTT had a idle period every data paket of OTA when I do OTA of FreeRTOS.

I doesn't happen when I use previsou version of LTS of FreeRTOS of CoreMQTT ver 1.

It affects the period of OTA because the wast idle time in here.
Below figure is the waveform I got do the OTA using FreeRTOS v202210.01 with CoreMQTT v2.
As you can see the idle period is here at the end of OTA packets.

I checked why this issue is happned, then the below code is requested the Receive API not to check the total packet size.
Then, the Receive API is waiting until timeout, because the recevied packet size is smaller than the upper API requested.

coreMQTT/source/core_mqtt.c

Line 1685 in 03290fe

recvBytes = pContext->transportInterface.recv( pContext->transportInterface.pNetworkContext,

CoreMQTT receive API says the a byte recevie doesn't block. So we have implemeted it.
But V2 of CoreMQTT happened avobe issue.

https://freertos.github.io/coreMQTT/v2.1.1/group__mqtt__callback__types.html#ga227df31d6daf07e5d833537c12130167

Fix topic name parsing

The value of the topic name variable is taking the value of the topic name + payload
The error is in the line 1448

Example:

• TopicName --> $aws/things/test/jobs/start-next/accepted{"clientToken":"test","timestamp":1712955349}

Must be

• TopicName --> $aws/things/test/jobs/start-next/accepted

Documentation - Commented Code Examples

The example code showing how to use MQTT_Init needs to be fixed. From example snippet

MQTTContext_t mqttContext;
TransportInterface_t transport;
MQTTFixedBuffer_t fixedBuffer;
uint8_t buffer[ 1024 ];

// Clear context.
memset( ( void * ) &mqttContext, 0x00, sizeof( MQTTContext_t ) );

// Set transport interface members.
transport.pNetworkInterface = &someNetworkInterface;

transport.pNetworkInterface does not exist. See below.
From coreMQTT/source/interface/transport_interface.h:

typedef struct TransportInterface
{
    TransportRecv_t recv;               /**< Transport receive interface. */
    TransportSend_t send;               /**< Transport send interface. */
    NetworkContext_t * pNetworkContext; /**< Implementation-defined network context. */
} TransportInterface_t;

LIMIT ON INCOMING MESSAGE SIZE/LENGTH

Is there any limit of incoming messaged to mqtt. I am facing an issue where i am able to receive short message from Aws iot shadow, but when the shadow document has lots of fields(nesting) i am not able to receive the message.

Question regarding CA certificate.

Is the CA certificate also sent to AWS-iot core during the TLS connection?

No new tagged version release for over a year

Hi,
I'm wondering why there hasn't been an official release in over a year? We took v2.1.1 considering it was stable, but when the all the fixes going for keep alive handling going to be merged into an official release? Some of these things have been fixed for quite some time!
Thank you!

MQTT v5

Provide support for the new MQTT5 connectivity protocol for all FreeRTOS users. MQTT v5 builds on the MQTT v3.1.1 protocol used by most devices to connect to AWS IoT Core but adding a number of new features. While the most prominent features are impacting large (Industrial) IoT devices allowing them to send multi-megabyte messages (not applicable to the typically constrained FreeRTOS MCU based devices), making the protocol integration available future-proofs FreeRTOS and will strengthen the value proposition of both FreeRTOS and IoT products.

Is it safe to call coreMQTT APIs from a user callback?

Hi,

Is it safe for a user callback to call a coreMQTT API?

I seem to be seeing that calling MQTT_ProcessLoop from a user callback wouldn't be safe if the nested call hits an error path and calls discardStoredPacket (here), setting the index to 0 (here), and causing an overflow in the parent call (here), but I hope to be wrong.

coreMQTT State Access Audit

Description

CoreMQTT likely relies upon state - state which may be accessible across FreeRTOS tasks. We want to verify CoreMQTT is task/thread safe by ensuring that state access across FreeRTOS tasks is done so behind a synchronization mechanism.

Help is wanted to audit the repository for state which can be used outside of the MQTT task/thread.

This work is being tracked by the FreeRTOS team however it may not be prioritized for some time. Your help in identifying problem state variables/structures will help us in correcting these issues faster. As always, if you are able to fix the issue, we appreciate the help and will gladly look at your PR.

Related Issues

• OTA library state issue