IoT Edge produces too much traffic with default configuration, it is not conducive for billable networks such as cellular. about iotedge

[uriel-kluk]

We opened a support ticket and we can close the current issue. Thanks

[nyanzebra]

@varunpuranik / @veyalla do we have any customer facing documentation on the performance breakdowns of mqtt vs amqp?

1. This might be marketplace module specific, do you have a specific set of modules in mind? For example, the SimulatedTemperatureSensor can have a custom route defined.
2. Believe those messages correspond to device token refreshes for connections, are you saying you want control over that frequency?
3. The queue cleanup should be after messages are sent and doing a batch cleanup of messages from underlying store. Can double check this, but think this happens every so often to clean up state from the underlying database.

For 4,5,6 @vipeller or @varunpuranik any guidance on these?

[uriel-kluk]

Thanks @nyanzebra,

Believe those messages correspond to device token refreshes for connections, are you saying you want control over that frequency?

My only motivation to ask about token refreshnes is to reduce traffic. I see a periodic trace every 5 minutes reauthenticating connected clients. I hope this is not creating cellular traffic.

But the main question remains the same, what is causing downloads, if the application is just pumping telemetry?

[nyanzebra]

@uriel-kluk is the graph for network traffic to/from device? There should be occasional twin data sent to device to see if any changes in modules need to happen as well as some connectivity checks, but would be surprised if it is significant. I think the next steps are to dig deeper.

If you have metrics it will be interesting to see what the following say:

• edgehub_gettwin_total
• edgeAgent_total_network_in_bytes

Additionally, if you can provide a support bundle that will be useful.

[uriel-kluk]

Hi Robert, here is a chart that shows the docker networkstats​ by container.  Clearly the traffic comes from edgeHub, in this ilustration, SimulatorModule is just reporting telemetry periodically. and you can see the line follows along and the delta is almost flat. But edgeAgent​ had a spike of 35K. I am going to move all the heads to AMQP to see if I gain anything. But I am almost sure it has to do with authentication. Note: The edge device uses x509 to authenticate with iothub, and the modules use SasTokens ```c# var mqttSetting = new MqttTransportSettings(TransportType.Mqtt_Tcp_Only) { KeepAliveInSeconds = 5 * 60 }; var options = new ClientOptions { SasTokenTimeToLive = TimeSpan.FromHours(31), SasTokenRenewalBuffer = 50, ModelId = ModelId, }; ITransportSettings[] settings = { mqttSetting }; // OpenAsync a connection to the Edge runtime _client = await ModuleClient.CreateFromEnvironmentAsync(settings, options); if (_client == null) { Log.LogError("Failed to connect to IoT Hub"); return; } ```

…

________________________________ From: Robert Jang ***@***.***> Sent: Thursday, March 28, 2024 12:04 PM To: Azure/iotedge ***@***.***> Cc: Uriel Kluk ***@***.***>; Mention ***@***.***> Subject: Re: [Azure/iotedge] IoT Edge produces too much traffic with default configuration, it is not conducive for billable networks such as cellular. (Issue #7252) @uriel-kluk<https://github.com/uriel-kluk> is the graph for network traffic to/from device? There should be occasional twin data sent to device to see if any changes in modules need to happen as well as some connectivity checks, but would be surprised if it is significant. I think the next steps are to dig deeper. If you have metrics<https://learn.microsoft.com/en-us/azure/iot-edge/how-to-access-built-in-metrics?view=iotedge-1.4> it will be interesting to see what the following say: * edgehub_gettwin_total * edgeAgent_total_network_in_bytes Additionally, if you can provide a support bundle that will be useful. — Reply to this email directly, view it on GitHub<#7252 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJ2NCA652ENCF5XPDNSVAETY2REQJAVCNFSM6AAAAABFLGYH7OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRVG4YDEOJQGM>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[uriel-kluk]

Hi @nyanzebra ,

Attached support bundle.

Please look at this chart:

It is clear that the edgeHub is renogotiating credentials every hour. Is there a way to control the frequency?
Identity attestation uses x.509 in this illustration.

Also, the custom module is doing authentication every hour. this is less impactful but yet it creates traffic.
This is the code I am using to connect the custom module with the edgeHub:
edge-device-support-bundle.tar.gz

ITransportSettings[] GetTransportSettings()
            {
                switch (transportType)
                {
                    case TransportType.Mqtt:
                    case TransportType.Mqtt_Tcp_Only:
                        return new ITransportSettings[] { new MqttTransportSettings(TransportType.Mqtt_Tcp_Only) };
                    case TransportType.Mqtt_WebSocket_Only:
                        return new ITransportSettings[] { new MqttTransportSettings(TransportType.Mqtt_WebSocket_Only) };
                    case TransportType.Amqp_WebSocket_Only:
                        return new ITransportSettings[] { new AmqpTransportSettings(TransportType.Amqp_WebSocket_Only) };
                    default:
                        return new ITransportSettings[] { new AmqpTransportSettings(TransportType.Amqp_Tcp_Only) };
                }
            }

            var settings = GetTransportSettings();
            var options = new ClientOptions
            {
                ModelId = ModelId,
                SasTokenTimeToLive = TimeSpan.FromDays(1),
            };

            // OpenAsync a connection to the Edge runtime
            _moduleClient = await ModuleClient.CreateFromEnvironmentAsync(settings, options);

[nyanzebra]

@uriel-kluk would you mind trying to set DeviceScopeCacheRefreshRateSecs to a much larger number (default is 3600s or 1hr). I am hoping this reduces the download quite a bit, it also shouldn't effect edgeHub really.

[uriel-kluk]

Thank you @nyanzebra, I tried almost every parameter, but I am getting old and lazy, and because I saw the leaf description, I dismissed. Fingers crossed, this solves the issue. I should be able to provide an update in about an hour :)

Last few days with the default one hour:

[david-emakenemi]

@uriel-kluk did the solution that Robert provided solve the issue?

[claya75]

@david-emakenemi @nyanzebra Unfortunately, no. Thanks for all the great input on keeping our IoT Edge traffic lean on cellular. I've been digging into this with @uriel-kluk and with Wireshark I found some spots where we're using more bandwidth than we'd like or expect. I've attached some captures and charts to help illustrate.

We would love to get your take on understanding why we'd be getting so much Rx traffic on edgeHub as shown below, and/or how to reduce it.

This screenshot is of a Wireshark capture with all Tx traffic filtered out, so this is only Rx traffic coming into our edge device, viewed both in bytes downloaded over time and the delta time between packets over time.

This is a typical 24 hour period reported with our SuperSIM from Twilio.

Capture files (.pcapng) both for 2.5hrs, one with TCP and TLS only filtered.
captures.zip

[nyanzebra]

@claya75 & @uriel-kluk

Looking at the attached pcap, it looks like packets regarding certs are few and while large, aren't the majority of downloads.

When looking further it looks like there is a consistent traffic being sent from 20.49.109.144, which I assume to be your iothub, of about 135 length (TLS record len 64). Given the size, these are likely not PUBACKs within MQTT.

I will try to reproduce by setting up a local device and see if can monitor traffic w/o TLS to see if can get some answers. Will also ask IoTHub team if they have any ideas.

[uriel-kluk]

Thank you for Robert looking deeper. We are changing the runtime settings as follows: "systemModules": { "edgeAgent": { "type": "docker", "env": { "SendRuntimeQualityTelemetry": { "value": false }, "ConfigRefreshFrequencySecs": { "value": 86400 }, "DisableDeviceAnalyticsMetadata": { "value": true }, "MetricsEnabled": { "value": false }, "RuntimeLogLevel": { "value": "info" }, "UpstreamProtocol": { "value": "Mqtt" } }, "settings": { "image": "mcr.microsoft.com/azureiotedge-agent:DEFAULT_RT_IMAGE", "createOptions": { "LogConfig": { "Type": "json-file", "Config": { "max-size": "1m", "max-file": "3" } } } } }, "edgeHub": { "type": "docker", "status": "running", "restartPolicy": "always", "env": { "UpstreamProtocol": { "value": "Mqtt" }, "OptimizeForPerformance": { "value": true }, "HttpSettings__Enabled": { "value": false }, "AmqpSettings__Enabled": { "value": false }, "CloudConnectionIdleTimeoutSecs": { "value": 86400 }, "ConfigRefreshFrequencySecs": { "value": 86400 }, "RuntimeLogLevel": { "value": "info" }, "DeviceScopeCacheRefreshRateSecs": { "value": 86400 }, "MinTwinSyncPeriodSecs": { "value": 86400 }, "ConnectivityCheckFrequencySecs": { "value": 300 } }, "settings": { "image": "mcr.microsoft.com/azureiotedge-hub:DEFAULT_RT_IMAGE", "createOptions": { "HostConfig": { "PortBindings": { "5671/tcp": [ { "HostPort": "5671" } ], "8883/tcp": [ { "HostPort": "8883" } ], "443/tcp": [ { "HostPort": "443" } ] } } } } } },

…

________________________________ From: Robert Jang ***@***.***> Sent: Tuesday, April 9, 2024 11:16 AM To: Azure/iotedge ***@***.***> Cc: Uriel Kluk ***@***.***>; Mention ***@***.***> Subject: Re: [Azure/iotedge] IoT Edge produces too much traffic with default configuration, it is not conducive for billable networks such as cellular. (Issue #7252) @claya75<https://github.com/claya75> & @uriel-kluk<https://github.com/uriel-kluk> Looking at the attached pcap, it looks like packets regarding certs are few and while large, aren't the majority of downloads. When looking further it looks like there is a consistent traffic being sent from 20.49.109.144, which I assume to be your iothub, of about 135 length (TLS record len 64). Given the size, these are likely not PUBACKs within MQTT. I will try to reproduce by setting up a local device and see if can monitor traffic w/o TLS to see if can get some answers. Will also ask IoTHub team if they have any ideas. — Reply to this email directly, view it on GitHub<#7252 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJ2NCAZD4ZT5TGQN6UAP2MDY4QH5HAVCNFSM6AAAAABFLGYH7OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBVGU4DMMRZHE>. You are receiving this because you were mentioned.

[ryanwinter]

@uriel-kluk is this still an outstanding issue?

@nyanzebra do you have any update here?