Manual IoT Edge CA certificate renewal error about iotedge

[bilalsellak]

Is this message the only thing you are seeing? It would make sense that edgeAgent would be unable to decrypt the stored configuration since keys rotated, and I can reproduce the message when using persistent host storage, but they occur on startup, and iotedge is in a running and healthy state afterwards

[shaeussler]

Hi @bilalsellak, that's the only error message that I see in the logs. I'm persist edgeAgent and edgeHub storage on the host https://learn.microsoft.com/en-us/azure/iot-edge/how-to-access-host-storage-from-module?view=iotedge-1.5

[bilalsellak]

Thanks for linking that. What I am asking is if your edge device works as intended, even though you see that message on restart?

[shaeussler]

No, my edge device did not restart after the command "sudo iotedge config apply". If I delete the storage of the edgeAgent and edgeHub, then my edge device restart and work again.

[bilalsellak]

Hmm. I am unable to reproduce this following the same steps.

I can reproduce the error message, but from what I can tell, its expected due to using local storage and iotedge trying to access the old keys.

Other than that, iotedge is in a working state. Can you share ss of what modules are running and the iotedge check after doing
sudo iotedge system stop and then
sudo iotedge config apply

[RichardHubertPtw]

Hi Bilal, thanks for handling this. Here my observations whereby Siegfried (shaeussler) must provide conclusive information tomorrow during normal business hours.

1. Siegfried said he had to delete the (Rocks) DB directory contents on the OS to achieve a successful restart. Did you not have to do this?

2. We are unsure if we can just "trash" the DB files in every case to resolve the startup problem. Is there nothing that can go wrong by deleting those files? ... we don't know everything they are used for and we are using the iot-edge in both online and offline mode for significant periods. When we do the key refresh, we are online, of course.

3. So the error can be "officially ignored", in production, in this case?

Bottom line (from my perspecive) at the moment is: if you can tell us that, in production, the the error is "inert" and so is deleting the Rocks DB directories contents, then I guess we may have a way to achieve our deterministic key update.

Looking forward to clearing this up with you all. Sincere thanks.

[bilalsellak]

@RichardHubertPtw

1. No, I did not have to delete any local storage to have a successful restart
2. Host storage is used when the device is offline. Since when you refresh your keys, you're online, it would not cause any issues
3. Yes, from my testing and what the error looks to be from.

[shaeussler]

Hmm. I am also unable to reproduce the error again.
I tested my EST server Azure IoT Edge setup with provisioning IoT Edge devices at scale on Linux using X.509 certificates https://learn.microsoft.com/en-us/azure/iot-edge/how-to-provision-devices-at-scale-linux-x509?view=iotedge-1.5&tabs=individual-enrollment%2Cubuntu and provisioning IoT Edge devices at scale with a TPM on Linux https://learn.microsoft.com/en-us/azure/iot-edge/how-to-provision-devices-at-scale-linux-x509?view=iotedge-1.5&tabs=individual-enrollment%2Cubuntu and cannot reproduce the error. After testing the certificate renewal https://learn.microsoft.com/en-us/azure/iot-edge/tutorial-configure-est-server?view=iotedge-1.5#test-certificate-renewal I saw AZIOT_KEYS_RC_ERR_EXTERNAL errors in the logs but the iotedge system starts successful without deleting the local storage.