FW 3.2.7 breaks RADIUS certificates about ubios-cert

[ouaibe]

Allright, I "think" I got to the end of it by modifying the right key/values in the mongoDB. This seems to survive the System config reload, the service unifi restart and the full UDM hardware reboot.

⚠️ Again, I'm not responsible for you breaking your MongoDB, have some backups before!

The following one-liners are used to load the right certs in the right collection by updating existing values (this mongosh command looks for "key: radius" combo to upsert values in place -which creates them if not present, you might want to check that you only have one "key: radius" combo before ending up with more than one document):

mongo --quiet --port 27117 ace --eval "var certContent='$(base64 -w 0 /data/udapi-config/raddb/certs/server.pem)'; db.setting.findOneAndUpdate({key : \"radius\"}, {\$set: {'server_certificate': certContent}}, {upsert: true, returnNewDocument: true})"
mongo --quiet --port 27117 ace --eval "var certContent='$(base64 -w 0 /data/udapi-config/raddb/certs/server-key.pem)'; db.setting.findOneAndUpdate({key : \"radius\"}, {\$set: {'server_certificate_key': certContent}}, {upsert: true, returnNewDocument: true})"
mongo --quiet --port 27117 ace --eval "var certContent='$(base64 -w 0 /data/udapi-config/raddb/certs/ca.pem)'; db.setting.findOneAndUpdate({key : \"radius\"}, {\$set: {'ca_certificate': certContent}}, {upsert: true, returnNewDocument: true})"

One thing I haven't tested is if the other changes to the JSON files are necessary if this is done first. It might be the case that only this change to the MongoDB is needed ultimately.

Also, I'm not sure how this will fare with updates, Ubiquiti has already changed all certs w/o telling its users, so this might have to be adjusted with future updates of Unifi OS.

[ouaibe]

I know this is "late" in the discussion, but I can also confirm that the certs are being replaced by udapi-server starting process:

From journalctl -u udapi-server:

...
ubios-udapi-server[1467192]: svc-radius-server-service:      +(services):   Start deleted->running service radiusServer
ubios-udapi-server[1467192]: svc-radius-server-service: applying full configuration
ubios-udapi-server[1467192]: svc-radius-server-service: using provisioned server certificate
ubios-udapi-server[1467192]: svc-radius-server-service: using provisioned server key
ubios-udapi-server[1467192]: svc-radius-server-service: using provisioned CA certificate
ubios-udapi-server[1467192]: svc-radius-server-service: starting the service
...

And using inotifywait to monitor file changes in the certs directory:

/etc/freeradius/3.0/certs# inotifywait -mr .
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.


./ MODIFY server.pem
./ OPEN server.pem
./ MODIFY server.pem
./ CLOSE_WRITE,CLOSE server.pem
./ MODIFY server-key.pem
./ OPEN server-key.pem
./ MODIFY server-key.pem
./ CLOSE_WRITE,CLOSE server-key.pem
./ MODIFY ca.pem
./ OPEN ca.pem
./ MODIFY ca.pem
./ CLOSE_WRITE,CLOSE ca.pem
./ OPEN server.pem
./ ACCESS server.pem
./ CLOSE_NOWRITE,CLOSE server.pem
./ OPEN ca.pem
./ ACCESS ca.pem
./ CLOSE_NOWRITE,CLOSE ca.pem
./ OPEN ca.pem
./ ACCESS ca.pem
./ CLOSE_NOWRITE,CLOSE ca.pem
./ OPEN server-key.pem
./ ACCESS server-key.pem
./ CLOSE_NOWRITE,CLOSE server-key.pem
./ OPEN dh
./ ACCESS dh
./ CLOSE_NOWRITE,CLOSE dh
./ OPEN,ISDIR
./ ACCESS,ISDIR
./ CLOSE_NOWRITE,CLOSE,ISDIR

That being said, I'm unable to find the source of these certificates, nor which exact process does the replacement.

[ouaibe]

Allright, I found the solution (which I had in front of my eyes for a while, except I missed a layer of b64 encoding).

Turns out, with 3.2.7, the certificates (CA, Server, and a bunch of others) have been moved to the udapi-server configuration state/file in /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state. This is confirmed by the service being launched as /usr/bin/ubios-udapi-server -c /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state --silent.

The JSON file elements where this can be found are .services.radiusServer.caCertificate.crt .services.radiusServer.serverCertificate.crt and .services.radiusServer.serverCertificate.key but they are not in PEM format, they are Base64 encoded over their PEM format.

Upon service start, the certificates are taken from the ubios-udapi-server.state JSON file, decoded from Base64 and written to /etc/freeradius/3.0/certs/ as ca.pem server.pem and server-key.pem accordingly.

As such, it's possible to replace these entries in the JSON file and (probably?) make them persistent in this configuration.

Here are one-liners using jq examples to load your existing certificates into the JSON configuration file, provided that you already have a ca.pem (CA Certificate content in PEM format), server.pem (Server Certificate content in PEM format) and server-key.pem (Server Key Content in PEM format).

⚠️DO BACKUP YOUR JSON CONFIG FILE, I AM NOT RESPONSIBLE FOR YOU BREAKING YOUR UNIFI HARDWARE.⚠️

jq --arg filecontent "$(cat /data/udapi-config/raddb/certs/ca.pem)" '.services.radiusServer.caCertificate.crt = $filecontent | (.services.radiusServer.caCertificate.crt |= @base64)' /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state > /data/udapi-config/ubios-udapi-server/ubios-udapi-server.new.state ; mv /data/udapi-config/ubios-udapi-server/ubios-udapi-server.new.state /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state

jq --arg filecontent "$(cat /data/udapi-config/raddb/certs/server.pem)" '.services.radiusServer.serverCertificate.crt = $filecontent | (.services.radiusServer.serverCertificate.crt |= @base64)' /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state > /data/udapi-config/ubios-udapi-server/ubios-udapi-server.new.state ; mv /data/udapi-config/ubios-udapi-server/ubios-udapi-server.new.state /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state

jq --arg filecontent "$(cat /data/udapi-config/raddb/certs/server-key.pem)" '.services.radiusServer.serverCertificate.key = $filecontent | (.services.radiusServer.serverCertificate.key |= @base64)' /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state > /data/udapi-config/ubios-udapi-server/ubios-udapi-server.new.state ; mv /data/udapi-config/ubios-udapi-server/ubios-udapi-server.new.state /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state

Note that this is probably also true for the unifi-core certificates, but I haven't looked into it specifically.

So why did Ubiquiti make that change? I can't tel, but hopefully it is because eventually that JSON config file (that seem to hold the entirety of the UDM config) will be modified by the Web UI and allow the user to load their own certificates ? One can dream :)

That being said, forcing a new CA chain onto all their users without any early warning is certainly not professional, with many users complaining about connection refusals from Apple (and Android) devices that suddently get a certificate change (for another untrusted one, mind you) https://community.ui.com/questions/UniFi-OS-3-2-7-CA-Certificate-changes-break-Radius-backed-802-1x-authentication/9a5d4fb9-f791-4bec-81b5-3b9cc0c7853a#answer/23a7c99a-e1f4-410f-95fb-670b4fb426d3

[alxwolf]

@ouaibe Great analysis!

Will have a look into the web server certificate in the other issue e#62 - the JSON contains another cert, suitable for web server, which is (again) the standard self-signed cert by UI (at least on my machine).

So far, restarting unifi-core did not break the web certificate. And I can only support what you write in your final paragraph...

but, maybe this is just a small step we have to overcome on the path to having it managed automatically.

[Tijn1974]

Allright, I found the solution (which I had in front of my eyes for a while, except I missed a layer of b64 encoding).

Turns out, with 3.2.7, the certificates (CA, Server, and a bunch of others) have been moved to the udapi-server configuration state/file in /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state. This is confirmed by the service being launched as /usr/bin/ubios-udapi-server -c /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state --silent.

The JSON file elements where this can be found are .services.radiusServer.caCertificate.crt .services.radiusServer.serverCertificate.crt and .services.radiusServer.serverCertificate.key but they are not in PEM format, they are Base64 encoded over their PEM format.

Upon service start, the certificates are taken from the ubios-udapi-server.state JSON file, decoded from Base64 and written to /etc/freeradius/3.0/certs/ as ca.pem server.pem and server-key.pem accordingly.

As such, it's possible to replace these entries in the JSON file and (probably?) make them persistent in this configuration.

Here are one-liners using jq examples to load your existing certificates into the JSON configuration file, provided that you already have a ca.pem (CA Certificate content in PEM format), server.pem (Server Certificate content in PEM format) and server-key.pem (Server Key Content in PEM format).

⚠️DO BACKUP YOUR JSON CONFIG FILE, I AM NOT RESPONSIBLE FOR YOU BREAKING YOUR UNIFI HARDWARE.⚠️

jq --arg filecontent "$(cat /data/udapi-config/raddb/certs/ca.pem)" '.services.radiusServer.caCertificate.crt = $filecontent | (.services.radiusServer.caCertificate.crt |= @base64)' /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state > /data/udapi-config/ubios-udapi-server/ubios-udapi-server.new.state ; mv /data/udapi-config/ubios-udapi-server/ubios-udapi-server.new.state /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state

jq --arg filecontent "$(cat /data/udapi-config/raddb/certs/server.pem)" '.services.radiusServer.serverCertificate.crt = $filecontent | (.services.radiusServer.serverCertificate.crt |= @base64)' /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state > /data/udapi-config/ubios-udapi-server/ubios-udapi-server.new.state ; mv /data/udapi-config/ubios-udapi-server/ubios-udapi-server.new.state /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state

jq --arg filecontent "$(cat /data/udapi-config/raddb/certs/server-key.pem)" '.services.radiusServer.serverCertificate.key = $filecontent | (.services.radiusServer.serverCertificate.key |= @base64)' /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state > /data/udapi-config/ubios-udapi-server/ubios-udapi-server.new.state ; mv /data/udapi-config/ubios-udapi-server/ubios-udapi-server.new.state /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state

Note that this is probably also true for the unifi-core certificates, but I haven't looked into it specifically.

So why did Ubiquiti make that change? I can't tel, but hopefully it is because eventually that JSON config file (that seem to hold the entirety of the UDM config) will be modified by the Web UI and allow the user to load their own certificates ? One can dream :)

That being said, forcing a new CA chain onto all their users without any early warning is certainly not professional, with many users complaining about connection refusals from Apple (and Android) devices that suddently get a certificate change (for another untrusted one, mind you) https://community.ui.com/questions/UniFi-OS-3-2-7-CA-Certificate-changes-break-Radius-backed-802-1x-authentication/9a5d4fb9-f791-4bec-81b5-3b9cc0c7853a#answer/23a7c99a-e1f4-410f-95fb-670b4fb426d3

All thanks @ouaibe . This did solve the problem for me.

[ouaibe]

I am running into the exact same issue, also not directly using that script but I referred to it for debug since it was said to be compatible with 3.2.7.

I have been looking into it for a few hours now, to no avail. Something is overwriting the certs served by freeradius in /etc/freeradius/3.0/certs/*.pem with the same exact certificates (Ca/Server) every time udapi-server service is restarted.

The certificates in /data/udapi-config/raddb/certs are ignored and the UDM serves its own cert and CA instead of the manually generated one. It also doesn't matter whether the custom created CA cert is part of the platform trusted certs (for instance manually added via update-ca-certificates).

I'm wondering how that script gets around this and manages to get its LE-backed certificates served persistently instead for EAP?

[alxwolf]

@ouaibe just checked my logs and most recent cert update on my box was before updating it to 3.2.7. Updated the README to reflect that there seems to be an issue with 3.2.7 Early Access.

Will need to sift through that, any hints appreciated.

[therealpaulgg]

official UI thread for their 3.2.7 release, I posted a comment but havent heard anything back

https://community.ui.com/releases/UniFi-OS-Dream-Router-3-2-7/bdc93e02-3f71-4196-8553-46664c605e13

[alxwolf]

official UI thread for their 3.2.7 release, I posted a comment but havent heard anything back

yes, thanks, just saw that.

On your question for downgrade - not sure if that works (for you...), but switching to the official firmware from EA may bridge it for your case?

And I'm still on the fence if the suggestion to put in a feature request for this was malevolent or uninformed: such requests exist since ages and this repo will be archived the minute I see it implemented in this "semi-pro" product...

[alxwolf]

@therealpaulgg just info: Running systemctl restart unifi-core did not overwrite the /data/unifi-core/config/unifi-core.* files (web interface) for me.

@ouaibe no chance yet to test RADIUS as I don't use it right now.

[therealpaulgg]

See that's the unfortunate thing - I am only on official. I never participated in early access - but they've pushed it to us regardless. I dont even have a different release channel as an option

interesting on your restart not changing files...maybe there's a solution for us.

edit: I've been using SSH to do all this management...you use Web interface? not even sure how that works

[ouaibe]

@alxwolf thanks for checking. On my end, nothing in /data/ got overwritten, it's really only the certs in freeradius's config (/etc/freeradius/...) that did.
I am not an EA user: 3.2.7 for UDM-Pro's is on the Official channel now.

[alxwolf]

@therealpaulgg - UniFi has three places for certs, "Web Interface" being one (which requires restarting unifi-core for updating) - all done by SSH.

@ouaibe and @therealpaulgg: you're both using an internal CA - a while ago, I experienced that UniFiOS will remove invalid certificates and overwrite those with default self-signed ones.

openssl x509 -in /data/udapi-config/raddb/certs/server.pem -text -noout should show a full certificate chain for RADIUS, openssl x509 -in /data/unifi-core/config/unifi-core.crt -text -noout (for web interface) as well.

Would you mind checking your certs, if they are actually valid and include a full chain up to a trusted root certificate? (and that's where the black box is with me not knowing which root certs are trusted by UniFiOS...)

Hypothesis right now is that somehow the UDM "does not like" your certs and by that overwrites them.

[ouaibe]

I thought this could be a possibility, but note that it worked flawlessly for years prior to 3.2.7 and I didn't change anything before this update.

The irony is that my certs validate using openssl verify but Unifi's do not (which is confirmed by the fact that Android complains about them).

Here's my cert chain validation (note that I installed my CA in the local cert store using update-ca-certs and confirmed it's loaded as part of the other CAs in /etc/ssl/certs):

$openssl verify -show_chain -verbose /data/udapi-config/raddb/certs/server.pem

/data/udapi-config/raddb/certs/server.pem: OK
Chain:
depth=0: <SERVER_CERT_INFO>
depth=1: <CA_CERT_INFO>

$openssl crl2pkcs7 -nocrl -certfile /data/udapi-config/raddb/certs/server.pem | openssl pkcs7 -print_certs -noout
subject=<SERVER_CERT_INFO>
issuer=C = <CA_CERT_INFO>

Here's Unifi's cert chain validation:

$openssl verify -show_chain -verbose /etc/freeradius/3.0/certs/server.pem

C = USA, ST = Utah, L = Salt Lake City, O = ui.com, CN = standard.ui.com
error 20 at 0 depth lookup: unable to get local issuer certificate
error /etc/freeradius/3.0/certs/server.pem: verification failed

Which is probably because the CA isn't in the local store, but we can specify the CA manually using -untrusted:

 openssl verify -show_chain -verbose -untrusted /etc/freeradius/3.0/certs/ca.pem /etc/freeradius/3.0/certs/server.pem
C = USA, ST = Utah, L = Salt Lake City, O = ui.com, CN = UI Certificate Authority
error 19 at 1 depth lookup: self signed certificate in certificate chain
error /etc/freeradius/3.0/certs/server.pem: verification failed

$openssl crl2pkcs7 -nocrl -certfile /etc/freeradius/3.0/certs/server.pem | openssl pkcs7 -print_certs -noout
subject=C = USA, ST = Utah, L = Salt Lake City, O = ui.com, CN = standard.ui.com
issuer=C = USA, ST = Utah, L = Salt Lake City, O = ui.com, CN = UI Certificate Authority

Note that I had backed up the "older" certificates prior to the 3.2.7 upgrade, and they definitely changed their cert chain with that update (i.e. they did "something).

Here's the info on the previous one:

$openssl verify -show_chain -verbose -untrusted /data/udapi-config/raddb/certs/ca.pem /data/udapi-config/raddb/certs/server.pem
C = US, ST = New York, L = New York, O = Ubiquiti Inc., CN = UbiOS RADIUS Certificate Authority
error 19 at 1 depth lookup: self signed certificate in certificate chain
error /data/udapi-config/raddb/certs/server.pem: verification failed

$openssl crl2pkcs7 -nocrl -certfile /data/udapi-config/raddb/certs/server.pem | openssl pkcs7 -print_certs -noout
subject=C = US, ST = New York, O = Ubiquiti Inc., CN = UbiOS RADIUS Server Certificate
issuer=C = US, ST = New York, L = New York, O = Ubiquiti Inc., CN = UbiOS RADIUS Certificate Authority

I also thought that maybe they had now put their root CA in their known keystore and were re-generating from this upon udapi server boot, but alas the contents/fingerprint (and generation dates) don't match.

keytool -v -list -keystore /data/unifi/data/keystore -storepass aircontrolenterprise
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: unifi
Creation date: Dec 15, 2022
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=UniFi, OU=UniFi, O=Ubiquiti Inc., L=New York, ST=New York, C=US
Issuer: CN=UniFi, OU=UniFi, O=Ubiquiti Inc., L=New York, ST=New York, C=US
Serial number: 639b54d1
Valid from: Thu Dec 15 12:09:37 EST 2022 until: Wed Mar 19 13:09:37 EDT 2025
Certificate fingerprints:
         SHA1:  <REDACTED>
         SHA256: <REDACTED>
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  serverAuth
]

#2: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
  DNSName: UniFi
]

It's very possible that their concept of "invalid" (causing the replacement) morphed into "it's not ours", but it's hard to tell because whatever validation they're doing is completely opaque.

[alxwolf]

So, seems some more people noticed it.

https://community.ui.com/questions/UniFi-OS-3-2-7-CA-Certificate-changes-break-Radius-backed-802-1x-authentication/9a5d4fb9-f791-4bec-81b5-3b9cc0c7853a

[therealpaulgg]

my internal CA is as follows:

Root CA
Intermediate CA
Router certificate

the router certificate is signed by my intermediate CA.

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            83:a3:ae:9a:21:e4:00:4c:d2:89:5a:99:f9:41:fb:ad
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: O = PaulGG Intranet, CN = PaulGG Intranet Intermediate CA
        Validity
            Not Before: Dec 14 12:04:01 2023 GMT
            Not After : Dec 15 12:05:01 2023 GMT
        Subject: CN = router.paulgg.int
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:a2:dc:42:d7:35:b3:dd:5b:96:86:29:4c:64:e7:
                    32:48:1c:0a:bb:e6:5f:43:ad:63:de:19:12:3e:3f:
                    a5:52:f3:20:1e:30:23:c4:ac:56:aa:d8:b8:5d:ae:
                    d8:fe:6b:e8:ce:c5:b6:e6:c8:20:36:e0:98:35:3f:
                    b5:c3:d3:51:cc:91:ca:1d:38:a7:c8:9c:9e:7b:9a:
                    5e:41:e1:5b:38:7e:b8:7d:6e:a3:d7:51:11:b9:c7:
                    1c:be:f7:44:b2:a9:f5:63:1a:f3:43:64:ea:11:f1:
                    97:4b:6b:a6:e4:95:e1:05:98:45:4f:63:8b:ae:ac:
                    8d:aa:7d:9b:a5:d8:b8:1d:61:5e:c1:9b:3a:d8:91:
                    45:57:a0:d6:b7:1c:d4:88:4e:01:e5:c3:61:b4:6d:
                    68:1d:c8:2c:00:9a:e8:fb:0c:76:f1:f0:b8:d2:22:
                    3b:35:c8:9c:b9:80:db:7b:d9:ee:b3:af:b0:0d:fc:
                    70:b8:ef:0b:f6:4c:39:b5:c8:d0:9c:86:b3:71:a1:
                    a5:42:26:f7:3e:82:4c:23:a0:84:67:f6:0a:a9:34:
                    eb:ac:f2:f6:bc:a5:f5:6e:1a:34:e8:2a:a6:7e:42:
                    90:28:b9:db:34:1d:0d:b7:a1:1a:65:d3:61:55:ad:
                    ec:52:0a:fe:f5:11:41:1d:21:12:25:fc:02:e8:4d:
                    9d:cd
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Authority Key Identifier:
                keyid:58:1E:16:42:18:5A:CF:A5:DA:29:2E:37:50:AF:4E:5D:1B:CE:B9:93

            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Key Identifier:
                98:41:E0:9B:1A:F8:23:F4:8F:6C:D7:53:37:A4:4E:D6:79:D3:A9:57
            X509v3 Subject Alternative Name:
                DNS:router.paulgg.int
            1.3.6.1.4.1.37476.9000.64.1:
                [email protected].+V5B126r4pfQJE_PkrMvSvc_gFds2wEs5EhzVSdSF-dQ
    Signature Algorithm: ecdsa-with-SHA256
         30:45:02:21:00:be:81:ea:60:62:19:71:3e:0b:54:48:43:79:
         56:5f:b5:59:23:dc:21:1b:94:16:07:a6:74:f6:5a:24:ff:af:
         43:02:20:25:29:02:8e:64:0f:01:b0:4e:b8:3d:47:8c:c5:b1:
         f1:80:46:86:fd:48:55:df:65:fb:af:fb:2e:24:eb:8f:a9

I just checked /etc/ssl/certs and looks like there's some stuff that has been deleted, including my root certificate from the keystore.

When I have some more time I'll play around with this...

[alxwolf]

Well, it seems to be a somewhat bigger mess created by UI (with production systems losing WiFi connection according to statement in UI Community linked above.

If that's the case, I'd expect the release to be pulled at least, if not rolled back.

Glenn knows a lot to everything about the certificate story in UniFi OS, I hope he will address the topic. And as X-Mas is approaching - maybe this leads to a well-documented and automated, GUI based approach. Or not.

[therealpaulgg]

Added my intermediate CA in to my root keystore, verified with openssl verify. Still wasnt happy - and reset my certs.

Yeah hopefully UI team addresses. Not sure what else I can do for now.

[alxwolf]

@ouaibe may I suggest you address UI-Glenn directly as he explicitly offers it here

https://community.ui.com/releases/UniFi-OS-Dream-Machines-3-2-7/251dfc1e-f4dd-4264-a080-3be9d8b9e02b#comment/0c1a969e-5196-4ee5-8b37-7a6ee730e841

[ouaibe]

@alxwolf thanks, I have already posted in that thread and I'm not seeing an offer from UI-Glenn to exchange on that problem specifically, but I'll look into it.

[ouaibe]

Thanks to you for all the hard work you did put into helping users side-load their own certs, it really is appreciated! 🤗

[ouaibe]

Just to finalize this, it seems that after some Unifi platform reboot happened, the .state file was reset to its original values.

It is therefore my guess that the real cold config file is /data/udapi-config/udapi-net-cfg.json and that's what's must be modified so that the certificates survives full reboots.

⚠️ Beware though this could mean that they are part of the cloud backup of Unify (and so you're sending your keys to their cloud).

In order to address that, you need to proceed with the same modification of that file:

jq --arg filecontent "$(cat /data/udapi-config/raddb/certs/ca.pem)" '.services.radiusServer.caCertificate.crt = $filecontent | (.services.radiusServer.caCertificate.crt |= @base64)' /data/udapi-config/udapi-net-cfg.json > /data/udapi-config/udapi-net-cfg.new.json ; mv /data/udapi-config/udapi-net-cfg.new.json /data/udapi-config/udapi-net-cfg.json

jq --arg filecontent "$(cat /data/udapi-config/raddb/certs/server.pem)" '.services.radiusServer.serverCertificate.crt = $filecontent | (.services.radiusServer.serverCertificate.crt |= @base64)' /data/udapi-config/udapi-net-cfg.json > /data/udapi-config/udapi-net-cfg.new.json ; mv /data/udapi-config/udapi-net-cfg.new.json /data/udapi-config/udapi-net-cfg.json

jq --arg filecontent "$(cat /data/udapi-config/raddb/certs/server-key.pem)" '.services.radiusServer.serverCertificate.key = $filecontent | (.services.radiusServer.serverCertificate.key |= @base64)' /data/udapi-config/udapi-net-cfg.json > /data/udapi-config/udapi-net-cfg.new.json ; mv /data/udapi-config/udapi-net-cfg.new.json /data/udapi-config/udapi-net-cfg.json

After that, you can validate that both files are the same:

diff <(jq -r .services.radiusServer.serverCertificate.key /data/udapi-config/udapi-net-cfg.json) <(jq -r .services.radiusServer.serverCertificate.key /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state)

diff <(jq -r .services.radiusServer.serverCertificate.crt /data/udapi-config/udapi-net-cfg.json) <(jq -r .services.radiusServer.serverCertificate.crt /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state)

diff <(jq -r .services.radiusServer.caCertificate.crt /data/udapi-config/udapi-net-cfg.json) <(jq -r .services.radiusServer.caCertificate.crt /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state)

[ouaibe]

I thought I had it but there's some other circumstance where both of these files get overwritten with a Unifi certificate (e.g. changing the Network setting and causing a re-provision of APs/the UDM). There must be some other place where these are stored, still investigating.

[alxwolf]

One thing bugging me is that the UniFi Easy Let's Encrypt script by Glenn Rietveld (what a coincidence, his name is Glenn... right?) still uses the well-known places (before 3.2.7) when updating the RADIUS certs.

[ouaibe]

Indeed, and there's something going on with MongoDB: it also stores the certificates as well. I haven't had the opportunity to go through it all (I'm not super skilled in NoSQL DBs), but I think this might be the source for the replacement when the UDM restarts on its own (let's say you enable/disable a Wifi network, that triggers the issue on my end).

What I recently found out is that I have some weirdnesses in the logs when the UDM restarts, something seems to not like the JSON config files and causes a "re-migration/re-init", but I have zero idea why that is, and I tried to sync udapi-net-cfg.json and ubios-udapi-server.state for the problematic sections but to no avail).

ubios-udapi-server[1333054]: config-migrate-helper: Trying to migrate config due to inconsistency (invalid config .versionDetail: must be between 22 and 22: .interfaces) to resolve following issue: invalid config: configuration syntax is invalid: is required: .services.radiusServer.policyGroups.type
ubios-udapi-server[1333054]: config-migrate-helper: Starting config .versionFormat 'v2' migration for /tmp/udapi-fastapply-<REDACTED>.cfg.tmp
<BUNCH_OF_CONF_MIGRATE_HELPER_SCRIPTS>
ubios-udapi-server[1333054]: config-migrate-helper: Migrated config is valid

I looked quickly into the udapi_server.py and its use of the config migration scripts and why it might have been complaining, but it seems to be an endless rabbit hole of managing config versions over different OS releases...

The Glenn Rietveld (no clue if that's a naming coïncidence) scripts are also doing something with MongoDB. I have posted in that thread in the Ubiquiti Community forums but didn't get any response...

I'll continue drilling into mongodb (using mongo --port 27117 the interesting stuff seems to be in the ace DB, e.g. mongo --quiet --port 27117 ace --eval "db.getCollection('setting').find({}).forEach(printjson);" | grep -i certificate) later during the holiday break, as I'd like to get to the bottom of this, but the in-line replacement in the JSON files clearly isn't resilient.