SSL: CERTIFICATE_VERIFY_FAILED with newest version (0.8.0a13) about pyaarlo

[twrecked]

The cloudflare message might be because the code now uses a random user agent by default. Which obviously isn't working so well. Try passing user_agent="arlo" to PyArlo to restore the original behavior.

The certs failing is more interesting. Can you try this command from the same environment?

openssl s_client -connect mqtt-cluster.arloxcld.com:443

You should see something like this, the Verify return code is the important piece:

SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 8CD7896F7927073364701294F3CF9AF951020C70634DC002F68788D55317BFE4
    Session-ID-ctx: 
    Master-Key: A4191C8A52053C5299EE0B319C7BF289C32293F3A1D345CE7C3F78DB0377D398B7407F8E5D6D819AB83104F81D2FCC50
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1638193735
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---
DONE

I don't do anything special for the SSL connections in MQTT but one to try might be to comment this line out of backend.py.

self._ev_client.tls_set_context(ssl.create_default_context())

[m0urs]

openssl output is this:

depth=3 C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=2 C = US, O = Amazon, CN = Amazon Root CA 1
verify return:1
depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
verify return:1
depth=0 CN = mqtt-cluster.arloxcld.com
verify return:1
CONNECTED(00000003)
---
Certificate chain
 0 s:CN = mqtt-cluster.arloxcld.com
   i:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
 1 s:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
   i:C = US, O = Amazon, CN = Amazon Root CA 1
 2 s:C = US, O = Amazon, CN = Amazon Root CA 1
   i:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
 3 s:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
   i:C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFqDCCBJCgAwIBAgIQC2pJRMKLYlskflqRg3SebjANBgkqhkiG9w0BAQsFADBG
MQswCQYDVQQGEwJVUzEPMA0GA1UEChMGQW1hem9uMRUwEwYDVQQLEwxTZXJ2ZXIg
Q0EgMUIxDzANBgNVBAMTBkFtYXpvbjAeFw0yMTAzMTcwMDAwMDBaFw0yMjA0MTUy
MzU5NTlaMCQxIjAgBgNVBAMTGW1xdHQtY2x1c3Rlci5hcmxveGNsZC5jb20wggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDgMdNvpuGMxKR99+ISzzUMlysK
8gHbjB8jcnI3qxaE5dMNA9GlgXcSZSosB/qJzgT+q6od1PKvnJyTp11XkDRhKtCu
LrFVZT3KSYMHYigJbPE9OcwGh6hyXo8uWOmOP6Mh5M6hrxXe37wL4Q3KB45ugBMb
CuHhR8oYa/8yXgYiv57zb0JAatmEb6p3AGDm8VLdPTHoocPCoNWRr9LjN1m0NES+
pMtHxhf/CdjfdtUZpaF1ZeF/FsqWznTxL2TWbzio8EuJ2i4/iMJauicb4Epbg6zU
qXU4tHw2OVOt4kh4rVVW2sqkgO3xBksaP92GvU8mL35phmr6KhLg+/39W5hxAgMB
AAGjggKyMIICrjAfBgNVHSMEGDAWgBRZpGYGUqB7lZI8o5QHJ5Z0W/k90DAdBgNV
HQ4EFgQULCDbF+U7glHh6eoPNTdPQqfqqJQwYAYDVR0RBFkwV4IZbXF0dC1jbHVz
dGVyLmFybG94Y2xkLmNvbYIcbXF0dC1jbHVzdGVyLXoyLmFybG94Y2xkLmNvbYIc
bXF0dC1jbHVzdGVyLXoxLmFybG94Y2xkLmNvbTAOBgNVHQ8BAf8EBAMCBaAwHQYD
VR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMDsGA1UdHwQ0MDIwMKAuoCyGKmh0
dHA6Ly9jcmwuc2NhMWIuYW1hem9udHJ1c3QuY29tL3NjYTFiLmNybDATBgNVHSAE
DDAKMAgGBmeBDAECATB1BggrBgEFBQcBAQRpMGcwLQYIKwYBBQUHMAGGIWh0dHA6
Ly9vY3NwLnNjYTFiLmFtYXpvbnRydXN0LmNvbTA2BggrBgEFBQcwAoYqaHR0cDov
L2NydC5zY2ExYi5hbWF6b250cnVzdC5jb20vc2NhMWIuY3J0MAwGA1UdEwEB/wQC
MAAwggECBgorBgEEAdZ5AgQCBIHzBIHwAO4AdQBGpVXrdfqRIDC1oolp9PN9ESxB
dL79SbiFq/L8cP5tRwAAAXhBaroWAAAEAwBGMEQCICUl5gJ/B944q9lw/X4So/91
JqUNBWVg/10wq6PftrwHAiBo1PRuTGNU3Mv2cPH05bHJhqcKWTpn0SGBpQD9uQxn
wAB1ACJFRQdZVSRWlj+hL/H3bYbgIyZjrcBLf13Gg1xu4g8CAAABeEFquhcAAAQD
AEYwRAIgGTGPqhUmZ1iDZybjqPu7AAGl0uF3irdAtKubpbVALygCIDh4MpfYaWdl
SnABSd4Jqnmh75sDHb6hQyX675+J3mPpMA0GCSqGSIb3DQEBCwUAA4IBAQAGIhf4
8s+PMbzYwTLxQ5+r9K0opJEt3WtllQy0f0i+/hnhRo5v6l72wnNe5+ugMpnJPw3Y
W6KzOmWXlg+6wnhKFOuhZ7CLpfiTop2ZC/uO10rqmoSGBr17NFM27N70iflvy6eg
4yNy3sNx9rmL94TWdN1p2F5V779hWy95XaQHy4VeaD0hbiFgRhFeKP+AgmX6IVhW
hbR78QNAOBNPFfWHjr/OA12tiiuHqO5d1IUitokFt2cEZE5grpAxI1SP5IJduRT+
TfyT7uBoegztjG8iEohQb5Bmqr57rruV12jwCgTe4BCXm8ef9id5sRJm9WCKmFpX
BusM6vAPWRuQoZDW
-----END CERTIFICATE-----
subject=CN = mqtt-cluster.arloxcld.com

issuer=C = US, O = Amazon, OU = Server CA 1B, CN = Amazon

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 5390 bytes and written 453 bytes
Verification error: unable to get local issuer certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 82A1D89B67ED9C3CCD8B04FE50DD95E7EBB08CEA745DBD25ECCB0ABD031717BD
    Session-ID-ctx: 
    Master-Key: 8AE1E923E8C08B94BE3FF7F23350473227A1ACA6BBB02EA7811ACF0BEC1B53B94D88BF16C27847737033DF566688AC95
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1638201262
    Timeout   : 7200 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
    Extended master secret: no
---
HTTP/1.1 400 Bad Request
connection: close
content-length: 0

closed

[m0urs]

self._ev_client.tls_set_context(ssl.create_default_context())

This seems to solve the SSL verification issue ...

[twrecked]

And interestingly, removing that line breaks it for me...

Your openssl verify failed - why did you get this Verify return code: 20 (unable to get local issuer certificate)?

Can you try it with another website? And can you try running it this way and pasting trace.txt to here?

strace openssl s_client -connect mqtt-cluster.arloxcld.com:443 2> trace.txt

[m0urs]

Other websites seem to have the same issue. If I add this parameter, then all websites including the arlo host, are verifiying correct:

openssl s_client -connect mqtt-cluster.arloxcld.com:443 -CApath /etc/ssl/certs

This behaviour of openssl is described on several articles in the Internet. Maybe one need to add that path somewhere in Python as well?

Included is the strace output with and without the CAPath parameter.

trace_with.txt
trace_without.txt

[twrecked]

I have this inside /usr/lib/ssl so it's all linking back to the same place.

ha-pyaarlo>pyaarlo$ ls -l /usr/lib/ssl/
total 4
drwxr-xr-x 2 root root 4096 Nov 18 07:59 misc
lrwxrwxrwx 1 root root   14 Nov 10  2015 certs -> /etc/ssl/certs
lrwxrwxrwx 1 root root   20 Aug 24 21:13 openssl.cnf -> /etc/ssl/openssl.cnf
lrwxrwxrwx 1 root root   16 Nov 10  2015 private -> /etc/ssl/private

What does this give you?

openssl version  -d

And what debian are you using? I'll start an image here and try the code.

[m0urs]

On that machine I am using Debian Stretch (9.13).

 # openssl version  -d
OPENSSLDIR: "/usr/local/ssl"

I have build a newer version of openssl, that is the reason why it points to "/usr/local":

 # openssl version
OpenSSL 1.1.1l  24 Aug 2021

# ls -l /usr/lib/ssl/
total 4
lrwxrwxrwx 1 root root   14 Mar 29  2018 certs -> /etc/ssl/certs
drwxr-xr-x 2 root root 4096 Mar  6  2021 misc
lrwxrwxrwx 1 root root   20 Feb 18  2021 openssl.cnf -> /etc/ssl/openssl.cnf
lrwxrwxrwx 1 root root   16 Mar 29  2018 private -> /etc/ssl/private

Aahhhh. I think I see the problem. Just recreated my "certs" and "private" directory within the new ssl directory as a symbolic link:

/usr/local/ssl # ll
total 44K
lrwxrwxrwx 1 root staff   14 Nov 29 19:19 certs -> /etc/ssl/certs
drwxr-sr-x 2 root staff 4.0K Oct 18 17:47 certs.org
-rw-r--r-- 1 root staff  412 Oct 18 17:47 ct_log_list.cnf
-rw-r--r-- 1 root staff  412 Oct 18 17:47 ct_log_list.cnf.dist
drwxr-sr-x 2 root staff 4.0K Oct 18 17:47 misc
-rw-r--r-- 1 root staff  11K Oct 18 17:47 openssl.cnf
-rw-r--r-- 1 root staff  11K Oct 18 17:47 openssl.cnf.dist
lrwxrwxrwx 1 root staff   16 Nov 29 19:20 private -> /etc/ssl/private
drwxr-sr-x 2 root staff 4.0K Oct 18 17:47 private.org

And now it seems that the cert is verified correctly!

[m0urs]

Just did a quick test with the new code on this Debian machine and it seems to connect now again ... So I think this really solved it. Sorry, this was my fault and thanks for giving me a hint in the right direction ;-)

[twrecked]

No worries. Glad it's going now. Let me know how the MQTT goes, I feel like the client is more robust than the SSE one so hopefully it will be better.

It's also a lot easier to debug because Chrome will display the response packets correctly.

[m0urs]

Let me know how the MQTT goes

Yes, I will give it a try in the next days and will report back. Especially as I had some issues in the past weeks after the connection has been established for some hours. Hopefully this will be solved as well.