CA Certs included are Expired about freeipa-letsencrypt

[dtucny]

I'm not using these scripts, however, I do use LetsEncrypt with FreeIPA and noticed today when looking at monitoring that one of my IPA servers was only a few weeks away from it's certificate expiring. Looking into the cause revealed that it was due to the change in intermediate certs at LE and the FreeIPA tools not handling it all too well, with the ipa-server-certinstall command getting an error:
ipapython.admintool: ERROR: Peer's certificate issuer is not trusted (certutil: certificate is invalid: Peer's Certificate issuer is not recognized.
). Please run ipa-cacert-manage install and ipa-certupdate to install the CA certificate.

This server previously had a certificate issued by the X3 LE issuer, but, on it's most recent renewal, got an R3 issued certificate.

The commands from @Necronian should be enough to fix it, for now at least, but, there appear to be additional LE changes coming, so, you may want to add some of the additional certs too.

As mentioned by a few people above, https://letsencrypt.org/certificates/ gives details on what issuers are currently valid and which will be coming soon, based on this I've done the following:

wget -O /etc/ssl/`hostname -f`/isrgrootx1.pem  https://letsencrypt.org/certs/isrgrootx1.pem
wget -O /etc/ssl/`hostname -f`/isrg-root-x2.pem https://letsencrypt.org/certs/isrg-root-x2.pem
wget -O /etc/ssl/`hostname -f`/lets-encrypt-r3.pem https://letsencrypt.org/certs/lets-encrypt-r3.pem
wget -O /etc/ssl/`hostname -f`/lets-encrypt-e1.pem https://letsencrypt.org/certs/lets-encrypt-e1.pem
wget -O /etc/ssl/`hostname -f`/lets-encrypt-r4.pem https://letsencrypt.org/certs/lets-encrypt-r4.pem
wget -O /etc/ssl/`hostname -f`/lets-encrypt-e2.pem https://letsencrypt.org/certs/lets-encrypt-e2.pem
ipa-cacert-manage install /etc/ssl/`hostname -f`/isrgrootx1.pem
ipa-cacert-manage install /etc/ssl/`hostname -f`/isrg-root-x2.pem
ipa-cacert-manage install /etc/ssl/`hostname -f`/lets-encrypt-r3.pem
ipa-cacert-manage install /etc/ssl/`hostname -f`/lets-encrypt-e1.pem
ipa-cacert-manage install /etc/ssl/`hostname -f`/lets-encrypt-r4.pem
ipa-cacert-manage install /etc/ssl/`hostname -f`/lets-encrypt-e2.pem
ipa-certupdate

Where I keep my certs under /etc/ssl/hostname -f/, this gets and installs all the current and coming active and backup issuer certs from LE and is a manual step due to the changes.

I use acme.sh (https://acme.sh) for handling the cert renewal, with a renew-hook running the following contained in /root/bin/newcert.sh:

# combine the chain and the root CA as the fullchain misses the root into a pkcs12 cert as needed by ipa
openssl pkcs12 -export -out /etc/ssl/`hostname -f`/cert.p12 -inkey /etc/ssl/`hostname -f`/privkey.pem -in /etc/ssl/`hostname -f`/fullchain.pem -certfile /etc/ssl/`hostname -f`/isrgrootx1.pem -passout pass:
# install the new cert
ipa-server-certinstall -w -d /etc/ssl/`hostname -f`/cert.p12 --dirman-password='YOURPASSWORD' --pin=
# restart services using the cert
service httpd restart
service dirsrv@YOUR-REALM restart

The command to issue the cert using acme.sh being:

acme.sh --issue -d `hostname -f` -w /var/www/html --certpath /etc/ssl/`hostname -f`/cert.pem --keypath /etc/ssl/`hostname -f`/privkey.pem --capath /etc/ssl/`hostname -f`/ca.pem --fullchainpath /etc/ssl/`hostname -f`/fullchain.pem --renew-hook "/root/bin/newcert.sh"

[olemathias]

@jsievertde and @laurenegerton
If you still are having issues running ipa-certupdate with CERTIFICATE_VERIFY_FAILED

I was having the same issue and solved it by temporarily disable SSL checks. Please note that I have no idea if this will break something or even work for you. Only tested on CentOS 8, so very likely different paths on other systems. Use at own risk :)

In /usr/lib/python3.6/site-packages/ipalib/util.py:
Comment out:

# ctx.verify_mode = ssl.CERT_REQUIRED
# ctx.check_hostname = True
# ctx.load_verify_locations(cafile)

Add after the lines above:

ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

Run:

kinit adminuser
ipa-certupdate -v

Revert changes above and run ipa-certupdate to verify everything is ok!

[amohideen]

@afrothundaaaa can you kindly paste the updated code in here, please?

[rcritten]

I'm curious what you did as well. What errors did you see? I'm unable to reproduce it. From what I can tell the certificates in the repo match those at letsencrypt.org.

It may be related to the distro (or my system) already having the LE certificates available, I don't know.

[amohideen]

I am on Fedora 31
uname -a: 17.7.0 Darwin Kernel Version 17.7.0: Fri Oct 30 13:34:27 PDT 2020; root:xnu-4570.71.82.8~1/RELEASE_X86_64 x86_64

(freeipa package broken in the latest debian, so I could not use it)

It might be the distro as you suggested. To be honest, I did not check the code, I ran it blindly. I will read through the code over the weekend and see if I have to make changes to the directory structure within the code.

Also, according to https://letsencrypt.org/certificates/, they have now retired DSTRootCAX3.pem, the cross-signed root certificate.

maybe this need to be updated? or include or include a wget to download the correct certificates, inside the code for consistency?

I will check and update on it.

[afrothundaaaa]

Hello all. Sorry I wasn't getting notifications for this.

@amohideen - You are correct. I did just this by downloading the certificates manually, and replacing the names in the script. I actually modified this heavily to also tie in to acme.sh project so that you can do DNS API integration.

I would suggest the wget to pull the latest root certs and to not include them within the script itself if that were possible.

Thanks!

[Necronian]

Yes I rand the setup script and it worked, sort of, but then I could no longer sign in because of a cert error.

wget 'https://letsencrypt.org/certs/isrgrootx1.pem' 'https://letsencrypt.org/certs/lets-encrypt-r3.pem'
ipa-cacert-manage install isrgrootx1.pem -n isrgrootx1 -t C,,
ipa-certupdate -v
ipa-cacert-manage install lets-encrypt-r3.pem -n letsencryptr3 -t C,,
ipa-certupdate -v

And things appear to be working now.

Edit... I have no idea how to formatting

[afrothundaaaa]

@dtucny - this is perfect. While your additional renewal script for acme.sh isn't what i needed, the wget script is perfect and works with the freeipa-letsencrypt script.

I have made changes, including a QOL update to automatically replace the ipa-httpd.conf file with the FQDN of the server, and submitted a pull request.

Thanks everyone!

[jsievertde]

@dtucny

I seem to have a further problem as even after replacing the certs with "ipa-cacert-manage install"
I get the following error on centos8:

cannot connect to 'https://ldap.schunternet.de/ipa/json': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)

Might you have an Idea what's going wrong.
The Website I can visit and the correct cert is served but login is impossible.

[laurenegerton]

Same problem as @jsievertde - correct cert but login via web UI response is:
Invalid Authentication
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)

[dtucny]

@jsievertde Have you restarted the server? I'm unable to reproduce this on CentOS 8.

[jsievertde]

@dtucny
Yes I updated and restarted the server multiple times.
It seems to only be a local symptom as every connection that is from outside and asks for the cert and certifies it non locally is working.

Maybe something with the python3.6 version the IPA-Server is using in the background.
I will try to take some more time on Wednesday to find more out. As the only thing that's impacted for us right now is the automatic removal and addition of new users and we can live without it for the next few days.

Thanks for your time.

[amohideen]

The solution (worked for me) was (on Fedora 31): I manually installed the certificates.

1: ipa-cacert-manage install "$WORKDIR/ca/DSTRootCAX3.pem" -n DSTRootCAX3
2: ipa-certupdate -v
3: ipa-cacert-manage install "$WORKDIR/ca/LetsEncryptAuthorityX3.pem" -n letsencryptx3
4: ipa-certupdate -v

after this stage

I ran the /renew-le.sh (This will ask for the passphrase), In a different terminal/tab cd into /var/lib/ipa/passwds/
cat the password file and copied the passphrase text. back to the original terminal where it was asking for the password, I pasted it.

everything went smooth.

(This worked for me)

[jsievertde]

Okay my problem seems to be this:
After running ipa-healthcheck

{
    "source": "ipahealthcheck.ipa.certs",
    "check": "IPAOpenSSLChainValidation",
    "result": "ERROR",
    "uuid": "ce4f91f3-5369-4ceb-bac3-8bf9c1b8f83f",
    "when": "20210113130112Z",
    "duration": "0.018320",
    "kw": {
      "key": "/var/lib/ipa/certs/httpd.crt",
      "reason": "CN = ldap.schunternet.de\nerror 20 at 0 depth lookup: unable to get local issuer certificate\n",
      "msg": "Certificate validation for /var/lib/ipa/certs/httpd.crt failed: CN = ldap.schunternet.de\nerror 20 at 0 depth lookup: unable to get local issuer certificate\n"
    }

For me it seems like my freeipa-server didn't properly activate the let's encrypt ca-certs.

[rcritten]

@jsievertde its unclear what problem you're having as you mention local and non-local and it isn't clear which is working, or what sequence of commands you've run.

[jsievertde]

Okay I'm sorry.
We deploy the let's encrypt certificate with a script that is based on this repository.
Starting with the 6.01.2020 we got the following error while connecting to our instance:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 344, in connect
    ssl_context=context)
  File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 354, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket
    _context=self, _session=session)
  File "/usr/lib64/python3.6/ssl.py", line 776, in __init__
    self.do_handshake()
  File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='ldap.XXX.de', port=443): Max retries exceeded with url: /ipa/session/login_password (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/ipa_user_control.py", line 87, in <module>
    client.login('admin', config[CONFIG_IPA_SECTION]['password'])
  File "/usr/local/lib/python3.6/site-packages/python_freeipa/client.py", line 172, in login
    return self._login(self._host, username, password)
  File "/usr/local/lib/python3.6/site-packages/python_freeipa/client.py", line 193, in _login
    response = self._session.post(login_url, headers=headers, data=data, verify=self._verify_ssl)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 581, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='ldap.XXX.de', port=443): Max retries exceeded with url: /ipa/session/login_password (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))

We looked into the underlying issue and it looked like that we were missing the new ca certs from lets encrypt.
But even after installing we can't use the command ipa-certupdate

While contacting the Webserver from a second system works as expected and the certificate is valid.
Even if we use curl or wget from the cli of the ldap-server we can get a working response.

So for me it seems as IPA itself is unable to pick up the changes.

I'm sorry if my writing wasn't as informative as I intended it to be.

[Yukigamine]

I use acme.sh (https://acme.sh) for handling the cert renewal, with a renew-hook running the following contained in /root/bin/newcert.sh:

Thank you @dtucny! I have been looking around the internet for an example of someone using Acme.sh and trying to scrap together bits and pieces of what I found until I reached your example here!

I tweaked it a little to pull in the original httpd password like in renew-le.sh just because, but you got me up and running!

[laurenegerton]

@olemathias Thanks - that fix worked for us.

[pubyun]

@jsievertde and @laurenegerton If you still are having issues running ipa-certupdate with CERTIFICATE_VERIFY_FAILED

I was having the same issue and solved it by temporarily disable SSL checks. Please note that I have no idea if this will break something or even work for you. Only tested on CentOS 8, so very likely different paths on other systems. Use at own risk :)

In /usr/lib/python3.6/site-packages/ipalib/util.py: Comment out:

# ctx.verify_mode = ssl.CERT_REQUIRED
# ctx.check_hostname = True
# ctx.load_verify_locations(cafile)

Add after the lines above:

ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

Run:

kinit adminuser
ipa-certupdate -v

Revert changes above and run ipa-certupdate to verify everything is ok!

i have cert expired problem with ipa-server-4.9.6-10.

i modify /usr/lib/python3.6/site-packages/ipalib/util.py:

# ipa-certupdate
Connect error: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed (certificate has expired)
The ipa-certupdate command failed.