Jenkins is unable to contact update center about build

[targos]

Still a problem after #3637 and a full system package update + reboot.

[richardlau]

I took a backup of /etc/ssl/certs/java/cacerts in /tmp/javahttpsclient/backup and ran:

(from https://askubuntu.com/a/1030380)

sudo dpkg --purge --force-depends ca-certificates-java
sudo apt-get install ca-certificates-java

This has replaced /etc/ssl/certs/java/cacerts:

# ls -al /etc/ssl/certs/java/cacerts
-rw-r--r-- 1 root root 160820 Mar 11 13:53 /etc/ssl/certs/java/cacerts
#

which still isn't the same size as the other Ubuntu 22.04 systems (🤷) but does allow java JavaHttpsClient https://updates.jenkins.io/update-center.json 1 to connect successfully (🎉).

I've restarted Jenkins on ci.nodejs.org so that Java picks up the refreshed certificates and the warning has gone away on https://ci.nodejs.org/manage/pluginManager/advanced:

[targos]

There were OS updates, including one for the JRE, so I applied them all and rebooted.
The problem is still present.

@nodejs/build-infra any idea?

[sxa]

Interesting. Do you know if this message started appearing just after the update to 2.440?

(I'm going to be performing a similar update on another server tomorrow!)

[targos]

It happened before the update to 2.440.1.

[richardlau]

It looks like it's Java related somehow.

On ci.nodejs.org:

curl -sL https://updates.jenkins.io/update-center.json

works.

Using the example simple Java HTTPS Client from https://access.redhat.com/solutions/973783 (note I had to recompile the client with javac JavaHttpsClient.java as I couldn't directly run the class file from the archive):

java JavaHttpsClient https://updates.jenkins.io/update-center.json 1

or (with debug info)

java -Djavax.net.debug=all JavaHttpsClient https://updates.jenkins.io/update-center.json 1

fails.

On ci-release.nodejs.org both curl and java JavaHttpsClient succeed.

FWIW I've placed the JavaHttpsClient in /tmp/javahttpsclient on both machines.

[richardlau]

ci.nodejs.org:

root@infra-digitalocean-ubuntu14-x64-1:~# lsb_release -d
Description:    Ubuntu 22.04.4 LTS
root@infra-digitalocean-ubuntu14-x64-1:~# java -version
openjdk version "17.0.10" 2024-01-16
OpenJDK Runtime Environment (build 17.0.10+7-Ubuntu-122.04.1)
OpenJDK 64-Bit Server VM (build 17.0.10+7-Ubuntu-122.04.1, mixed mode, sharing)
root@infra-digitalocean-ubuntu14-x64-1:~#

vs ci-release.nodejs.org:

root@infra-ibm-ubuntu1804-x64-1:~# lsb_release -d
Description:    Ubuntu 20.04.6 LTS
root@infra-ibm-ubuntu1804-x64-1:~# java -version
openjdk version "17.0.10" 2024-01-16
OpenJDK Runtime Environment (build 17.0.10+7-Ubuntu-120.04.1)
OpenJDK 64-Bit Server VM (build 17.0.10+7-Ubuntu-120.04.1, mixed mode, sharing)
root@infra-ibm-ubuntu1804-x64-1:~#

Given that curl worked on both machines, I don't think it should make a difference, but ci-release is on IBM Cloud whereas ci.nodejs.org is Digital Ocean.

[richardlau]

Examining the debug output
ci.nodejs.org:

javax.net.ssl|DEBUG|10|main|2024-03-08 13:09:40.048 EST|TrustStoreManager.java:162|Inaccessible trust store: /usr/lib/jvm/java-17-openjdk-amd64/lib/security/jssecacerts
javax.net.ssl|DEBUG|10|main|2024-03-08 13:09:40.049 EST|TrustStoreManager.java:113|trustStore is: /usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts
trustStore type is: pkcs12
trustStore provider is:
the last modified time is: Tue Jun 20 16:36:33 EDT 2023
javax.net.ssl|DEBUG|10|main|2024-03-08 13:09:40.049 EST|TrustStoreManager.java:334|Reload the trust store
javax.net.ssl|DEBUG|10|main|2024-03-08 13:09:40.127 EST|TrustStoreManager.java:342|Reload trust certs
javax.net.ssl|DEBUG|10|main|2024-03-08 13:09:40.130 EST|TrustStoreManager.java:347|Reloaded 173 trust certs

ci-release.nodejs.org:

javax.net.ssl|DEBUG|10|main|2024-03-08 18:08:59.022 UTC|TrustStoreManager.java:162|Inaccessible trust store: /usr/lib/jvm/java-17-openjdk-amd64/lib/security/jssecacerts
javax.net.ssl|DEBUG|10|main|2024-03-08 18:08:59.023 UTC|TrustStoreManager.java:113|trustStore is: /usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts
trustStore type is: pkcs12
trustStore provider is:
the last modified time is: Fri May 26 06:13:49 UTC 2023
javax.net.ssl|DEBUG|10|main|2024-03-08 18:08:59.023 UTC|TrustStoreManager.java:334|Reload the trust store
javax.net.ssl|DEBUG|10|main|2024-03-08 18:08:59.112 UTC|TrustStoreManager.java:342|Reload trust certs
javax.net.ssl|DEBUG|10|main|2024-03-08 18:08:59.116 UTC|TrustStoreManager.java:347|Reloaded 129 trust certs

Both try for an inaccessible (file isn't there /usr/lib/jvm/java-17-openjdk-amd64/lib/security/jssecacerts) before falling back to /usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts. ci has loaded more certificates (173) than ci-release (129).

ci.nodejs.org:

root@infra-digitalocean-ubuntu14-x64-1:/tmp/javahttpsclient# ls -al /usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts
lrwxrwxrwx 1 root root 27 Jan 17 06:09 /usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts -> /etc/ssl/certs/java/cacerts
root@infra-digitalocean-ubuntu14-x64-1:/tmp/javahttpsclient# ls -al /etc/ssl/certs/java/cacerts
-rw-r--r-- 1 root root 206799 Jun 20  2023 /etc/ssl/certs/java/cacerts
root@infra-digitalocean-ubuntu14-x64-1:/tmp/javahttpsclient#

ci-release.nodejs.org:

root@infra-ibm-ubuntu1804-x64-1:/tmp/javahttpsclient# ls -al /usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts
lrwxrwxrwx 1 root root 27 Jan 17 11:09 /usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts -> /etc/ssl/certs/java/cacerts
root@infra-ibm-ubuntu1804-x64-1:/tmp/javahttpsclient# ls -al /etc/ssl/certs/java/cacerts
-rw-r--r-- 1 root root 150808 May 26  2023 /etc/ssl/certs/java/cacerts
root@infra-ibm-ubuntu1804-x64-1:/tmp/javahttpsclient#

[sxa]

Yeah it looks like it's related to ubuntu system stuff. It seems to work ok with Temurin's 17.0.10 (and 21 - I've extracted both under your directory if you want to test).

[richardlau]

Interesting. Do you know if this message started appearing just after the update to 2.440?

This issue was opened on 19 Feb. If I look at the certificates in my web browser for https://updates.jenkins.io/current/update-center.json

it looks like the current certificate was issued on 17 Feb -- my suspicion as to what has triggered this is https://letsencrypt.org/2023/07/10/cross-sign-expiration.html which had this happen earlier in the month that would change the default certificate chain:

On Thursday, Feb 8th, 2024, we stopped providing the cross-sign by default in requests made to our /acme/certificate API endpoint. For most Subscribers, this means that your ACME client will configure a chain which terminates at ISRG Root X1, and your webserver will begin providing this shorter chain in all TLS handshakes. The longer chain, terminating at the soon-to-expire cross-sign, will still be available as an alternate chain which you can configure your client to request.

The JavaHttpsClient debug output shows both machines getting a two certificate chain.

[richardlau]

ci.nodejs.org:

root@infra-digitalocean-ubuntu14-x64-1:/tmp/javahttpsclient# ls -al /usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts
lrwxrwxrwx 1 root root 27 Jan 17 06:09 /usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts -> /etc/ssl/certs/java/cacerts
root@infra-digitalocean-ubuntu14-x64-1:/tmp/javahttpsclient# ls -al /etc/ssl/certs/java/cacerts
-rw-r--r-- 1 root root 206799 Jun 20  2023 /etc/ssl/certs/java/cacerts
root@infra-digitalocean-ubuntu14-x64-1:/tmp/javahttpsclient#

The size of /etc/ssl/certs/java/cacerts on ci.nodejs.org differs from other Ubuntu 22.04 machines in our CI:

$ ssh test-digitalocean-ubuntu2204-x64-1 ls -al /etc/ssl/certs/java/cacerts
-rw-r--r-- 1 root root 156996 Nov  6 12:45 /etc/ssl/certs/java/cacerts
$ ssh test-digitalocean-ubuntu2204-x64-2 ls -al /etc/ssl/certs/java/cacerts
-rw-r--r-- 1 root root 156996 Nov  6 12:48 /etc/ssl/certs/java/cacerts
$ ssh ci ls -al /etc/ssl/certs/java/cacerts
-rw-r--r-- 1 root root 206799 Jun 20  2023 /etc/ssl/certs/java/cacerts
$

[targos]

Amazing work, thank you!