Internal hosted UC not supported about plugin-installation-manager-tool

[daniel-beck]

Depends; the cleaner solution would be an improvement to Juseppe IMO, while not relying on this file would be more compatible.

FWIW release-history is pretty terrible: It doesn't list more than one release of a given plugin per day, i.e. some releases are just not listed. So seems unsuitable beyond use as a data source for an automated Twitter account.

[timja]

https://stackoverflow.com/a/44605372/4951015

or pass JAVA_OPTS to the plugin cli which contain a custom truststore...
https://github.com/jenkinsci/docker/blob/master/jenkins-plugin-cli.sh#L3

[timja]

Ok the recommendation is to use a custom image:
https://github.com/jenkinsci/helm-charts/tree/main/charts/jenkins#consider-using-a-custom-image

that way you don't need the Jenkins project image to be up in order for your Jenkins to start, (assuming you're downloading plugins on startup)

[abdennour]

I went with another path. and now it works: Nexus + repos proxy + one repo groups all proxies under one + env vars.

export JENKINS_UC=http://nexus.cicd/repository/updates.jenkins.io/update-center.json
export JENKINS_UC_EXPERIMENTAL=http://nexus.cicd/repository/updates.jenkins.io/experimental/update-center.json
export JENKINS_PLUGIN_INFO=http://nexus.cicd/repository/updates.jenkins.io/plugin-versions.json
export JENKINS_INCREMENTALS_REPO_MIRROR=http://nexus.cicd/repository/repo.jenkins-ci.org/incrementals
export JENKINS_UC_DOWNLOAD=http://nexus.cicd/repository/all-jenkins-repos/download

This is works !
Thank you @timja

[timja]

Hi

You need to determine which truststore is being used by your Java installation and ensure it's updated with your internal cert, (or get a trusted cert via something like let's encrypt normally makes things a lot easier).

Here's a post about how to locate it: https://www.ibm.com/support/pages/how-identify-ssltls-keystore-and-truststore-being-used-java%E2%84%A2-application

[sagar-khanna]

Thanks @timja for a quick response.

I have it in my Dockerfile like this:

RUN jenkins-plugin-cli --verbose --jenkins-update-center https://<INTERNAL_FQDN>/repository/jenkins-plugins/update-center.json -f /usr/share/jenkins/plugins.txt

So do I need to use keytool to import my root ca into the Java's trust store inside the Jenkins docker image?

[timja]

Yes you do, or add your custom truststore to your image

[sagar-khanna]

I'll try keystore and get back shortly.

I have already tried adding custom trust store like below but it didn't work:
COPY <INTERNAL_ROOT_CA>.pem /usr/local/share/ca-certificates/ RUN chmod 644 /usr/local/share/ca-certificates/<INTERNAL_ROOT_CA>.pem && update-ca-certificates

Additionally is it possible to add a feature that sets a flag to the cli to ignore self-signed certificates?

[timja]

I would rather document the proper solution, so once you get it working would be good if you can report back

[sagar-khanna]

Still fails

Step 10/21 : RUN keytool -import -trustcacerts -alias "<REDACTED>" -file /usr/local/share/ca-certificates/<REDACTED>.pem -keystore /usr/local/openjdk-8/jre/lib/security/cacerts -storepass changeit -noprompt
 ---> Running in bd55ee69a176
Certificate was added to keystore
Removing intermediate container bd55ee69a176
 ---> 3f3d42dcd4ed
Step 11/21 : RUN keytool -list -keystore /usr/local/openjdk-8/jre/lib/security/cacerts -storepass changeit | grep -i <REDACTED>
 ---> Running in 6cce19a2fe76
<REDACTED>, Nov 16, 2020, trustedCertEntry,
Removing intermediate container 6cce19a2fe76

I'm out of ideas now :-(

Wouldnt it be a proper solution to add a flag to the jenkins-plugin-cli tool to ignore SSL warning for self-signed cert?

[timja]

have you verified that's the trust store actually in use?

you can also for using your own truststore as well.

[sagar-khanna]

Ignore the above error. The CA certificate was invalid, used the correct one and now its past that problem.

However now I'm stuck here:
Cache miss for: plugin-versions

Basically I'm not hosting this https://updates.jenkins.io/plugin-versions.json, I am using https://github.com/jenkinsci/juseppe to generate the release-history and update-centre json files and it doesn't support plugin-versions.json file at the moment.

Is there a way to skip experimental, incrementals and plugin info ?

[timja]

It looks like it should work if you override the configuration and point it at your update-center.json

is it not release-history.json btw? just looking at the docs?

[sagar-khanna]

No, in release-history.json the key is called release-history whereas for plugins.version its called plugins.
The content is also different, release-history gives info about the releases itself whereas plugins gives history of the plugin versions.

I just did an override pointing to the update-centre.json for all three URLs and now it fails because it can't find certain keys in the data:
org.json.JSONException: JSONObject["requiredCore"] not found.

[timja]

💩 any thoughts @oleg-nenashev / @daniel-beck, is it better to add to juseppe these files or should we handle that these files may not be there?

[sagar-khanna]

I copied the plugin-versions.json from update.jenkins.io and put it in our Nexus repo however still getting the exception (given below) for some plugins.
The exception doesn't provide the name of the plugin so I put the plugins list in a for loop and echo the plugin name before the install command however because it runs in async mode the output is all jumbled up so I can't really tell from 150+ plugins that I'm trying to install for which plugin requiredCore is missing in plugins-version.json if at all.

org.json.JSONException: JSONObject["requiredCore"] not found.
        at org.json.JSONObject.get(JSONObject.java:572)
        at org.json.JSONObject.getString(JSONObject.java:859)
        at io.jenkins.tools.pluginmanager.impl.PluginManager.getPluginDependencyJsonArray(PluginManager.java:629)
        at io.jenkins.tools.pluginmanager.impl.PluginManager.resolveDependenciesFromJson(PluginManager.java:742)
        at io.jenkins.tools.pluginmanager.impl.PluginManager.resolveDirectDependencies(PluginManager.java:794)
        at io.jenkins.tools.pluginmanager.impl.PluginManager.resolveRecursiveDependencies(PluginManager.java:831)
        at io.jenkins.tools.pluginmanager.impl.PluginManager.findPluginsAndDependencies(PluginManager.java:490)
        at io.jenkins.tools.pluginmanager.impl.PluginManager.start(PluginManager.java:153)
        at io.jenkins.tools.pluginmanager.impl.PluginManager.start(PluginManager.java:113)
        at io.jenkins.tools.pluginmanager.cli.Main.main(Main.java:63)
JSONObject["requiredCore"] not found.```

[timja]

I think we need to add this to Juseppe really

[sagar-khanna]

I think we need to add this to Juseppe really
Does that mean orgs not using juseppe but using this tool should work out a way to make sure these files are available?

Also would you like for me to raise an issue in the juseppe repo given that we should be generating these files there anyways?

We are kind of blocked from rolling out Jenkins using Docker at the moment because of this, I'll try to use a workaround but not 100% sure at the moment if it works.

[timja]

Please do raise the issue, you should be able to use the (deprecated) install-plugins.sh script for now.

[sagar-khanna]

Tried it, took me a while to get all the external requests pointing to Nexus.
Similar problem though with that script, SSL certificate was not getting picked up inspite of passing CURL_OPTS -k. I'll try again tomorrow and let you know.

In the meantime, will raise the issue in juseppe.

[abdennour]

When i was using the bash script (/usr/local/bin/install_plugins.sh ) everything was working fine because it understand CURL_OPTS='-k', however after using this tool, i cannot see a way to skip the ssl. :(

[timja]

When i was using the bash script (/usr/local/bin/install_plugins.sh ) everything was working fine because it understand CURL_OPTS='-k', however after using this tool, i cannot see a way to skip the ssl. :(

You need to add your cert to the truststore for your java installation or pass a custom truststore on the command line

[abdennour]

but the docker container of jenkins is running without root user.

jenkins@jenkins-staging-0:/$ export KEYSTORE_PASS=changeit
jenkins@jenkins-staging-0:/$ export KEYSTOREFILE=${JAVA_HOME}/jre/lib/security/cacerts
jenkins@jenkins-staging-0:/$ ${JAVA_HOME}/bin/keytool -import -noprompt -trustcacerts -alias jenkins-io -file /jenkins-certs/jenkinsio.crt -keystore ${KEYSTOREFILE} -storepass ${KEYSTORE_PASS}
Certificate was added to keystore
keytool error: java.io.FileNotFoundException: /usr/local/openjdk-8/jre/lib/security/cacerts (Permission denied)

i am now in miserable situation

[abdennour]

i understand that these 3 lines requires creating new image:

User root
RUN ... keystore
USER jenkins

However, i am avoiding creating new image even though we had auto delivery of all of our custom images.
Instead, we are using volume mounts from configmap which includes the keystool command + initContainers

[abdennour]

I like the second option:

or pass JAVA_OPTS to the plugin cli which contain a custom truststore...
https://github.com/jenkinsci/docker/blob/master/jenkins-plugin-cli.sh#L3

I am trying it now by using these docs

[abdennour]

Trying the second option

by running 2 commands:

KEYSTOREFILE=/tmp/jx.jks
${JAVA_HOME}/bin/keytool -import -noprompt -trustcacerts -alias jenkins-io -file /jenkins-certs/jenkinsio.crt -keystore ${KEYSTOREFILE} -storepass ${KEYSTORE_PASS}

## Then 
export JAVA_OPTS="$JAVA_OPTS -Djavax.net.ssl.keyStore=${KEYSTOREFILE} -Djavax.net.ssl.keyStoreType=jks -Djavax.net.ssl.keyStorePassword=changeit"

## Then
jenkins-plugin-cli --war "/usr/share/jenkins/jenkins.war" --verbose --plugins token-macro:2.12

I got this error now :

io.jenkins.tools.pluginmanager.impl.UpdateCenterInfoRetrievalException: Error getting update center json at io.jenkins.tools.pluginmanager.impl.PluginManager.getJson(PluginManager.java:606) at io.jenkins.tools.pluginmanager.impl.PluginManager.getUCJson(PluginManager.java:625) at io.jenkins.tools.pluginmanager.impl.PluginManager.start(PluginManager.java:150) at io.jenkins.tools.pluginmanager.impl.PluginManager.start(PluginManager.java:117) at io.jenkins.tools.pluginmanager.cli.Main.main(Main.java:76)

Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)

Notes

• Note that curl -O --cacert /jenkins-certs/jenkinsio.crt https://updates.jenkins.io/download/plugins/structs/1.20/structs.hpi works successfully.
  I mean we don't have doubt that /jenkins-certs/jenkinsio.crt is a right certificate.

• Note that the script pass JAVA_OPTS :

jenkins@jenkins-staging-0:/tmp$ cat /bin/jenkins-plugin-cli
#!/bin/bash

exec /bin/bash -c "java $JAVA_OPTS -jar /usr/lib/jenkins-plugin-manager.jar $*"

My question is :

• am i use it the right JAVA_OPTS (-Djavax.net.ssl.keyStore,... so on)
• What about the others (Djavax.net.ssl.trustStore , ... so on) ? should i put them ? if so, what are the values?

[timja]

Did you start from your system trust store, copy it to somewhere you can write it, import your cert on top of it?

[abdennour]

copy it to somewhere you can write it
i did cat ${JAVA_HOME}/jre/lib/security/cacerts > /tmp/mytrustted-stor
, import your cert on top of it?
how ?

If you could do to me a favor and list all steps including keytool command. seems that there is order and some options in cmds

[abdennour]

Ok ! so seems like i succeeded 👍 to do the right steps.
Credits to @timja then this link

But now , i got a different error 👎 when i installed any plugin with the tool :

Retrieving update center information
Cache entry expired
Cache miss for: update-center-2.263.3
io.jenkins.tools.pluginmanager.impl.UpdateCenterInfoRetrievalException: Error getting update center json
        at io.jenkins.tools.pluginmanager.impl.PluginManager.getJson(PluginManager.java:606)
        at io.jenkins.tools.pluginmanager.impl.PluginManager.getUCJson(PluginManager.java:625)
        at io.jenkins.tools.pluginmanager.impl.PluginManager.start(PluginManager.java:150)
        at io.jenkins.tools.pluginmanager.impl.PluginManager.start(PluginManager.java:117)                                                     at io.jenkins.tools.pluginmanager.cli.Main.main(Main.java:76)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)                                                          at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:607)
        at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:666)
        at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)                                                                         at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
        at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
        at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:268)                                   at java.net.URL.openStream(URL.java:1068)
        at org.apache.commons.io.IOUtils.toString(IOUtils.java:2795)
        at io.jenkins.tools.pluginmanager.impl.PluginManager.getJson(PluginManager.java:600)
        ... 4 more
Error getting update center json

[timja]

that's connecting to updates.jenkins.io