Bug (or "undocumented feature?") in dependency resolution using `--latest true` about plugin-installation-manager-tool

[gpaciga]

As someone who just spent yesterday trying to understand this, option 3 is what I expected the behaviour to be before I learned that --latest was required to have pinned versions respected. Given that I can set myplugin:latest if I wanted to always use the latest, it does not make sense that myplugin:1.2.3 also effectively means latest by default.

My expectation is:

• if a dependency is not listed, use latest
• if a dependency is listed, use whatever is pinned
• if a pinned version is lower than something another plugin requires, throw an error and let me sort it out. ("pluginX requires pluginY version B but you have specified version C")

That last point is similar to what currently happens, except the error would be because I pinned something incompatible, not because the tool chose to ignore my pinned versions.

The only case where I would want to pin something meaning "version X or higher" without meaning "latest" is if I want to accept patch or minor updates only (e.g. like npm where pinning ~1.2.3 means 1.2.4 is acceptable but 1.3 is not).

[haminhcong]

I encountered similiar isssues. Today I need build a custom Jenkins Docker Image from base Docker Image jenkins/jenkins:2.289.1-jdk11, with jenkins-plugin-cli tool to install some Jenkin plugins to Jenkins Docker Image:

installPlugins:
   - kubernetes:1.29.4
   - workflow-aggregator:2.6
   - git:4.7.1
   - configuration-as-code:1.51
   - kubernetes-client-api:4.13.2-1

If I use option --latest true, then Dockerfile will like that:

FROM jenkins/jenkins:2.289.1-jdk11
RUN jenkins-plugin-cli --verbose --latest true --plugins \
        kubernetes-client-api:4.13.2-1 kubernetes:1.29.4 workflow-aggregator:2.6 git:4.7.1 configuration-as-code:1.51 

then with the same behavior with PR: https://github.com/jenkinsci/plugin-installation-manager-tool/pull/325/files the Docker build process failed, due to I pinned kubernetes-client-api: plugin version to 4.13.2-1

If I use option --latest false, then Dockerfile will like that:

FROM jenkins/jenkins:2.289.1-jdk11
RUN ls -alh
RUN jenkins-plugin-cli --verbose --latest false --plugins \
         kubernetes-client-api:4.13.2-1 kubernetes:1.29.4 workflow-aggregator:2.6 git:4.7.1 configuration-as-code:1.51 

Then Docker build process succeed. But when I inspect installed plugins versions in build log, It showed that some installed plugins versions is incorrect:

https://pastebin.com/zcnmBmB4

$ docker build -f Dockerfile .
Sending build context to Docker daemon  8.192kB
Step 1/4 : FROM jenkins/jenkins:2.289.1-jdk11
 ---> 416c6656c1cd
Step 2/4 : RUN ls -alh
 ---> Using cache
 ---> d80f607455a9
Step 3/4 : RUN jenkins-plugin-cli --verbose --plugins kubernetes-client-api:4.13.2-1 kubernetes:1.29.4 workflow-aggregator:2.6 git:4.7.1 configuration-as-code:1.51 --latest false
 ---> Running in 560577e701c3
No .txt or .yaml file containing list of plugins to be downloaded entered.

Will install new plugin kubernetes 1.29.4
Will install new plugin pipeline-milestone-step 1.3.1
Will install new plugin snakeyaml-api 1.27.0
Will install new plugin trilead-api 1.0.13
Will install new plugin workflow-multibranch 2.20
Will install new plugin docker-workflow 1.14
Will install new plugin handlebars 1.1
Will install new plugin pipeline-stage-tags-metadata 1.3.2
Will install new plugin git-client 3.7.1
Will install new plugin variant 1.4
Will install new plugin scm-api 2.6.4
Will install new plugin kubernetes-client-api 4.13.2-1
Will install new plugin pipeline-stage-step 2.3
Will install new plugin pipeline-model-extensions 1.7.2
Will install new plugin configuration-as-code 1.51
Will install new plugin workflow-cps-global-lib 2.11
Will install new plugin workflow-support 2.20
Will install new plugin mailer 1.32.1
Will install new plugin jackson2-api 2.12.1
Will install new plugin pipeline-rest-api 2.10
Will install new plugin workflow-scm-step 2.12
Will install new plugin pipeline-model-api 1.3.2
Will install new plugin lockable-resources 2.3
Will install new plugin pipeline-input-step 2.8
Will install new plugin metrics 4.0.2.6
Will install new plugin momentjs 1.1
Will install new plugin git-server 1.7
Will install new plugin credentials-binding 1.13
Will install new plugin workflow-aggregator 2.6
Will install new plugin kubernetes-credentials 0.8.0
Will install new plugin pipeline-stage-view 2.10
Will install new plugin pipeline-build-step 2.7
Will install new plugin credentials 2.3.15
Will install new plugin pipeline-model-definition 1.3.2
Will install new plugin caffeine-api 2.9.1-23.v51c4e2c879c8
Will install new plugin workflow-step-api 2.23
Will install new plugin plain-credentials 1.7
Will install new plugin matrix-project 1.18
Will install new plugin bouncycastle-api 2.18
Will install new plugin docker-commons 1.14
Will install new plugin jquery-detached 1.2.1
Will install new plugin ace-editor 1.0.1
Will install new plugin git 4.7.1
Will install new plugin icon-shim 1.0.3
Will install new plugin authentication-tokens 1.4
Will install new plugin workflow-durable-task-step 2.22
Will install new plugin ssh-credentials 1.18.1
Will install new plugin cloudbees-folder 6.15
Will install new plugin durable-task 1.35
Will install new plugin workflow-basic-steps 2.11
Will install new plugin workflow-job 2.25
Will install new plugin jsch 0.1.55.2
Will install new plugin pipeline-graph-analysis 1.1
Will install new plugin junit 1.3
Will install new plugin structs 1.22
Will install new plugin apache-httpcomponents-client-4-api 4.5.13-1.0
Will install new plugin workflow-cps 2.89
Will install new plugin display-url-api 2.3.1
Will install new plugin script-security 1.76
Will install new plugin pipeline-model-declarative-agent 1.1.1
Will install new plugin branch-api 2.0.18
Will install new plugin workflow-api 2.41

In above plugins versions, workflow-cps 2.89 depends on: https://github.com/jenkinsci/workflow-cps-plugin/blob/1768bff98453bfd94e889874dfe8eea0eef1a4cb/pom.xml#L72

Plugin-Dependencies: 
...
workflow-api:2.41
workflow-scm-step:2.11
workflow-step-api:2.23
workflow-support:3.7
...

and workflow-aggregator:2.6 depends on: https://github.com/jenkinsci/workflow-aggregator-plugin/blob/b28dc6c72545a28561ffee43188b983d72594ec7/pom.xml#L81

Plugin-Dependencies: 
...
workflow-multibranch 2.20
workflow-scm-step 2.6
workflow-step-api 2.16
workflow-support 2.20
...

If plugin version resolve correctly, workflow-support plugin must be resolved to at least version 3.7. But with option --latest false, jenkins-plugin-cli resloved this plugin to version 2.20 as build log.

Will install new plugin workflow-support 2.20

due to this behavior, when I start Docker Image after build, Jenkins cannot started successfully:

2021-06-19 13:03:00.072+0000 [id=28]    SEVERE    jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading plugin Pipeline: Groovy v2.89 (workflow-cps)
java.io.IOException: Failed to load: Pipeline: Groovy (2.89)
 - Update required: JavaScript GUI Lib: ACE Editor bundle plugin (1.0.1) to be updated to 1.1 or higher
 - Update required: Pipeline: Supporting APIs (2.20) to be updated to 3.7 or higher
    at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:963)
    at hudson.PluginManager$2$1$1.run(PluginManager.java:550)
    at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
    at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
    at jenkins.model.Jenkins$5.runTask(Jenkins.java:1129)
    at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
    at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

Compare with install-plugins.sh, workflow-support plugins version is resolved to compatible version with all depended plugins. With following plugins.txt file and Dockerfile:

kubernetes-client-api:4.13.2-1 
kubernetes:1.29.4 
workflow-aggregator:2.6 
git:4.7.1 
configuration-as-code:1.51

FROM jenkins/jenkins:2.289.1-jdk11
RUN ls -alh /usr/share/jenkins/ref/plugins
COPY plugins.txt /usr/share/jenkins/plugins.txt
RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/plugins.txt

When build image process run, plugins versions is resolved like that:

$ docker build --no-cache -f Dockerfile .
Sending build context to Docker daemon   7.68kB
Step 1/4 : FROM jenkins/jenkins:2.289.1-jdk11
 ---> 416c6656c1cd
Step 2/4 : COPY plugins.txt /usr/share/jenkins/plugins.txt
 ---> 852bd8a627ad
Step 3/4 : RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/plugins.txt
 ---> Running in 2ecf28e3df3f
WARN: install-plugins.sh is deprecated, please switch to jenkins-plugin-cli
Creating initial locks...
Analyzing war /usr/share/jenkins/jenkins.war...
#...

WAR bundled plugins:

Installed plugins:
ace-editor:1.1
apache-httpcomponents-client-4-api:4.5.13-1.0
authentication-tokens:1.4
bootstrap4-api:4.6.0-3
bootstrap5-api:5.0.1-2
bouncycastle-api:2.20
branch-api:2.6.4
caffeine-api:2.9.1-23.v51c4e2c879c8
checks-api:1.7.0
cloudbees-folder:6.15
configuration-as-code:1.51
credentials-binding:1.25
credentials:2.5
display-url-api:2.3.5
durable-task:1.37
echarts-api:5.1.2-2
font-awesome-api:5.15.3-3
git-client:3.7.2
git:4.7.1
git-server:1.9
handlebars:3.0.8
jackson2-api:2.12.3
jquery3-api:3.6.0-1
jsch:0.1.55.2
junit:1.50
kubernetes-client-api:4.13.2-1
kubernetes-credentials:0.8.0
kubernetes:1.29.4
lockable-resources:2.11
mailer:1.34
matrix-project:1.19
metrics:4.0.2.8
momentjs:1.1.1
pipeline-build-step:2.13
pipeline-graph-analysis:1.11
pipeline-input-step:2.12
pipeline-milestone-step:1.3.2
pipeline-model-api:1.8.5
pipeline-model-definition:1.8.5
pipeline-model-extensions:1.8.5
pipeline-rest-api:2.19
pipeline-stage-step:2.5
pipeline-stage-tags-metadata:1.8.5
pipeline-stage-view:2.19
plain-credentials:1.7
plugin-util-api:2.3.0
popper2-api:2.5.4-2
popper-api:1.16.1-2
scm-api:2.6.4
script-security:1.77
snakeyaml-api:1.29.1
ssh-credentials:1.19
structs:1.23
trilead-api:1.0.13
variant:1.4
workflow-aggregator:2.6
workflow-api:2.45
workflow-basic-steps:2.23
workflow-cps-global-lib:2.20
workflow-cps:2.92
workflow-durable-task-step:2.39
workflow-job:2.41
workflow-multibranch:2.26
workflow-scm-step:2.13
workflow-step-api:2.23
workflow-support:3.8
Cleaning up locks
Removing intermediate container 2ecf28e3df3f
 ---> 634042187e2f

An example, install-plugins.sh resloved workflow-support to version 3.8, which is a compatible version with both workflow-cps 2.89 and workflow-aggregator:2.6

[timja]

#308 (comment)

no I think it should allow you to use that version, it should only selecting latest of transitive dependencies if it's defined on the top level AND is compatible then it should select what you have defined

[timja]

A limitation with option 1 is that when a new dependency is added to a plugin it won’t get updated until someone notices that it’s missing from their plugin list

I still think 3 is the most sane option...

(I don’t have a lot of time for this repo and to get something fixed the quickest way is to send a PR)

[timja]

above was merged

[baptistemesta]

I encountered the same issue. I thought at first that some plugin did really require a higher version of one of my plugin but I then discovered that it was due to the "latest" option.

We update our Jenkins through a versioned/automated build. Given that, we fixed the version of plugins for reproducibility concerns.

So it was a little confusing to have that behavior by default for plugins we explicitly set the version.

[MarkEWaite]

I was also confused by the message that was displayed when I had enumerated each plugin version explicitly. I ran the command:

$ rm -rf plugins
$ java -jar ../jenkins-plugin-manager-2.9.0.jar \
        --war ../jenkins-2.277.1.war \
        --plugins "git:4.6.0 git-client:3.6.0" \
        --plugin-download-directory plugins
Plugin git:4.6.0 depends on git-client:3.7.0, but there is an older version defined on the top level - git-client:3.6.0,

I was confused that the message says "Plugin git:4.6.0 depends on git-client:3.7.0" when I know that git plugin 3.6.0 depends on git client plugin 3.6.0, not git client plugin 3.7.0. In my case, it would have helped if the phrasing of the message made it clear that the explicitly declared dependency on git-client:3.6.0 was being upgraded to git-client:3.7.0 because I had not provided the command line argument --latest false

I had the desired results when I added --latest false to the command line like this:

$ rm -rf plugins
$ java -jar ../jenkins-plugin-manager-2.9.0.jar \
        --war ../jenkins-2.277.1.war \
        --plugins "git:4.6.0 git-client:3.6.0" \
        --plugin-download-directory plugins \
        --latest false

Would it be enough to rework the user error message to say:

Plugin git:4.6.0 depends on git-client.  Using git-client:3.7.0 because --latest is true, but there is an older version defined on the top level - git-client:3.6.0,

[pwillis-els]

Since multiple people have experienced this bug (i've talked to a few more), can we come up with some possible fixes and have one implemented?

1. Don't change anything, but clearly mark out the new behavior. The behavior here is clearly different than the old Jenkins plugin install scripts, so I think this should be explained prominently in the docs for this tool. This is basically a breaking behavior in certain circumstances (when you want to use pinned versions and not the latest ones).

2. Change the plugin manager's default to be --latest false, and instead add a new default option --minimum-valid true (or another name). This option would use the latest minimum version number to satisfy all dependency requirements, when a version was not explicitly pinned. Thus the user can choose which versions of what plugin to pin, and the tool will fill in any gaps with the minimum required functionality. If the user needs the latest of everything, they can use --latest true.

3. Modify the behavior of --latest true to respect pinned versions and not try to upgrade a plugin version if the user has already pinned a different version. (That is to say, when resolving dependencies, if a pinned version forces a dependency to use a version other than latest, and nothing is pinned for that dependency, then do not use latest, but instead the minimum resolvable version that does not override the dependency trees of any pinned plugins. This would basically be more like --maximum-valid true, and the plugin manager may have to do more work to discover the latest versions of dependencies that don't conflict with pinned dependencies' graphs)

Thoughts?

[MarkEWaite]

I prefer option 1, describe the behavior more precisely so that users understand that they have a choice with plugin installation manager that was not available with the plugin installation shell scripts.

In those cases where the precise versions of all plugins are specified, the --latest false argument provides the desired result.

[pwillis-els]

Until someone can submit a PR to provide solution 3, I have opened the following PR for solution 1. #325 @timja can we get this merged in?

There should really be a better long-term solution, but I think a stop-gap is warranted so users can figure out their issue quickly.

[mangitkaur]

Hi , on marking latest as false, I am still getting dependencies issue, can you please help?

[MarkEWaite]

Hi , on marking latest as false, I am still getting dependencies issue, can you please help?

Best to open a separate issue with precise details that describe the problem you are seeing. Most of the problems that I've seen with dependencies have been due to users providing an incomplete list of plugins.