jenkinsci / plugin-compat-tester

Jenkins Plugin Compatibility Tester

Plugin Compatibility Tester (PCT)

Generates a compatibility matrix for plugins against Jenkins core.

Running PCT

The PCT CLI requires passing a WAR file containing plugins (generated from jenkinsci/bom, for example) as input; the versions of the plugins are inferred from the WAR file contents.

java -jar target/plugins-compat-tester-cli.jar \
    --war "$(pwd)/megawar.war" \
    --working-dir "$(pwd)/pct-work"

To test a subset of plugins in the WAR, use --include-plugins:

java -jar target/plugins-compat-tester-cli.jar \
    --war "$(pwd)/megawar.war" \
    --include-plugins ssh-slaves,credentials \
    --working-dir "$(pwd)/pct-work"

You can run the CLI with the --help argument to get a full list of supported options.

Running PCT with custom Java versions

PCT simply invokes Maven, which relies on the JAVA_HOME environment variable. If you want to use a custom Java version, set JAVA_HOME appropriately before running PCT.

Useful links

• Devoxx '11 BOF Presentation about Plugin Compat Tester is available here. This presentation is partially obsolete (GAE feature was removed in recent versions)

plugin-compat-tester's Issues

Remove `pubsub-light-plugin` workaround

Pending the release of jenkinsci/pubsub-light-plugin#100 the pubsub-light-plugin workaround in cloneFromScm should be removed.

Remove Kubernetes credentials workarounds

Pending the release of jenkinsci/kubernetes-credentials-plugin#37 and jenkinsci/kubernetes-credentials-provider-plugin#75 the Kubernetes credentials workarounds in cloneFromScm should be removed.

`-localCheckoutDir` does not work with multi-module projects

(The plugin-compat-tester component cannot be used in Jira anymore, so I guess GitHub issues are preferred?)

The -localCheckoutDir option does not work with multi-module projects even when there is an implementation of AbstractMultiParentHook for the plugin in question. Specifically, MultiParentCompileHook.isSnapshotMultiParentPlugin returns false, so the default process-test-classes goal is used instead of MultiParentCompileHook changing the goal to install, which fails (JENKINS-62658 describes the issue with process-test-classes more generally).

The problem seems to be that AbstractMultiParentHook intentionally doesn't do anything when localCheckoutDir is used, which means that these properties never get specified, so the properties are null here and the multi-module-specific logic gets skipped.

See #383 (comment) for some additional discussion specific to workflow-cps.

JaCoCo hook no longer runs

Jenkins and plugins versions report

Environment

Plugin hooks are not run as the code is checking incorrect data (pom) that is not fully populated.

The maven model is a minimal representation of the actual pom and is both brittle and incomplete.

As mentioned in #486 (comment)

What Operating System are you using (both controller, and any agents involved in the problem)?

N/A

Reproduction steps

create a war with containing jacoco

run the PCT Tests

Expected Results

all jacoco tests pass (and the jacoco hook is run correctly)

Actual Results

the tests fail as the jacoco hook is not excersized and does not add jacoco:prepare-agent to the mvn arguments when adding tests.

ExecutionFileLoaderTest.testLoadBundleWithExecFile and a few others require a jacoco file which is provided when running the tests.

Anything else?

possibly affects all other plugins as well that are not using incrementals (that would inline the parent groupId) as well as potential plugins that are but are to be tested locally

--local-checkout-dir runs tests twice for plugins in the war

Jenkins and plugins versions report

Environment

Paste the output here

What Operating System are you using (both controller, and any agents involved in the problem)?

all

Reproduction steps

• create a war which includes a plugin (e.g. warnings-ng)
• clone the sources for the plugin to a local directory
• run the pct and specify the local checkout directory (java -jar target\plugins-compat-tester-cli.jar test-plugins --war mywar.war` --working-dir=C:\workarea\temp\ --include-plugins=warnings-ng --local-checkout-dir=C:\workarea\source\github\jenkinsci\warnings-ng-plugin\

Expected Results

the tests for warnings-ng are only run from the local checkout

Actual Results

tests are run for both the version of warnings-ng in the war and the version in the local clone

Anything else?

I introduced this issue in #510, previously the behaviour was defined if you provided a single plugin id to be included in the command line, but did not work for multi-module checkouts.

I expect using --local-checkout-dir to use that checkout for the list of plugins to test, and ignore anything plugins in the war, as I expect this to be used mainly by developers either
a) trying to validate a fix to an issue is working (or even reproduce a specific issue)
b) test a plugin that is not in the war

Remove unused command-line parameters

There are two known programmatic consumers of this deliverable (one proprietary, one open-source). A number of command-line arguments are consumed by neither. If they are also not used by any non-programmatic (i.e., human) consumers, then they represent dead code (i.e., a maintenance burden) that could be deleted.

Remove `hudson.model.UpdateSite.Plugin`

hudson.model.UpdateSite.Plugin and org.jenkins.tools.test.model.PluginInfos are value objects that contain largely the same data. Once #427 is implemented the former could likely be trivially deleted in favor of the latter for a simpler and more consistent object model.

Delete plugin selection modes other than `-war`

jenkinsci/bom and @cloudbees both run PCT using the -war option. To simplify the code base, we should delete all other run modes—checking the update center (including both explicit -updateCenterUrl and the default UC), -parentCoordinates (whatever that does), -bom (#202), probably -overridenPlugins [sic] (#88).

As of this writing, bom uses

• -war
• -includePlugins
• -workDirectory
• -reportFile
• -m2SettingsFile
• -mavenProperties
• -excludeHooks org.jenkins.tools.test.hook.TransformPom (see #387)
• -mavenPropertiesFile
• -skipTestCache

CloudBees also uses

• -storeAll
• -fallbackGitHubOrganization
• -mvn
• -hookPrefixes
• -externalHooksJars
• -testJDKHome
• -localCheckoutDir
• -includePlugins

CC @jtnord

Opt-in for Maven Failsafe plugin is broken

PluginWithFailsafeIntegrationTestsHook was meant to be a way for plugins to opt-in to having their Maven Failsafe plugin tests executed, but the class is not marked as abstract. Since

plugin-compat-tester/src/main/java/org/jenkins/tools/test/model/hook/PluginCompatTesterHooks.java

Line 186 in 0838e05

if (!Modifier.isAbstract(c.getModifiers())) {

executes all non-abstract hooks, PluginWithFailsafeIntegrationTestsHook gets executed unconditionally, resulting in workarounds like https://github.com/jenkinsci/bom/blob/c72362731c98313189c04025c3b7bc181da4c121/pct.sh#L22-L25. The correct solution is to mark PluginWithFailsafeIntegrationTestsHook as abstract, which will allow the abovementioned workaround to be removed but will also like expose other plugins which should be explicitly opting into Failsafe but aren't.

Remove `bootstrap5-api`, `data-tables-api`, and `echarts-api` workarounds

These workarounds are only necessary for the BOM 2.375.x line, which should be removed at some point in the next few months.

Remove usages of `org.springframework.core.io.ClassPathResource`

There are three usages of org.springframework.core.io.ClassPathResource that look like they could easily be replaced with native Java Platform functionality. That would be easier to maintain than a dependency on a big third-party library like Spring.

Support for newer JEP-229 format

See jenkinsci/incrementals-tools#24 (comment)

fyi @jglick

Noticed in jenkinsci/bom#859

fallbackSCM no longer honoured agter '404

Jenkins and plugins versions report

Post #404 the fallback repository no longer works

21:05:16  #############################################
21:05:16  #############################################
21:05:16  ##
21:05:16  ## Starting to test redacted_plugin_id redacted_plugin_tag against MavenCoordinates[groupId=org.jenkins-ci.plugins, artifactId=plugin, version=2.375.3-cb-1]
21:05:16  ##
21:05:16  #############################################
21:05:16  #############################################
21:05:16  
21:05:16  
21:05:16  
21:05:16  
21:05:16  
21:05:16  2023-02-10 21:05:16.147+0000 [id=1]	INFO	o.j.t.test.PluginCompatTester#testPluginAgainst: Created plugin checkout directory /jenkins/workspace/builders_URR-pr-builder_PR-8512/output-redacted_plugin_id/work/redacted_plugin_id
21:05:16  2023-02-10 21:05:16.148+0000 [id=1]	INFO	o.j.t.test.PluginCompatTester#getScmTag: Using SCM tag d6b8042a06de4aaaf0942ad79036095b853eea02 from POM
21:05:16  2023-02-10 21:05:16.150+0000 [id=1]	INFO	o.j.t.test.PluginCompatTester#clone: Checking out from SCM connection URL scm:git:https://github.com/jenkinsci/redacted_plugin_id-plugin.git at d6b8042a06de4aaaf0942ad79036095b853eea02
21:05:16  2023-02-10 21:05:16.387+0000 [id=1]	INFO	o.j.t.test.PluginCompatTester#clone: Checking out from SCM connection URL scm:git:[email protected]:jenkinsci-cert/redacted_plugin_id-plugin.git at d6b8042a06de4aaaf0942ad79036095b853eea02
21:05:17  2023-02-10 21:05:17.296+0000 [id=1]	SEVERE	o.j.t.test.PluginCompatTester#testPluginAgainst: Failed to check out plugin sources
21:05:17  java.io.IOException: git fetch origin failed with exit status 128: fatal: remote error: upload-pack: not our ref d6b8042a06de4aaaf0942ad79036095b853eea02
21:05:17  	at org.jenkins.tools.test.PluginCompatTester.clone(PluginCompatTester.java:708)
21:05:17  	at org.jenkins.tools.test.PluginCompatTester.cloneFromSCM(PluginCompatTester.java:612)
21:05:17  Caused: java.io.UncheckedIOException
21:05:17  	at org.jenkins.tools.test.PluginCompatTester.cloneFromSCM(PluginCompatTester.java:623)
21:05:17  	at org.jenkins.tools.test.PluginCompatTester.testPluginAgainst(PluginCompatTester.java:491)
21:05:17  	at org.jenkins.tools.test.PluginCompatTester.testPlugins(PluginCompatTester.java:318)
21:05:17  	at org.jenkins.tools.test.PluginCompatTesterCli.main(PluginCompatTesterCli.java:199)
21:05:17  2023-02-10 21:05:17.298+0000 [id=1]	SEVERE	o.j.t.test.PluginCompatTester#testPlugins: Internal error while executing a test for core 2.375.3-cb-1 and plugin redacted_plugin_id redacted_plugin_tag. Please submit a bug to plugin-compat-tester
21:05:17  java.io.IOException: git fetch origin failed with exit status 128: fatal: remote error: upload-pack: not our ref d6b8042a06de4aaaf0942ad79036095b853eea02
21:05:17  	at org.jenkins.tools.test.PluginCompatTester.clone(PluginCompatTester.java:708)
21:05:17  	at org.jenkins.tools.test.PluginCompatTester.cloneFromSCM(PluginCompatTester.java:612)
21:05:17  Caused: java.io.UncheckedIOException
21:05:17  	at org.jenkins.tools.test.PluginCompatTester.cloneFromSCM(PluginCompatTester.java:623)
21:05:17  	at org.jenkins.tools.test.PluginCompatTester.testPluginAgainst(PluginCompatTester.java:491)
21:05:17  Caused: org.jenkins.tools.test.exception.PluginSourcesUnavailableException: Failed to check out plugin sources
21:05:17  	at org.jenkins.tools.test.PluginCompatTester.testPluginAgainst(PluginCompatTester.java:505)
21:05:17  	at org.jenkins.tools.test.PluginCompatTester.testPlugins(PluginCompatTester.java:318)
21:05:17  	at org.jenkins.tools.test.PluginCompatTesterCli.main(PluginCompatTesterCli.java:199)
21:05:17  [namaste]: PCT execution finished!
21:05:17  [namaste]: Report available in: /jenkins/workspace/builders_URR-pr-builder_PR-8512/output-redacted_plugin_id/out/report.xml
21:05:17  [namaste]: Finished in: 13s
21:05:17  2023-02-10 21:05:17,821 INFO  [io.quarkus] (main) namaste stopped in 0.022s  

The tag and ref does exist in the jenkins-cert directory.

What Operating System are you using (both controller, and any agents involved in the problem)?

Linux but probably all.

Reproduction steps

1. create an war containing staged (but not yet public) security releases.
   2.run the pct for a specific plugin that has security fixes

Expected Results

PCT runs surefire tests for the plugin

Actual Results

clone/checkout fails and no tests are run

Anything else?

could probably be tested using a forked repository with a tag/release (without being pushed anywhere)

Remove `google-metadata-plugin` workaround

Pending the release of jenkinsci/google-metadata-plugin#50 the google-metadata-plugin workaround in cloneFromScm should be removed.

Remove `cloudbees-folder` and `custom-folder-icon` workarounds

These workarounds were only necessary for the BOM 2.361.x line, which was removed in jenkinsci/bom#2121.

Remove workarounds introduced in #510

#510 introduced a large amount of temporary code which ought to be deleted once plugins have been adapted accordingly.

Remove `failIfNoTests=false`

I cannot see a good reason why we are unconditionally setting failIfNoTests=false. If anything it smells like a workaround for some specific set of plugins, in which case we should only set it for those plugins and not unconditionally.

Remove `node-iterator-api-plugin` workaround

Pending the merge & release of jenkinsci/node-iterator-api-plugin#11 the node-iterator-api-plugin workaround in cloneFromScm should be removed.

Get rid of `Reflections` in `PluginCompatTesterHooks`

context: #392 (comment)

Introduced in #16, this is an overly complicated and fragile way to find implementations of something in the classpath. Use ServiceLoader etc.

Remove support for missing `<scm>` section

It turned out that there were a handful (less than 5) of plugins that were missing an SCM section, resulting in #451. It would be ideal to fix those plugins and then to rip out support for plugins without an SCM section along with the second half of this if statement:

    public static String getScmTag(PomData pomData, String name, String version) {
        String scmTag;
        if (pomData.getScmTag() != null) {
            scmTag = pomData.getScmTag();
            LOGGER.log(Level.INFO, "Using SCM tag {0} from POM", scmTag);
        } else {
            scmTag = name + "-" + version;
            LOGGER.log(Level.INFO, "POM did not provide an SCM tag; inferring tag {0}", scmTag);
        }
        return scmTag;
    }

This cleanup would then cascade into the inlining of this function and the removal of the getParentProjectName() method, of which this is the only consumer.

The reason it would be ideal is that it is not very maintainable to have to maintain two code paths when the second is so rarely used.

AD plugin IT is skiped

Jenkins and plugins versions report

Environment

https://github.com/jenkinsci/plugin-compat-tester/pull/462
https://github.com/jenkinsci/active-directory-plugin

What Operating System are you using (both controller, and any agents involved in the problem)?

Linux docker enabled

Reproduction steps

run the PCT from the PR against active-directory plugin

Expected Results

all the surefire tests are run (or skipped)

Actual Results

the *IT sirefire tests are not run

Anything else?

this is working as expected given the PR - and was working by accident before as running failsafe would run any tests matching *IT which these tests are.

Subsequent to the fix to the failsafe hook the tests are no longer run.

since #473 docker ci status fails

Jenkins and plugins versions report

this repo

ci/dockercloud-stage — Your image failed to build in Docker Hub

What Operating System are you using (both controller, and any agents involved in the problem)?

N/A

Reproduction steps

look at commit status' for anything filed since #473 was merged.

notice they have a failing check ci/dockercloud-stage — Your image failed to build in Docker Hub

whilst it may be co-incidental with the timing of the removal of the docker bits - I am not sure what docker could try and build without it.

The actual check error is {"error": "Our service is temporarily unavailable. We'll be back soon!"}

but either we should not be merging with this failing check as it is important - or we shouldn;t bother running it

Expected Results

all checks for PRs pass if the code is OK

Actual Results

one failing check even if the PR is good.

Anything else?

I have no idea where this integration is performed

Delete `TransformPom`

#379 (comment) CC @jtnord

Rationalize error handling

The error handling in this codebase is extremely convoluted. Some examples:

• The code makes copious use of the "log and throw" antipattern.
• Lots of code catches overly broad types like RuntimeException, Exception, or Throwable rather than a specific type.

More problems are likely to be discovered after reading the code in more detail. I might try to look at this after rationalizing the logging subsystem.

Native support for multi-module Maven projects

The current code has extremely special and convoluted handling for multi-module Maven projects. Every multi-module Maven project needs a custom hook in PCT in order to work at all, and even then there are various problems. For example, consider the case of a multi-module Maven project that contains a plugin with a dependency on another plugin in the same multi-module Maven project. An example of this is Pipeline: Declarative Extension Points API, which depends on Pipeline: Model API, both of which are in the same multi-module Maven project. In this case, our current workflow requires two different invocations of PCT, one for each plugin. However, this results in PCT unnecessarily compiling both the dependent and its dependency but only executing tests for the dependent.

At a high level, this should be rewritten so that given a set of plugins, PCT collects a set of repositories containing those plugins and then invokes Maven from the root of each repository. Then we would need no special handling for multi-module Maven projects.

Implementing this should be relatively straightforward: the list of plugins provided can be transformed into a list of repositories, with the main for loop in org.jenkins.tools.test.PluginCompatTester#testPlugins converted to loop over repositories rather than plugins, with the semantics of org.jenkins.tools.test.PluginCompatTester#testPluginAgainst changed to be "test repository against" rather than "test plugin against."

However, one complication is CI callers such as jenkinsci/bom. These callers take a megawar and list its plugins, invoking parallel runs of PCT, one for each plugin. With the change above to iterate over repositories over plugins, such consumers will want to pass in a list of all plugins that are contained in a given repository rather than passing in a single plugin. For example, to test the pipeline-model-definition-plugin repository, consumers would want to invoke PCT with --plugins pipeline-model-api,pipeline-model-definition,pipeline-model-extensions,pipeline-stage-tags-metadata; PCT should then coalesce these into the single repository pipeline-model-definition-plugin, do one clone, one build, and one test cycle. But how would consumers know to pass in that particular list of plugins?

One solution would be a new PCT flag that, for a given megawar, prints the mapping of repositories to plugins. For example, it could have output like this:

REPOSITORY                        PLUGINS
pipeline-model-definition-plugin  pipeline-model-api,pipeline-model-definition,pipeline-model-extensions,pipeline-stage-tags-metadata
text-finder-plugin                text-finder

Consumers would run PCT with this flag to determine the division of parallel branches, each line of the output being a parallel branch (one for each repository). The consumers would pass in the output of the PLUGINS column as the --plugins parameter to PCT in each branch.

artifacts are not published to releases repo (except the cli)

Jenkins and plugins versions report

all the modules have skip deployment (and install) set except the cli, however this is not good as users at least need the model to be able to write custom hooks.

Expected resullts

• when a release is made by the CD workflow all the artifacts are published

Actual Results

• only the CLI artifact is published.

See also jenkins-infra/repository-permissions-updater#2827 which may be required to be able to publish everything)

What Operating System are you using (both controller, and any agents involved in the problem)?

n/A

Reproduction steps

N/A

Expected Results

I can consume releases of the plugin-compat-tester-model

Actual Results

the releases of the model are 1 year old, nothing recent is available

Anything else?

No response

Remove `-storeAll` option

Even though CloudBees appears to use -storeAll I cannot see a good reason why anyone would want to use it. @imonteroperez @jtnord Is there a reason we want to keep this option or can we rip it out?

Remove custom XPath logic from `PluginRemoting`

PluginRemoting contains a lot of custom XPath logic to parse Maven pom.xml files. Since we have maven-model on our class path, all of this could be replaced with a few lines to delegate to MavenXpp3Reader which already knows how to parse pom.xml files.

Remove `popper2-api` workaround

Pending the release of jenkinsci/popper2-api-plugin@bf781e3 the popper2-api-plugin workaround in cloneFromScm should be removed.

Reduce or eliminate Maven Shade Plugin warnings

This component has quite a few warnings emanating from the Maven Shade Plugin. I am addressing some of them in various open PRs, but it would be good to check what remains after all open PRs are processed and fix as many of the remaining issues as is practical.

Remove `VersionComparator`

org.jenkins.tools.test.model.comparators.VersionComparator duplicates the functionality of hudson.util.VersionNumber and can simply be replaced with the latter.

Remove `json-lib`

org.jenkins.tools.test.PluginCompatTester#scanWAR builds a JSON object only to immediately parse it into hudson.model.UpdateSite.Data#Data. The intermediary step is unnecessary and we could instead directly construct the target object, eliminating our need for the json-lib JAR in the process.

do not ignore incorrect localCheckoutDir

Jenkins and plugins versions report

if the local checkout directory is provided but does not exist no error is raised

In reviewing #484 I was alerted to the fact that the pct considers the local checkout directory to not be provided if it does not exist.

This violates the principle of least surprise - it the option is provided yet the path does not exist the pct should return with an error rather than continuing on to test something that may not be what was intended.

What Operating System are you using (both controller, and any agents involved in the problem)?

windows - but this is cross platform.

Reproduction steps

run the pct with the localCheckout option set to a non existant directory

e.g.
java -Xmx1g -jar target\plugins-compat-tester-cli.jar -workDirectory c:\workarea\source\github\cloudbees\unified-release\output-caffeine-api\work -reportFile c:\workarea\source\github\cloudbees\unified-release\output-caffeine-api\out\report.xml -war c:\workarea\source\github\cloudbees\unified-release\products\core-cm\target\core-cm.war -storeAll -fallbackGitHubOrganization jenkinsci-cert -hookPrefixes com.cloudbees.hooks -externalHooksJars c:\workarea\source\github\cloudbees\unified-release\tests\cb-pct-hooks\target\cb-pct-hooks.jar -includePlugins caffeine-api -mavenPropertiesFile c:\workarea\source\github\cloudbees\unified-release\output-caffeine-api\work\maven.properties -mavenProperties jenkins.version=2.375.3-cb-1 -localCheckoutDir c:\workarea\does-not-exist

note that c:\workarea\does-not-exist does not exist :)

Expected Results

pct fails

Actual Results

pct runs and passes

Anything else?

No response

Update to Java 11

#389 failed with

Restricted to JDK 8 yet org.jenkins-ci:version-number:jar:1.11:compile contains hudson/util/VersionNumber$ListItem.class targeted to JDK 11

Perhaps @basil has something in preparation to update this repo to 11 along with other parts of the toolchain? On the other hand, PCT should be able to run on JDK 11 while still testing plugins with older baselines, so updating it is probably an independent project. From a quick glance,

plugin-compat-tester/pom.xml

Line 7 in b2c7632

<version>1.67</version>

is super old and so is

plugin-compat-tester/Jenkinsfile

Line 21 in b2c7632

sh 'docker run --rm -v $(pwd):/src -w /src maven:3.6.0-jdk-8 sh -c "rm -rf * .[a-zA-Z]*" || :'

and

plugin-compat-tester/Dockerfile

Line 40 in b2c7632

FROM maven:3.8.1-jdk-8

and

plugin-compat-tester/Makefile

Line 12 in b2c7632

JENKINS_VERSION=2.164.3

etc.