ot4i / ace-demo-pipeline

Demo pipeline for App Connect Enterprise

ACE demo pipeline

Demo pipeline for ACE to show how ACE solutions can be built in CI/CD pipelines using standard tools. The main focus is on how to use existing ACE capabilities in a pipeline, with the application being constructed to show pipeline-friendliness rather than being a "best practice" application. As part of this, the pipeline scripts are stored in this repo along with the application source to make the demo simpler, while in practice they would often be stored separately.

The overall goal is to deploy a REST HTTP application to an ACE integration server:

The application used to demonstrate the pipeline consists of a REST API that accepts JSON and interacts with a database via JDBC, with a supporting shared library containing a lot of the code (hereafter referred to as the "Tea REST application"). It is designed around indexing different types of tea, storing the name and strength of the tea along with assigning a unique integer id to each type so that it can be retrieved later. Audit data is logged as XML for each operation performed.

As this application exists to help demonstrate pipelines and how they work with ACE, there are some shortcuts in the code that would not normally be present in a production-ready application: the database table is created on-demand to make setup easier, the logging goes to the console instead of an audit service, etc.

Recent changes

• Minikube added as the default "plain Kubernetes" option.
• ACE-as-a-Service added as a deploy target (see below).

Technology and target options

This repo can be built in several different ways, and can deploy to different targets (see Getting started for suggestions on how to choose a target) from the same source as shown in this diagram:

Testing is split into "Unit Test" and "Component Test" categories, where "unit tests" are self-contained and do not connect to external services such as databases (so they can run reliably anywhere) while the term "component test" was used in the ACE product development pipeline to mean "unit tests that use external services (such as databases)". See ACE unit and component tests for a discussion of the difference between test styles in integration.

Pipeline technology options currently include:

• GitHub Actions for CI build and test before pull requests (PRs) are merged. This requires a GitHub instance that supports actions (not all Enterprise variants do), and credit enough to run the actions. There is currently no component testing nor a deploy target (though these could be added) for these builds.
• Tekton can be used to build, test, and deploy the Tea REST application to ACE runtime infrastructure such as Kubernetes containers. Tekton is the basis for many Kubernetes application build pipelines and also underpins RedHat OpenShift Pipelines.
• Jenkins can be used to build, test, and deploy the Tea REST application to ACE runtime infrastructure such as integration nodes. Jenkins is widely used in many organizations for build and deployment.

ACE deploy targets currently include:

• Kubernetes containers, with both standalone ACE containers and ACE certified containers (via the ACE operator code) as possible runtimes. Minikube (easily installed locally) and OpenShift can be used with the former, while the latter expects to deploy to the Cloud Pak for Integration (CP4i). See tekton/README.md#container-deploy-target for a description of the container deploy pipelines.
• ACE-as-a-Service (ACEaaS) running on Amazon Web Services (AWS). This option requires an instance (which can be a trial instance) of ACEaaS to be available but does not require ACE servers to managed directly (in virtual machines or containers) as the flows run entirely in the cloud. See demo-infrastructure/README-aceaas-pipelines.md for an overview of the pipelines deploying to ACEaaS.
• An ACE integration node, using an existing ACE integration node.

As can be seen from the diagram above, not all deployment targets have been configured for all of the pipeline technology options, but more could be added as needed.

As well as multiple options for pipelines and deploy targets, multiple build tools can be used to build the ACE flows, Java code, Maps, etc and test the application in the pipeline and locally:

• Standard ACE commands introduced at v12 (such as ibmint) can be used to build, deploy, and test the application.
• Maven can also be used, and was the default in the ACE v11 version of this repo.
• Gradle can be used to run builds and unit tests, but has not been enabled for component tests.
• The toolkit can build and run the application and tests, and also to check source into the GitHub repo.

Getting started

Regardless of the pipeline technology and deployment target, some initial steps are similar:

• Forking this repository is recommended as this allows experimentation with all aspects of the application and pipeline. PRs welcome, too!
• A database will be needed for the application to run correctly. GitHub Action CI builds can succeed without a database because they only run build and UT steps, but all other use cases require a database, and DB2 on Cloud (requires an IBM Cloud account) is one option that requires no local setup nor any payment. For DB2oC, create a "free tier" DB2 instance via "Create resource" on the IBM Cloud dashboard and download the connection credentials for use in the pipeline. See demo-infrastructure/cloud-resources.md for more details.
  • Note that component testing relies on the same DB2 on Cloud instance as the eventual application image; this is not a best practice, but does keep the demo simpler to get going, and so getting the DB2 instance credentials set up in Kubernetes and/or locally is necessary for the component tests.
• Installing the ACE toolkit locally is recommended, and the ACE v12 toolkit can clone the (forked) repo locally with the pre-installed eGit plugin. Although development and testing can be done online using a GitHub-hosted container (see README-codespaces for details), having the toolkit available locally is helpful for replicating the most common ACE development experience. See the developer edition download page for a free version (limited to one transaction per second per flow) if your organization does not have existing ACE licenses.

Beyond those common steps, the choice of pipeline and target determine the next steps. The simplest way to choose the pipeline is to choose the target (Kubernetes, ACEaaS, or integration nodes), and then pick one of the pipeline technologies that will deploy to that target. For advanced users who are already familiar with pipelines it may better to start with a familiar pipeline technology and then choose an available target.

• For Tekton deploying to Kubernetes, see tekton/README.md for instructions for the various container options and pipelines.
  • See also CP4i README for CP4i-specific variations, including component testing in a CP4i container (as opposed to a build pipeline container) to ensure credentials configurations are working as expected.
  • Note that the Tekton pipeline can also create temporary databases for use during pipeline runs; see temp-db2 for more details.
• Tekton-to-ACEaaS follows a similar pattern (see tekton/README.md#ace-as-a-service-target), but does not need a runtime container as the runtime is in the cloud. Credentials are needed for the cloud service.
• For Jenkins, see the Jenkins README for details and instructions on initial setup.
  • Integration node targets require host/port/server information.
  • Additional steps are required for ACE-as-a-Service credentials.

ace-demo-pipeline's Issues

Use ibmint instead of requiring Maven for Java build

ACE v12 provides Java build capability with the ibmint command, and this could be used instead of Maven to avoid having to build a separate ace-minimal-build container image. The original ACE v11 pipeline had to use Maven to avoid pulling in mqsicreatebar for the Java build, but this is no longer the case with ACE v12.

Maven would still be useful for pulling in the DB2 JDBC JAR file (currently stored in this repo), but the pipeline setup would be greatly simplified by avoiding Maven entirely . . .

ACE v11 toolkit Maven HTTPS problem

Placeholder: 11.0.0.11 toolkit Maven level fails to handle HTTPS redirection (Maven repo change).

<!-- force https -->
<repositories>
    <repository>
        <id>central</id>
        <url>https://repo1.maven.org/maven2</url>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
    </repository>
</repositories>
<pluginRepositories>
    <pluginRepository>
        <id>central</id>
        <url>https://repo1.maven.org/maven2</url>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
    </pluginRepository>
</pluginRepositories>

export IBM_JAVA_OPTIONS="-Dcom.ibm.jsse2.overrideDefaultTLS=true"

Gradle pipeline demo

Hi, we're familiar with the maven pipelines but would love to see a Gradle demo pipeline as well especially utilizing the new ibmint commands :)

Permission issue - Failed to open file HASharedWorkPath

Hi, I'm trying your approach but facing below issue at step maven-build of task maven-ace-build (tekton pipeline, ocp 4.6). Any idea what may be missing from your script and how to resolve it please. Thanks

Compiling log
[�[1;34mINFO�[m] �[1m---------------< �[0;36mace-demo-pipeline:demo-infrastructure�[0;1m >----------------�[m
[�[1;34mINFO�[m] �[1mBuilding demo-infrastructure 0.0.1 [6/7]�[m
[�[1;34mINFO�[m] �[1m--------------------------------[ jar ]---------------------------------�[m
[�[1;34mINFO�[m]
[�[1;34mINFO�[m] �[1m--- �[0;32mmaven-compiler-plugin:3.1:compile�[m �[1m(default-compile)�[m @ �[36mdemo-infrastructure�[0;1m ---�[m
[�[1;34mINFO�[m] Changes detected - recompiling the module!
[�[1;33mWARNING�[m] File encoding has not been set, using platform encoding ANSI_X3.4-1968, i.e. build is platform dependent!
[�[1;34mINFO�[m] Compiling 1 source file to /work/ace-demo-pipeline/demo-infrastructure/target/classes
[�[1;34mINFO�[m]
[�[1;34mINFO�[m] �[1m--- �[0;32mexec-maven-plugin:3.0.0:exec�[m �[1m(create-work-dir)�[m @ �[36mdemo-infrastructure�[0;1m ---�[m
mqsicreateworkdir: Copying sample server.config.yaml to work directory
1 file(s) copied.
Failed to open file /var/mqsi/registry/utility/HASharedWorkPath with error Permission denied
BIP2113E: IBM App Connect Enterprise internal error: diagnostic information ''Permission denied'', '13', ''/var/mqsi/registry/utility/HASharedWorkPath''.
An internal software error has occurred in IBM App Connect Enterprise. Further messages may indicate the effect of this error on the component.
Shutdown and restart the component. If the problem continues to occur, then restart the system. If the problem still continues to occur contact your IBM support center.
BIP8081E: An error occurred while processing the command.
An error occurred while the command was running; the command has cleaned up and ended.
Use messages prior to this one to determine the cause of the error.
Check for some common problems:
Does the user id have the correct authorities (for example a member of the mqbrkrs group)?
Is any operating system limit set too low to allow the command to run?
Is the environment correctly set up?
Correct the problem and retry the command, otherwise, contact your IBM support center.
[�[1;31mERROR�[m] Command execution failed.
�[1;31morg.apache.commons.exec.ExecuteException�[m: �[1;31mProcess exited with an error: 81 (Exit value: 81)�[m
�[1mat�[m org.apache.commons.exec.DefaultExecutor.executeInternal (�[1mDefaultExecutor.java:404�[m)
�[1mat�[m org.apache.commons.exec.DefaultExecutor.execute (�[1mDefaultExecutor.java:166�[m)
�[1mat�[m org.codehaus.mojo.exec.ExecMojo.executeCommandLine (�[1mExecMojo.java:982�[m)
�[1mat�[m org.codehaus.mojo.exec.ExecMojo.executeCommandLine (�[1mExecMojo.java:929�[m)
�[1mat�[m org.codehaus.mojo.exec.ExecMojo.execute (�[1mExecMojo.java:457�[m)
�[1mat�[m org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (�[1mDefaultBuildPluginManager.java:137�[m)
�[1mat�[m org.apache.maven.lifecycle.internal.MojoExecutor.execute (�[1mMojoExecutor.java:210�[m)
�[1mat�[m org.apache.maven.lifecycle.internal.MojoExecutor.execute (�[1mMojoExecutor.java:156�[m)
�[1mat�[m org.apache.maven.lifecycle.internal.MojoExecutor.execute (�[1mMojoExecutor.java:148�[m)
�[1mat�[m org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (�[1mLifecycleModuleBuilder.java:117�[m)
�[1mat�[m org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (�[1mLifecycleModuleBuilder.java:81�[m)
�[1mat�[m org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (�[1mSingleThreadedBuilder.java:56�[m)
�[1mat�[m org.apache.maven.lifecycle.internal.LifecycleStarter.execute (�[1mLifecycleStarter.java:128�[m)
�[1mat�[m org.apache.maven.DefaultMaven.doExecute (�[1mDefaultMaven.java:305�[m)
�[1mat�[m org.apache.maven.DefaultMaven.doExecute (�[1mDefaultMaven.java:192�[m)
�[1mat�[m org.apache.maven.DefaultMaven.execute (�[1mDefaultMaven.java:105�[m)
�[1mat�[m org.apache.maven.cli.MavenCli.execute (�[1mMavenCli.java:957�[m)
�[1mat�[m org.apache.maven.cli.MavenCli.doMain (�[1mMavenCli.java:289�[m)
�[1mat�[m org.apache.maven.cli.MavenCli.main (�[1mMavenCli.java:193�[m)
�[1mat�[m sun.reflect.NativeMethodAccessorImpl.invoke0 (�[1mNative Method�[m)
�[1mat�[m sun.reflect.NativeMethodAccessorImpl.invoke (�[1mNativeMethodAccessorImpl.java:90�[m)
�[1mat�[m sun.reflect.DelegatingMethodAccessorImpl.invoke (�[1mDelegatingMethodAccessorImpl.java:55�[m)
�[1mat�[m java.lang.reflect.Method.invoke (�[1mMethod.java:508�[m)
�[1mat�[m org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (�[1mLauncher.java:282�[m)
�[1mat�[m org.codehaus.plexus.classworlds.launcher.Launcher.launch (�[1mLauncher.java:225�[m)
�[1mat�[m org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (�[1mLauncher.java:406�[m)
�[1mat�[m org.codehaus.plexus.classworlds.launcher.Launcher.main (�[1mLauncher.java:347�[m)
[�[1;34mINFO�[m] �[1m------------------------------------------------------------------------�[m
[�[1;34mINFO�[m] �[1mReactor Summary:�[m

Use archive.apache.org for Maven downloads

dlcdn.apache.org only has the latest Maven minor version release, and don't have an obvious "latest" pointer; when they udate, then the download URL in demo-infrastructure/docker/ace-minimal-build/Dockerfile stops working.

While using the latest would be a good plan, an intermediate solution is to use https://archive.apache.org/dist/maven/maven-3//.tar.gz instead, as that contains the latest and also older releases; any Maven update can then be staged in this repo rather than being forced by Maven release schedules.