ibm / dbb-zappbuild

zAppBuild is a generic build solution for building z/OS applications using Apache Groovy build scripts and IBM Dependency Based Build (DBB) APIs.

zAppBuild

zAppBuild is a generic build solution for building z/OS applications using Apache Groovy build scripts and IBM Dependency Based Build (DBB) APIs.

Resources

• IBM Dependency Based Build Product Page
• IBM DBB Knowledge Center
• IBM/dbb Repository
• IBM IDZ Community
• IBM DBB Community

Contributing

For instructions on how to contribute enhancements and bug fixes to zAppBuild, please read the Contributions Guidelines.

How zAppBuild works

The zAppBuild repository is intended to be cloned to a single location on Unix Systems Services (USS) and used to build all of your z/OS applications. Global configuration properties are configured in the properties files in the build-conf directory. Specifying application-level properties is done by simply copying the supplied application-conf folder (located in the samples folder) to the application source repository you want to build and then verify/update the contained default configuration property values to ensure they meet the build requirements of your application. See the included MortgageApplication sample for an example of an application that has been modified to be built by zAppBuild.

IMPORTANT : The datasets.properties must be configured for your build machine before executing a build! See build-conf/README.md for more information.

Supported Languages

The zAppBuild sample provides the following language build scripts by default:

• Assembler.groovy
• BMS.groovy
• Cobol.groovy
• LinkEdit.groovy (for building link cards)
• PLI.groovy
• DBDgen.groovy
• PSBgen.groovy
• MFS.groovy
• ZunitConfig.groovy
• CRB.groovy
• Transfer.groovy (for transport non-buildable files like JCL or PROC into build libraries and register them as build output)

All language scripts both compile and optionally link-edit programs. The language build scripts are intended to be useful out of the box but depending on the complexity of your applications' build requirements, may require modifications to meet your development team's needs. By following the examples used in the existing language build scripts of keeping all application specific references out of the build scripts and instead using configuration properties with strong default values, the zAppBuild sample can continue to be a generic build solution for all of your specific applications.

Build Scope

The build scope of zAppBuild is an application which is loosely defined as one or more Git repositories containing all the z/OS source files required to build the application. There are no specific rules as to the structure of the repositories except that one repository must contain the high level application-conf folder provided by zAppBuild which contains all of the configuration properties for building the application programs.

NOTE: All source repositories that make up the application must be cloned on the build machine under a common workspace directory prior to calling build.groovy.

zAppBuild supports a number of build scenarios:

• Single Program - Build a single program in the application.
• List of Programs - Build a list of programs provided by a text file.
• Full Build - Build all programs (or buildable files) of an application.
• Impact Build - Build only programs impacted by source files that have changed since the last successful build.
• Impact Build with baseline reference - Build only programs impacted by source files that have changed by diff'ing to a previous configuration reference.
• Topic Branch Build - Detects when building a topic branch for the first time and will automatically clone the dependency data collections from the main build branch in order to avoid having to rescan the entire application.
• Merge Build - Build only changed programs which will be merged back into the mainBuildBranch by using a triple-dot git diff.
• Scan Source - Skip the actual building and only scan source files to store dependency data in collection (migration scenario).
• Scan Source + Outputs - Skip the actual building and only scan source files and existing load modules to dependency data in source and output collection (migration scenario with static linkage scenarios).
• Build Preview - Supplemental build option. Process all phases of the supplied build option, but will not execute the commands. A build report and a build result are generated with a specific status that excludes them in subsequent impact build calculations.

Instructions on invoking a zAppBuild is included in docs/BUILD.md as well as invocation samples for the above mentioned build scenarios including sample console logs.

Application-level build properties such as compile and link options can be defined in various ways to set global defaults, application-level overrides or even file level (member-level) overrides. The various supported strategies are documented in docs/FilePropertyManagement.md.

zAppBuild comes with a set of reporting features. It helps development teams to understand the impact of changed files across multiple applications. Another feature helps to identify conflicts due to concurrent development activities within their application. An overview of these features are documented in docs/REPORTS.md.

Links to additional documentation is provided in the table below.

Repository Legend

Folder/File	Description	Documentation Link
build-conf	This folder contains global configuration properties used by build.groovy and language build scripts.	build-conf/README.md
docs	This folder contains supplemental documentation to help explain the implementation and usage of zAppBuild features.	docs/README.md
languages	This folder contains the language specific build scripts that are associated to build files via script mappings (see samples/application-conf/files.properties) and called by build.groovy.	languages/README.md
samples/application-conf	The application-conf folder contains application specific configuration properties used by build.groovy and language build scripts. It is intended to be copied as a high level folder to the application repository and configured to meet the build requirments of the application. Ex. myAppRepository/application-conf	samples/application-conf/README.md
samples/MortgageApplication	This is an updated version of the original MortgageApplication sample designed to be built by zAppBuild.	samples/MortgageApplication/README.md
utilities	This folder contains utility scripts which provide common utility functions used by the various zAppBuild build scripts.	utilities/README.md
build.groovy	This is the main build script that is called to start the build process.	docs/BUILD.md
test	This folder contains testing framework for ZAppBuild which includes test scripts and related test content.	test/README.md

dbb-zappbuild's Issues

MFS.groovy method createPhase2Command does not return the MVSExec command

MFS.groovy should use temporary dataset allocations for REFRD, REFOUT, REFIN

Today, there is a hardcoded dependency on a REFERAL dataset. Should use temporary files.

Investigations impactBuild of nested projects

Investigating into impactBuild of nested projects:
While MortgageApplication sample is now shipped as a subfolder in zAppBuild, I see a that in an impactBuild scenario, the relative path to the file is not correctly resolved:

dbb-zappbuild/utilities/ImpactUtilities.groovy

Line 314 in 421c805

def fixFilePath(String file, String dir, boolean mustExist ) {

Impact Utilities returns the wrong filePath:

** Storing MortgageApplication : e3a10c29a59c8bf5f1b68d07d031d0701491b741
** Calculating changed files for directory /var/dbb/buildhome/workspace/Mortgage-Sample-Nested-Project/Mortgage-ImpactBuild-NestedPrj/samples/MortgageApplication
** Diffing baseline e3a10c29a59c8bf5f1b68d07d031d0701491b741 -> current 9b7953582a63f1f6db63f7ac06d5a957eb5c4cf7
*** Changed files for directory /var/dbb/buildhome/workspace/Mortgage-Sample-Nested-Project/Mortgage-ImpactBuild-NestedPrj/samples/MortgageApplication:
@@ fixFilePath (workspace+fixedFileName)/var/dbb/buildhome/workspace/Mortgage-Sample-Nested-Project/Mortgage-ImpactBuild-NestedPrj/samples/MortgageApplication/epsmpmt.cbl

Allow zAppBuild scriptmappings to support absolute paths

Currently zAppBuild can only invoke build scripts in the zAppBuild/languages folder. In order to allow customers to easily add other build scripts without having to add them to their owned forked repo, we need to be able to invoke build scripts outside of zAppBuild/languages folder.

zAppbuild enhancement - isSQL scanning in Copybooks

Issue:
When a main program contains no exec-sql stmt but one of its dependencies does, like a copybook, the build fails because the compiler option "SQL" is not provided.

Solution:
During the build, if a main program is not "isSQL", spin though its dependencies to determine if any "isSQL". If so, enable the 'isSQL" compiler option.

Add code coverage collector in the zUnit JCL

PR to come

CICS definitions are missing for the Mortgage application

If customer wants to try the CICS Mortgage application will need to define the CICS Resources..
It would be nice to add this along with the sample code.

DEFEPS.txt

Linking of CICS and SQL program - DFHECI

The sample code in zAppBuild's cobol.groovy is having this logic:
// add the link source code
linkCard = " INCLUDE TEMP"
if (buildUtils.isCICS(logicalFile)) {
linkCard = linkCard + "\n INCLUDE SYSLIB(DFHECI)"
}
if (buildUtils.isSQL(logicalFile)) {
linkCard = linkCard + "\n INCLUDE SYSLIB(DSNCLI)"
}

However this should not be best practice to my best knowledge.

DFHECI should be replaced with DFHELII (https://www.ibm.com/support/knowledgecenter/SSLTBW_2.4.0/com.ibm.zos.v2r4.ceea200/ceea200259.htm). According to my information would not work at all, as DFHECI nneds to be the first include, otherwise the program will abend - so it should be including DFHECI before INCLUDE TEMP. This is not something I have tested

Also we are including DSNCLI. That should only be for CICS programs. Batch programs should include DSNELI, IMS programs should include DFSLI000 and so on.

This should have an extra level of check if my assumptions are correct

Add dbb.file.tagging

With DBB 1.0.8, the CopyToHFS command was extended enable automated file tagging of NON-ASCII files. This leverages a new environment variable dbb.file.tagging.

I raise this issue to introduce the environment variables in the dbb-zappbuild sample.

For more details, please have a look at:
https://www.ibm.com/support/knowledgecenter/SS6T76_1.0.8/scriptorg.html

Add back missing LIBRARY entry in package bind.

Add back missing LIBRARY entry in package bind.

   BIND PACKAGE(&COLLID)    +                                
        MEMBER(&MEMBER)     +
        LIBRARY('&LIB')     +                                
        OWNER(&OWNER)       +                                
        QUALIFIER(&QUAL)    +                                
        ACTION(REPLACE)     +                                
        ISOLATION(CS)                                        
   END  

Alternative option to overwrite/set file properties

com.ibm.dbb.build.BuildProperties provides global build properties across all scripts. This class provides the method getFileProperty(property,buildfile) to retrieve property overwrites for a particular file or set of files.

A sample is contained in the MortgageApp:

dbb-zappbuild/samples/MortgageApplication/application-conf/file.properties

Lines 15 to 18 in fdee854

	#
	# Skip creating a load module for these programs as they will be statically linked to other programs
	cobol_linkEdit = false :: **/cobol/epsnbrvl.cbl, **/cobol/epsmlist.cbl

However, it might be cumbersome to manage many exceptions that way. And also if you have several combinations (values), like for compiler options, it is not easy to handle.

The request is to have file level properties in a individual property files following java.util.Properties instead of DBBs BuildProperties to allow overwrites / or new properties on a per build file basis.

Move build data sets creation to language scripts

Currently all build data sets required by zAppBuild language scripts are created in the build.groovy initialize method even if the build does not actually build all the languages. While this is not a major issue for team builds, it is mostly unnecessary for user builds and for large customer applications who have introduced more language scripts, could cause problems with dozens of unused data sets created for each developer. Moving the build data set creation to the language scripts should eliminate unused data set creation . . .

however . . .

we will also need to change the build script iteration logic to skip languages that have no files in the sub-list. Currently zAppBuild runs all language scripts even if the build sub-list is empty.

Automation of fileranks in static link scenario

This issue is for ordering he processing of programs, when programs are statically linked.

In this case we need to process the sub-modules first, so the recompiled submodule is included into the main-executable.

If we have 2 COBOL programs: MAINPGM1 which statically calls SUBPROG1. If we change SUBPROG1 and performs a impactBuild, we will identify that we have an dependency in the output collection to MAINPGM1, so both will be processed. However due to the alphabetic order, MAINPGM1 will be processed first, and therefore include the old version of SUBPROG1.

The current implementation of zAppBuild on allows us to do fileRanking, and we therefore need to identify all the submodules to get them to process first.

In many cases this is not defined specifically in existing Library Managers. In some cases the user relies on the validation that happens during promotion in the Library Manager, which can ensure that all executable with SUBPROG1 contains the latest version of SUBPROG1, and the user can then manipulate the order of the compiles.

add a flag for skip creation of the build subfolder containing the timestamp

the property buildOutDir is used to store build outputs on the build machine.

Today, for all build types (except for userBuild), zappbuild creates a subfolder with the build label - containing a timestamp - which makes post processing scripts cumbersome.

In a Jenkins or Gitlab pipeline, you would rather use the build-id to store outputs in a dedicated folder and by using it in the -outDir command line property.

Please add a property to skip the creation of the subfolder containing the build label.

Add BZUSAMP library to necessary dataset definitions

COBOL.groovy adds the BZUSAMP library, and the template for dataset definitions misses this entry.

dbb-zappbuild/build-conf/datasets.properties

Line 58 in 35adba0

SFELLOAD=

writing build reports takes long time

I am trying to run the zappbuild through command line . The program build itself runs pretty quick , but i think it take a long time to write the actual build report. I see a long pause after that below line and then it completes OK.

** Writing build report data to /u/xx/BuildReport ***

We just use the default templates that came with dbb. Is this expected ? Not sure if i am missing anything . Is there any recommendation ?

Assembler assembly error - No such property: compile for class: Assembler

Getting the following error trying to assemble an assembler program:

groovy.lang.MissingPropertyException: No such property: compile for class: Assembler
at Assembler.createAssemblerCommand(Assembler.groovy:146)
at Assembler$_run_closure1.doCall(Assembler.groovy:46)
at Assembler.run(Assembler.groovy:28)
at Assembler$run.callCurrent(Unknown Source)
at com.ibm.dbb.groovy.ScriptLoader._run(ScriptLoader.groovy:124)
at com.ibm.dbb.groovy.ScriptLoader$_run$1.call(Unknown Source)
at com.ibm.dbb.groovy.ScriptLoader.runScript(ScriptLoader.groovy:81)
at build$_run_closure1.doCall(build.groovy:55)
at build.run(build.groovy:47)

ImpactBuild should be triggered for Properties Changes

Changes in the properties files should pick the corresponding source code during impact build (even when corresponding source is not changed) . For example when we change the cobol_compileParms in the properties file, it will not pick all the source code affected during impact build as only source file changes are considered during impact build but in reality changing the compileParms actually changes the load module created. even though there is no change in the source module. This is also applicable for changes in file specific properties (like cobol_testcase = true :: **/testcase/*.cbl) also.

Substringing might fail in certain scenarios

I identified a scenario, where the existing logic crops too much of the string. It would be preferred to use the information from the scanner.

dbb-zappbuild/languages/ZunitConfig.groovy

Line 45 in 35adba0

playback = line.split("=")[1].substring(1,line.split("=")[1].length()-3)

When we do the linkedit during the cobol compilation links edit datasets are not created

Example: for an IMS application the linkedit step failed with this error:

*** Building file nazare-demo-imsapp/COBOL/LOADHIST.cbl
IKJ56228I DATA SET NAZARE.WAZI.IMSAPP.LINK NOT IN CATALOG OR CATALOG CAN NOT BE ACCESSED    
org.codehaus.groovy.runtime.InvokerInvocationException: com.ibm.dbb.build.BuildException: java.io.IOException: com.ibm.dbb.build.BuildException: BGZTK0016E An error occurred running BPXWDYN command 'free dd(SYS00102)'.

PR to come.

Can buildOrder have a default value in build.properties. Can application.properties be optional ?

Customer wants to minimize the application specific properties. Looks like buildOrder could be defined in build.properties for everyone.

Some properties are defined in both build.properties and application.properties. buildOrder is only in application.properties, I think it would make sense for it to be in build.properties.

I actually wonder if all properties of application.properties should be in build.properties as well, to provide defaults.

Note that loading application.properties is mandatory in build.groovy. If there are defaults for everything in build.properties, then we could make loading application.properties optional.

Missing a COBOL program on the provided Mortgage sample.

A COBOL program EPSCSMRT.cbl is missing. This cause a bad mortgage calculation when running the transaction EPSP.

This missing program also must be compiled with CICS option since is invoked via CICS LINK (like it is now epsmlist.cbl),
Once you add the missing program also need to update the file.properties :

# epsmlist needs to compile as CICS but does not have EXEC CICS statements   
# so is not automatically flagged as CICS by dependency scanner   
# added by Regi - July 27,2020 - EPSCSMRT also must be compiled with CICS option as added below 

 isCICS = true :: **/cobol/epsmlist.cbl, **/cobol/EPSCSMRT.cbl**`

If not compiling with CICS the transaction will have an ASRA abend.
Once this is done the application now works as it should be.
More details at #dbb-git-zsystems slack.

Cobol.groovy fails during user build due to non-existent BMS copybook dataset

Build dataset creation was recently moved from build.groovy to the individual language scripts to avoid having user build creating dozens of unused datasets. However the Cobol.groovy build script includes the BMS copybook dataset in its SYSLIB concatenation. Its conditional subject to the existence of BMS copybook name property but its defined in the build-conf folder, so it always exists even if the dataset has not been created. We need to use another condition. Perhaps checking the build order property for the BMS entry. Or just using JZOS to test if the dataset exists.

ImpactBuild fixGitDiffPath not working with some configuration

The new method fixGitDiffPath is not working with some configuration. It only works if the applicationSrcDirs=${application}. But if we want to specify folders under the ${application} to only scan source folders that we want to build (to avoid scanning the whole ${application} folder it does not work. In this case impact build find nothing. Example config:

applicationSrcDirs=nazare-demo-genapp/base/src/cobol,nazare-demo-genapp/base/src/bms

build.properties applicationConfRootDir documentation incorrect

The applicationConfRootDir property comment states that it defaults to ${workspace}/${application} however the value of the property really defaults to ${workspace}\ and the the ${application} folder is used to resolve to the application-conf folder.

This property can be used to relocate customer's application's configuration folders to USS as well as relocating them to the zAppBuild repository under a common folder like "applications":

applicationConfRootDir=${zAppBuildDir}/applications/

The documentation should be clarified for this example.

Adding db2 load lib to link steps

Adding missing SDSNLOAD allocation in db2 batch scenarios

MFS language script misses job.close()

The MFS language scripts, miss the MVSJob job.close() statement at

dbb-zappbuild/languages/MFS.groovy

Line 65 in 421c805

You may experience an allocation error:

org.codehaus.groovy.runtime.InvokerInvocationException: com.ibm.dbb.build.BuildException: com.ibm.dbb.build.BuildException: BGZTK0016E An error occurred running BPXWDYN command 'alloc dd(SEQBLKS) dsn(&&SEQBLK) cyl space(5,5) unit(vio) new'.

Add json files to excludeFileList

We want to avoid scanning these files during DBB build.

Managing renaming of files

Calculation of changed files need to manage renaming:

$ git --no-pager diff b2ddff5f028dd1e9dba93090364053f1bd7e36c2 a95c9a04e36ef5c3be14d3395fd9d3cbcc8ddbe3
diff --git a/cobol/epscmort.cbl b/cobol/epscmor2.cbl
similarity index 100%
rename from cobol/epscmort.cbl
rename to cobol/epscmor2.cbl

Allow file level property overwrite for the ImpactResolution rules

There might arise some situations, where the impactResolution rules need to be able to overwrite the impactResolution rule.

dbb-zappbuild/utilities/ImpactUtilities.groovy

Lines 66 to 72 in d0b5352

	// perform impact analysis on changed file
	if (props.verbose) println "** Performing impact analysis on changed file $changedFile"
	ImpactResolver impactResolver = createImpactResolver(changedFile, props.impactResolutionRules, repositoryClient)
	// get excludeListe
	List<PathMatcher> excludeMatchers = createPathMatcherPattern(props.excludeFileList)

user build may connect to the webApp in case of error reporting

User build should not create a repository client connection, but it's something that can happen today if the rc of a step reports an error to capture in the build report. We should make sure that we never create a repository connection when doing a user build

Enhance scanOnly scenario

As of today, zAppBuild contains the capability to populate the DBB collections with the dependencies on the source level by using the option --scanOnly --fullBuild. In this case the source code is scanned for dependencies, but no outputs are created. In this context, there is a known limitation that the framework does not populate the output collection. However, the scanning process can be performed via LinkEditScanner which is included in the other build types.

We would like to enhance the zAppBuild sample framework to be able to also populate the output collection to support the scenario of migrating to Git/DBB without the need of running an initial full build when using static linkage.

Needs to include also renamedFiles

On a topic branch build, we should also consider the renamedFiles, see this section:

dbb-zappbuild/utilities/ImpactUtilities.groovy

Line 37 in d0b5352

(changedFiles, deletedFiles) = calculateChangedFiles(lastBuildResult)

Set the default gateway log level

Value 2 is the default.

zUnit Execution - Use StepName.DDStatementName in generated JCL

It would be a better style to use the StepName.DDStatementName like REPLAY.BZUPLAY when generating the zUnit Run JCL:

dbb-zappbuild/languages/ZunitConfig.groovy

Lines 68 to 72 in ada77da

	//BZUPLAY DD DISP=SHR,
	// DSN=${props.zunit_bzuplayPDS}(${playback})
	//BZURPT DD DISP=SHR,
	// DSN=${props.zunit_bzureportPDS}(${member})
	//*

zUnit Processing - add support for tests without playback file

Today, there is an assumption, that the test configuration has a playback file. This is not always the case.

Defect with fullBuilds due to the new scan capability

Issue: fullBuilds don't update the collections

Impact analysis should be more specific about the collections being involved

Right now we involve the 2 collections, but for source impact, we should limit the impact to the source collection and then involve the results with the binary collection. There's a risk of getting additional impacts today that are not relevant.

Fix BMS groovy file for maxRc

The bind.groovy utility methods need to validate the Gateway properties

The bind.groovy utility script uses the TSO Gateway to perform the bind. There are two gateway types supported: 'legacy' and 'interactive'. The default gateway type is 'legacy'. The type determines what additional gateway properties need to be required. The properties are located in /build-conf/build.properties.

Type	Required Properties
legacy	dbb.gateway.logLevel
interactive	dbb.gateway.procedureName dbb.gateway.accountNumber dbb.gateway.groupId dbb.gateway.regionSize dbb.gateway.logLevel

When zUnit is not enabled the zUnit tests source code source failed to build

If we have zUnit test in an application repository and have file.properties setup to build these files the build can fail if we want to disable zUnit.

IKJ56228I DATA SET NAZARE.TUTO.GENAPP.TEST.COBOL NOT IN CATALOG OR CATALOG CAN NOT BE ACCESSED    
Caught: com.ibm.dbb.build.BuildException: java.io.IOException: com.ibm.dbb.build.BuildException: BGZTK0016E An error occurred running BPXWDYN command 'free dd(SYS00019)'.
com.ibm.dbb.build.BuildException: java.io.IOException: com.ibm.dbb.build.BuildException: BGZTK0016E An error occurred running BPXWDYN command 'free dd(SYS00019)'.
	at com.ibm.dbb.build.CopyToPDS.execute(CopyToPDS.java:89)
	at com.ibm.dbb.build.IExecute$execute.call(Unknown Source)
	at BuildUtilities.copySourceFiles(BuildUtilities.groovy:75)
	at BuildUtilities$copySourceFiles$8.call(Unknown Source)
	at Cobol$_run_closure1.doCall(Cobol.groovy:44)
	at Cobol.run(Cobol.groovy:34)
	at Cobol$run.callCurrent(Unknown Source)
	at com.ibm.dbb.groovy.ScriptLoader._run(ScriptLoader.groovy:124)
	at com.ibm.dbb.groovy.ScriptLoader$_run$1.call(Unknown Source)
	at com.ibm.dbb.groovy.ScriptLoader.runScript(ScriptLoader.groovy:81)
	at build$_run_closure1.doCall(build.groovy:55)
	at build.run(build.groovy:47)

Even if we disable zUnit the zUnit test source code is in the build list but the TEST PDS is not created.

I think that when we disable zUnit during DBB build we should not build the zUnit tests source code.

@dennis-behm FYI.

zUnit Execution - Provide a test summary in console

zUnitConfig.groovy currently displays the job return code.

Provide a more detailed report presenting successful, failed and test in error.

Support loading binary files to target PDS

The copyMode of CopyToPDS needs to be adjusted when we're dealing with a binary file

Add support for external SYSLIB property (in Build Report)

What do we need that for ?
The BuildReport can then be used as input to the functionality described here: https://github.com/IBM/dbb/tree/master/Pipeline/RunIDZCodeReview.
If the external SYSLIBs isn't present (copybooks) in the BuildReport, we can't use the BuildReport as input.

Thanks

Store buildResultProperties in the BuildReport.json

Information about the build are stored as DBB build result properties at the build result and are available through the dbb build group at the dbb web app. The BuildReport.json contains a reference to the build report entry for traceability.

Let's make all the build result properties available in the BuildResult.json to simpify post-build processing like packaging.

Store GitUrl + CommitAuthor as a build result property

In GitUtils, we already extract the remote url of the repository.

We should store this info along with the Commit Author as Build Result properties.

Identification of branch name in detached mode

While git does not provide an easy way to retrieve the the origin branch in a detached head status, the branch name is retrieved today through git show :

dbb-zappbuild/utilities/GitUtilities.groovy

Line 57 in eb908f9

String cmd = "git -C $gitDir show -s --pretty=%D HEAD"

When using git references like some orchestrators automatically add, you might end up with several entries for the git show:

>> git show -s --pretty=%D HEAD
HEAD, origin/demo, refs/pipelines/78

Pick the branch referenced for origin.

ImpactBuild not recognizing certain excludeFileList patterns

Customer reported the following issue today with an impact build.

Using the excludeList in the image below. The Jenkins log should that *.CFG and *.cfg files are still included in the scanner phase. The "file not error" seems to be a secondary issue.

DanB. has reviewed this with me and asked that I open this defect.

2020-03-24T08:21:15.834Z] ** Build result created for BuildGroup:ortqx-Feature/firstmvp-123456789-F001 BuildLabel:build.20200324.052113.021 at https://gbjnmp01:9443/dbb/rest/buildResult/14350
[2020-03-24T08:21:15.834Z] ** --impactBuild option selected. Building impacted programs for application ortqx 
[2020-03-24T08:21:16.196Z] ** Getting current hash for directory workspace/ortqx
[2020-03-24T08:21:16.196Z] ** Storing workspace/ortqx : d1889656c3c0e74509932e2319d75b79391f1a01
[2020-03-24T08:21:16.196Z] ** Getting baseline hash for directory workspace/ortqx
[2020-03-24T08:21:16.196Z] ** Storing workspace/ortqx : de591d5d014f9a3f7ba3648e06377208e0049b05
[2020-03-24T08:21:16.196Z] ** Calculating changed files for directory workspace/ortqx
[2020-03-24T08:21:16.196Z] *** Changed files for directory workspace/ortqx:
[2020-03-24T08:21:16.196Z] *** ortqx/cfg/QQ1CNCA1.CFG
[2020-03-24T08:21:16.196Z] *** ortqx/cfg/QQ1CRUN2.cfg
[2020-03-24T08:21:16.196Z] *** ortqx/cfg/qq1cnca1.cfg
[2020-03-24T08:21:16.196Z] *** ortqx/cobol/EQQM0017.cbl
[2020-03-24T08:21:16.196Z] *** ortqx/cobol/SQQM0017.cbl
[2020-03-24T08:21:16.196Z] *** ortqx/cobol/qq1crun2.cbl
[2020-03-24T08:21:16.196Z] *** Deleted files for directory workspace/ortqx:
[2020-03-24T08:21:16.196Z] ** Updating collection ortqx-Feature/firstmvp-123456789-F001
[2020-03-24T08:21:16.196Z] **### *** Scanning file ortqx/cfg/QQ1CRUN2.cfg (workspace/ortqx/cfg/QQ1CRUN2.cfg)**
[2020-03-24T08:21:16.196Z] *** Logical file for ortqx/cfg/QQ1CRUN2.cfg =
[2020-03-24T08:21:16.196Z] {"dli":false,"lname":"QQ1CRUN2","file":"ortqx\/cfg\/QQ1CRUN2.cfg","cics":false,"language":"PLI","sql":false}
[2020-03-24T08:21:16.196Z] *** Scanning file ortqx/cfg/QQ1CNCA1.CFG (workspace/ortqx/cfg/QQ1CNCA1.CFG)
[2020-03-24T08:21:16.196Z] Caught: com.ibm.jzos.ErrnoException: EDC5129I No such file or directory. ERRNO=129 ERRNO2=0x53b006c
[2020-03-24T08:21:16.196Z] com.ibm.jzos.ErrnoException: EDC5129I No such file or directory. ERRNO=129 ERRNO2=0x53b006c
[2020-03-24T08:21:16.196Z] 	at com.ibm.jzos.FileAttribute.stat(Native Method)
[2020-03-24T08:21:16.196Z] 	at com.ibm.jzos.FileAttribute.getStat(FileAttribute.java:595)
[2020-03-24T08:21:16.196Z] 	at com.ibm.dbb.build.internal.Utils.retrieveHFSEncoding(Utils.java:119)
[2020-03-24T08:21:16.196Z] 	at com.ibm.dbb.dependency.DependencyScanner.scan(DependencyScanner.java:77)
[2020-03-24T08:21:16.196Z] 	at com.ibm.dbb.dependency.IDependencyScanner$scan.call(Unknown Source)
[2020-03-24T08:21:16.196Z] 	at ImpactUtilities$_updateCollection_closure6.doCall(ImpactUtilities.groovy:197)
[2020-03-24T08:21:16.196Z] 	at ImpactUtilities.updateCollection(ImpactUtilities.groovy:191)
[2020-03-24T08:21:16.196Z] 	at ImpactUtilities$updateCollection$2.callCurrent(Unknown Source)
[2020-03-24T08:21:16.196Z] 	at ImpactUtilities.createImpactBuildList(ImpactUtilities.groovy:50)
[2020-03-24T08:21:16.196Z] 	at ImpactUtilities$createImpactBuildList$0.call(Unknown Source)
[2020-03-24T08:21:16.196Z] 	at build.createBuildList(build.groovy:404)
[2020-03-24T08:21:16.196Z] 	at build.run(build.groovy:36)
[2020-03-24T08:21:17.366Z] ** Build finished
[2020-03-24T08:21:17.488Z] [Pipeline] }

Simple way to add existing libraries to syslib concatenation

Current state: If you would like to add a set of libraries to the syslib concatenation for the compiler or the linker, it requires to modify the language script and probably introduce a new dataset property.

We would like to make it simpler to add existing libraries to the concatenation in a very generic way. Possible use cases are such as:

• adding a new Copy Library or Link library for 3rd party software.
• during the transition period in a phased migration to git, to add the libraries of the legacy system and make copybooks available to the dbb build.
• possibility to pick up copybooks from other application components managed in git, which have published their copybooks into a shared PDS library.

Ability to skip impact analysis for a set of files

In some cases, users would like to skip the impact analysis phase for a set of copybooks, include or macros files. This could be because this file is used in basically all the programs, and changes are assumed to be backward compatible.