GoCD plugin to keep pipelines and environments configuration in source-control in YAML. See this document to find out what are GoCD configuration repositories.
This is the second GoCD configuration plugin, which enhances some of shortcomings of JSON configuration plugin
- Format is concise. Unlike JSON, there are no unnecessary quotations
"
, brackets{}
and commas,
- Tries to enforce correctness where possible. By using maps instead of lists and shorter object graphs.
- Allows to have multiple pipelines and environments in single file. But doesn't force it.
- Comments in configuration files - YAML supports comments, so you can explain why pipeline/environment it is configured like this.
- Setup
- Example configuration
- File pattern
- Validation
- Format reference
- Format version
- Issues and questions
- Development
- License
Step 1: GoCD versions newer than 17.8.0
already have the plugin bundled. You don't need to install anything.
If you're using GoCD version older than 17.8.0, you need to install the plugin in the GoCD server. Download it from
the releases page and place it on the GoCD server in
plugins/external
directory.
Step 2: Follow the GoCD documentation to add a new configuration repository.
You can use the example repository at: https://github.com/tomzo/gocd-yaml-config-example.git
.
In your config repo (tomzo/gocd-yaml-config-example.git
in this case), ensure that your GoCD yaml config is suffixed with .gocd.yaml
. Any file ending in .gocd.yaml
is picked up by the plugin. Give it a minute or so for the polling to happen. Once that happens, you should see your pipeline(s) on your dashboard.
More examples are in test resources.
#ci.gocd.yaml
format_version: 9
environments:
testing:
environment_variables:
DEPLOYMENT: testing
secure_variables:
ENV_PASSWORD: "s&Du#@$xsSa"
pipelines:
- example-deploy-testing
- build-testing
pipelines:
mypipe1: # definition of mypipe1 pipeline
group: mygroup # note that the group name can contain only of alphanumeric & underscore characters
display_order: 10
label_template: "${mygit[:8]}"
lock_behavior: none
parameters: # list of parameters that can be configured for a pipeline
param1: value1
materials:
mygit: # this is the name of material, the name can contain only of alphanumeric & underscore characters
# keyword git says about type of material and url at once
git: http://my.example.org/mygit.git
branch: ci
myupstream: # this name does not matter, but there should be no 2 materials with the same name
# type is optional here, material type is implied based on presence of pipeline and stage fields
# type: dependency
pipeline: pipe2
stage: test
stages: # list of stages in order
- build: # name of stage
clean_workspace: true
jobs:
csharp: # name of the job
resources:
- net45
artifacts:
- build:
source: bin/
destination: build
- test:
source: tests/
destination: test-reports/
- test:
source: coverage.xml
tabs:
report: test-reports/index.html
tasks: # ordered list of tasks to execute in job csharp
- fetch:
pipeline: pipe2
stage: build
job: test
source: test-bin/
destination: bin/
- exec: # indicates type of task
command: make
arguments:
- "VERBOSE=true"
# shorthand for script-executor plugin
- script: ./build.sh ci
The default pattern is **/*.gocd.yaml
and **/*.gocd.yml
, which will recursively search the entire repository for all files ending with .gocd.yaml
or .gocd.yml
.
You can set a custom file pattern per configuration repository using the GoCD configuration UI:
Or in the config XML using <configuration>
:
<config-repos>
<config-repo pluginId="yaml.config.plugin" id="repo1">
<git url="https://github.com/tomzo/gocd-yaml-config-example.git" />
<configuration>
<property>
<key>file_pattern</key>
<value>pipeline.gocd.yaml</value>
</property>
</configuration>
</config-repo>
</config-repos>
You can validate if proposed GoCD YAML changes will be accepted by the server. Currently, 2 options are available:
You may find this introductory blog post useful.
There is an ongoing effort to allow in-depth validation of configuration before pushing configuration to the source control. This is provided by GoCD's preflight API and gocd-cli.
You have several options to configure validation tools on your workstation:
- If you have a local docker daemon, use the gocd-cli-dojo image. Follow the setup instructions in the image readme.
- If you don't want to use docker, you'll need to setup
gocd-cli
on your host.
Either way you'll have gocd
binary in your PATH
or inside the docker container.
This will check general validity of the yaml file, without talking to the GoCD server:
gocd configrepo syntax --yaml pipeline.gocd.yaml
This command will parse and submit your yaml file to the configured GoCD server.
gocd configrepo preflight --yaml -r dotnet-dojo pipeline.gocd.yaml
Where -r
is the configuration repository id, which you have earlier configured on GoCD server. You can check it on config repos page of your GoCD server, at /go/admin/config_repos
. It is in the upper left corner of each config repo.
For new repositories, when you haven't added config repo to the server yet, you can also check if it is correct, just skip the -r
argument.
gocd configrepo preflight --yaml pipeline.gocd.yaml
See official GoCD XML configuration reference for details about each element. Below is a reference of format supported by this plugin. Feel free to improve it!
- Format version
- Environment
- Environment variables
- Parameters
- Pipeline
- Stage
- Job
- Tasks
- Materials
- Secure variables
- YAML Aliases
Please note that it is now recommended to declare format_version
in each gocd.yaml
file, consistent across all your files.
Supports format_version
value of 9
. In this version, support of ignore_for_scheduling
for dependency materials has been added. Setting this attribute will skip scheduling the pipeline when the dependency material has changed.
Using a newer format_version
includes all the behavior of the previous versions too.
format_version: 9
pipelines:
...
environments:
Supports format_version
value of 7
and 8
. In version 7
, support for properties has been removed. In version 8
, support for mingle
as a tracking tool has been removed.
Using a newer format_version
includes all the behavior of the previous versions too.
Supports format_version
value of 6
. In this version, support of allow_only_on_success
attribute for approval in stage has been added. Setting this attribute to true
will allow the stage to be manually triggered only if the previous stage has passed successfully.
Using a newer format_version
includes all the behavior of the previous versions too.
format_version: 6
pipelines:
...
environments:
Supports format_version
value of 5
. In this version, support of username
and encrypted_password
for git and hg material has been added. In addition to that, hg will also support branch
attribute.
Using a newer format_version
includes all the behavior of the previous versions too.
format_version: 9
pipelines:
...
environments:
Supports format_version
value of 4
. In this version, support has been added to control the display order of pipelines.
This server version also supports format_version
of 3
and 2
. Using a newer format_version
includes all the behavior of the previous versions too.
format_version: 4
pipelines:
...
environments:
Supports format_version
value of 3
. In this version fetch artifact format was changed to include artifact_origin
.
This server version also supports format_version
of 2
. Using a newer format_version
includes all the behavior of the previous versions too.
format_version: 3
pipelines:
...
environments:
Supports format_version
value of 2
. In this version pipeline locking behavior was changed.
format_version: 2
pipelines:
...
environments:
Supports format_version
value of 1
. This is the initial version.
format_version: 1
pipelines:
...
environments:
A minimal pipeline configuration must contain:
- pipeline name - as a key in hash
- materials - a hash
- stages - a list or a
template
- as a key in hash
mypipe:
group: mygroup
materials:
mygit:
git: http://example.com/mygit.git
stages:
- build:
jobs:
build:
tasks:
- exec:
command: make
All elements available on a pipeline object are:
group
label_template
- locking
- tracking_tool or
mingle
- timer
- environment_variables
- parameters
secure_variables
- materials
- stages
template
pipe2:
group: group1
label_template: "foo-1.0-${COUNT}"
lock_behavior: none
tracking_tool:
link: "http://your-trackingtool/yourproject/${ID}"
regex: "evo-(\\d+)"
timer:
spec: "0 15 10 * * ? *"
environment_variables:
DEPLOYMENT: testing
secure_variables:
ENV_PASSWORD: "s&Du#@$xsSa"
materials:
...
stages:
...
mypipe:
group: group1
label_template: "foo-1.0-${COUNT}"
lock_behavior: none
parameters:
param1: value
materials:
mygit:
git: http://example.com/mygit.git
template: template1
Please note:
- Pipeline declares a group to which it belongs
When format_version
is 4
(see above), the order of display of pipelines on the GoCD dashboard can be influenced by setting the display_order
property.
- This is an integer property and the pipelines in a pipeline group will be ordered by this value.
- The default value for this property is
-1
. - Pipelines defined in GoCD's config XML will also default to -1.
- If multiple pipelines have the same
display_order
value, their order relative to each other will be indeterminate.
mypipeline1:
group: group1
display_order: 10
mypipeline2:
group: group1
display_order: -10
In the above example, since both pipelines are in the same group, pipeline2
will be shown ahead of pipeline1
. If any pipelines are defined in the GoCD config XML, then they will appear in between these two pipelines.
tracking_tool:
link: "http://your-trackingtool/yourproject/${ID}"
regex: "evo-(\\d+)"
timer:
spec: "0 15 10 * * ? *"
only_on_changes: yes
See Quartz about writing cron-like schedules.
For GoCD >= 17.12 and format_version: 2
and above:
lock_behavior: none
Where lock_behavior
is defined as in GoCD documentation can be one of:
lockOnFailure
- same aslocking: true
unlockWhenFinished
none
- samelocking: false
For GoCD < 17.12 and format_version: 1
:
locking: on
Where locking
is a boolean.
A minimal stage must contain jobs:
element or tasks:
in single-job stage case.
build:
jobs:
firstJob:
...
secondJob:
...
A custom stage:
test:
fetch_materials: yes
keep_artifacts: yes
clean_workspace: yes
approval:
type: manual
roles:
- manager
users:
- john
environment_variables:
TEST_NUM: 1
secure_variables:
PASSWORD: "!@ESsdD323#sdu"
jobs:
one:
...
two:
...
- When
keep_artifacts
is set totrue
GoCD will never delete artifacts in this stage. This is equivalent of XML'sartifactCleanupProhibited
. By default this isfalse
.
Stage can have approval,
which is success
by default. There are 2 ways to declare approval:
approval: manual
If you need to set associated users or roles:
approval:
type: manual
allow_only_on_success: true
roles:
- manager
users:
- john
You can set allow_only_on_success
to allow manual trigger only if the previous stage run is successful. The default value is false
.
Job is a hash starting with jobs name:
test:
timeout: 5
run_instances: 7
environment_variables:
LD_LIBRARY_PATH: .
tabs:
test: results.xml
resources:
- linux
artifacts:
- test:
source: src
destination: dest
- build:
source: bin
- external:
id: docker-release-candidate
store_id: dockerhub
configuration:
options:
Image: gocd/gocd-demo
Tag: v${GO_PIPELINE_LABEL}
secure_options:
some_secure_property: "!@ESsdD323#sdu"
tasks:
...
Note: timeout is added since 0.2.0 version of yaml plugin
Job configuration may define elastic agents profile id, as such:
elastic_profile_id: "docker.unit-test"
It MUST NOT be specified along with resources
.
Available in GoCD server since v16.12.0, yaml plugin 0.4.0.
There are 3 types of artifacts recognized by GoCD. Build
and Test
artifacts are stored on the GoCD server.
The source and the destination of the artifact that should be stored on the GoCD server must be specified.
- build:
source: bin
destination: binaries
- test:
source: reports
destination: test-reports
Artifacts of type external
are stored in an artifact store outside of GoCD.
The external artifact store's configuration must be created in the main GoCD config. Support for external artifact store config to be checked in as yaml is not available.
The external artifact store is referenced by the store_id
. The build specific artifact details that the artifact plugin needs to publish the artifact is provided as configuration
.
- external:
id: docker-release-candidate
store_id: dockerhub
configuration:
options:
Image: gocd/gocd-demo
Tag: v${GO_PIPELINE_LABEL}
secure_options:
some_secure_property: "!@ESsdD323#sdu"
S3 plugin 2.x example usage:
- external:
id: pkg
store_id: s3-eu-west-1
configuration:
options:
Source: installers/target/
Destination: ${GO_ARTIFACT_LOCATOR}
Part of job object can be number of job to runs:
run_instances: 7
Or to run on all agents:
run_instances: all
Tabs are a hash with <tab-name>: <path>
pairs.
Path should exist in GoCD servers artifacts.
tabs:
tests: test-reports/index.html
gauge: functional-reports/index.html
DEPRECATION NOTICE: Since GoCD version 19.9 and format_version 7, properties are no longer supported
Job can have properties, declared as a hash:
properties:
cov1: # this is the name of property
source: test.xml
xpath: "substring-before(//report/data/all/coverage[starts-with(@type,\u0027class\u0027)]/@value, \u0027%\u0027)"
performance.ind1.mbps:
source: PerfTestReport.xml
xpath: "//PerformanceSuiteReport/WriteOnly/MBps"
A common use case is that stage has only one job. This plugin provides a shorthand
to declared such stages - just omit the jobs:
and job name from configuration tree.
You can then declare job and stage options on the same (stage) level:
stages:
- build:
approval: manual
resources:
- cpp
tasks:
- exec:
command: make
Above configuration declares build
stage with build
job which executes make
task.
Minimal configuration of a git pipeline material:
- material name is
mygit
- git repository url is
http://example.com/mygit.git
mygit:
git: http://example.com/mygit.git
Above can be also written more explicitly:
mygit:
type: git
url: http://example.com/mygit.git
More customized git material is possible:
gitMaterial1:
git: "http://my.git.repository.com"
branch: feature12
blacklist:
- externals/**/*.*
- tools/**/*.*
destination: dir1
auto_update: false
shallow_clone: true
Since GoCD >= 16.7.0
whitelist is also supported,
you can specify whitelist
instead of blacklist
, as such
gitMaterial1:
git: "[email protected]"
branch: "feature12"
whitelist:
- src/**/*.*
For GoCD >= 19.4.0 and format_version: 5
and above:
You are advised to utilize username
and encrypted_password
for passing in material credentials as:
gitMaterial1:
git: "http://my.git.repository.com"
branch: feature12
username: my_username
encrypted_password: encrypted_value
- Instead of
encrypted_password
you may specifypassword
butencrypted_password
makes more sense considering that the value is stored in SCM. - Specifying credentials both in
attributes
andurl
will result in a validation error e.g.INVALID MERGED CONFIGURATION Number of errors: 1+ 1. Ambiguous credentials, must be provided either in URL or as attributes.;; - For Config Repo: https://your.config.repo.url at cbb047d78c239ab23b9565099e800c6fe4cc0anc
For details about each option, see GoCD XML reference
svnMaterial1:
svn: "http://svn"
username: "user1"
encrypted_password: "encrypted_value"
check_externals: true
blacklist:
- tools
- lib
destination: destDir1
auto_update: false
Instead of encrypted_password
you can specify password
.
hgMaterial1:
hg: repos/myhg
blacklist:
- externals
- tools
destination: dir1
auto_update: false
username: my_username
encrypted_password: encrypted_value
branch: feature
For GoCD >= 19.4.0 and format_version: 5
and above:
You are advised to utilize username
and encrypted_password
for passing in material credentials as:
hgMaterial1:
hg: repos/myhg
username: my_username
encrypted_password: encrypted_value
- Instead of
encrypted_password
you may specifypassword
butencrypted_password
makes more sense considering that the value is stored in SCM. - Specifying credentials both in
attributes
andurl
will result in a validation error e.g.INVALID MERGED CONFIGURATION Number of errors: 1+ 1. Ambiguous credentials, must be provided either in URL or as attributes.;; - For Config Repo: https://your.config.repo.url at cbb047d78c239ab23b9565099e800c6fe4cc0anc
In addition to that, you can also leverage branch
attribute to specify the branch for material
hgMaterial1:
hg: repos/myhg
branch: feature
p4Material1:
p4: "host.domain.com:12345"
username: johndoe
encrypted_password: encrypted_value
use_tickets: false
view: |
//depot/external... //ws/external...
//depot/tools... //ws/external...
blacklist:
- externals
- tools
auto_update: false
Instead of encrypted_password
you can specify password
.
TODO: - not supported by yaml plugin yet
myPluggableGit:
scm: someScmGitRepositoryId
destination: destinationDir
blacklist:
- dir1
- dir2
Since GoCD >= 19.2.0
defining new pluggable materials that are not defined
in the GoCD server is supported.
myPluggableGit:
plugin_configuration:
id: plugin_Id
version: 1 #plugin version
options:
url: [email protected]:tomzo/gocd-yaml-config-plugin.git
destination: destinationDir
blacklist:
- dir1
- dir2
This is a convenience for shorter and more consistent material declaration. When configuration repository is the same as one of pipeline materials, then you usually need to repeat definitions in XML and in JSON, for example:
materials:
foo:
git: "https://github.com/tomzo/gocd-json-config-example.git",
branch: ci
And in server XML:
<config-repos>
<config-repo pluginId="yaml.config.plugin" id="repo1">
<git url="https://github.com/tomzo/gocd-json-config-example.git" branch="ci" />
</config-repo>
</config-repos>
Notice that url and branch is repeated. This is inconvenient in case when you move repository, because it requires 2 updates, in code and in server XML.
Using configrepo
material type, above repetition can be avoided,
last example can be refactored into:
materials:
foo:
type: configrepo
Server interprets configrepo
material in this way:
Clone the material configuration of the repository we are parsing as is in XML and replace name, destination and filters (whitelist/blacklist), then use the modified clone in place of
configrepo
material.
To add a dependency on another pipeline stage:
mydependency:
pipeline: upstream-pipeline-1
stage: test
ignore_for_scheduling: false
Note: mydependency
is the name of material - it must be unique
myapt:
package: apt-repo-id
Every task is a hash starting with its type.
Type can be exec
, ant
, nant
, rake
, fetch
, plugin
or script
.
<type>:
<task-prop1>: <prop1-value>
Optionally any task can have run_if
and on_cancel
.
run_if
is a string. Valid values arepassed
,failed
,any
on_cancel
is a task object. Same rules apply as to tasks described on this page. See rake example.
exec:
run_if: any
working_directory: dir
command: make
arguments:
- -j3
- docs
- install
ant:
build_file: mybuild.xml
target: compile
run_if: passed
nant:
run_if: passed
working_directory: "script/build/123"
build_file: FilePath
target: Build
nant_path: NantExe
A minimal rake task with default values is very short
rake:
A complete rake example:
rake:
run_if: passed
working_directory: sample-project
build_file: SomeRakefile
target: build
on_cancel:
rake:
working_directory: sample-project
build_file: CancelRakefile
target: cancel
fetch:
run_if: any
artifact_origin: gocd # default value
pipeline: pipe2
stage: upstream_stage
job: upstream_job
is_file: yes
source: result
destination: test
artifact_origin
if not specified then, defaults togocd
, which is consistent with previousformat_version
1 and 2.pipeline
if not specified, then defaults to current pipeline.
fetch:
run_if: any
artifact_origin: external
pipeline: pipe2
stage: upstream_stage
job: upstream_job
artifact_id: upstream_artifact_id
configuration:
options:
DestOnAgent: foo
secure_options:
some_secure_property: "!@ESsdD323#sdu"
S3 plugin 2.x example usage:
- fetch:
artifact_origin: external
stage: plan
job: plan
artifact_id: plan
configuration:
options:
IsFile: true
Destination: terraform
Source: deployment.tfplan
plugin:
options:
ConverterType: jsunit
secure_options:
password: "ssd#%fFS*!Esx"
run_if: failed
configuration:
id: xunit.converter.task.plugin
version: 1
Because script-executor plugin requires quite a lot of boiler-plate configuration there is a shorthand for defining tasks with it:
script: ./build.sh compile
Above is equivalent of
plugin:
options:
script: ./build.sh compile
configuration:
id: script-executor
version: 1
You can declare multi-line scripts too:
script: |
./build.sh compile
make test
Above executes a 2-line script:
./build.sh compile
make test
If you want to execute a single long line script, but break it into multiple lines in YAML,
you can do this with >
as such:
script: >
./build.sh compile &&
make test
Above executes a single line script:
./build.sh compile && make test
NOTE: The agents should be a guid, which is currently impossible to get for user
testing:
environment_variables:
DEPLOYMENT: testing
secure_variables:
ENV_PASSWORD: "s&Du#@$xsSa"
pipelines:
- example-deploy-testing
- build-testing
agents:
- 123
Environment variables can be declared in Environments, Pipelines, Stages and Jobs.
In YAML you have 2 keywords for secure (encrypted) variables and standard variables.
environment_variables:
DEPLOYMENT: testing
FOO: bar
secure_variables:
# this value is encrypted by Go's private key (Note in 16.7.0 there is no easy way to obtain such value yet)
MY_PASSWORD: "s&Du#@$xsSa"
Parameters can be declared at the pipeline level.
parameters:
param1: value1
production: no
For versions of GoCD >= 17.1:
See the encryption API.
For versions of GoCD <= 16.12:
There is no easy way to generate obtain encrypted value from GoServer, alternatively you can login into go-server and execute the following command to generate encrypted value
sudo -u go bash -c 'echo -n 'YOUR-INPUT' | openssl enc -des-cbc -a -iv 0 -K $(cat /etc/go/cipher)'
Among all configuration elements there are boolean values, which can be defined using any of the keywords below (as in yaml specs):
- true -
y|Y|yes|Yes|YES|true|True|TRUE|on|On|ON
- false -
n|N|no|No|NO|false|False|FALSE|off|Off|OFF
YAML Aliases (specification) are supported and provide a way to avoid duplication.
Aliases can be defined anywhere in the configuration as long as they are valid configuration elements.
- exec:
command: make
arguments:
- clean
- &verbose_arg "VERBOSE=true" # define the alias
- exec:
command: make
arguments:
- world
- *verbose_arg # use the alias
There is also a dedicated top-level common
section which allows you to have all aliases in one place and where you don't need to worry about correct placement within the configuration.
format_version: 4
common:
verbose_arg: &verbose_arg "VERBOSE=true"
build_tasks: &build_tasks
- exec:
command: make
arguments:
- clean
- exec:
command: make
arguments:
- world
pipelines:
pipe1:
stages:
- build:
jobs:
build:
tasks: *build_tasks
test:
tasks:
- *build_tasks # task list aliases can also be mixed with additional tasks in the same job
- exec:
command: make
arguments:
- test
Similarly stages can be re-used:
format_version: 4
common:
build_stages: &build_stages
- stage1:
jobs:
# ...
- stage2:
jobs:
# ...
pipelines:
pipeline_one:
stages: *build_stages
pipeline_two:
stages:
- *build_stages
- stage3:
jobs:
# ...
- If you have questions on usage, please ask them on the gitter chat room dedicated for configrepo-plugins
- If you think there is a bug, or you have an idea for a feature and you are not sure if it's plugin's or GoCD fault/responsibity, please ask on the chat first too.
Please note this brief overview of what is done by the plugin:
- parsing yaml into json when GoCD server asks for it.
And this is done by the GoCD server:
- complex logic merging multiple config repo sources and XML
- validation of pipelines/stages/jobs/tasks domain
- any UI rendering
To build and test this plugin, you'll need java jdk >= 8.
If you have local java environment, then you may run all tests and create a ready to use jar with:
./gradlew test jar
You don't need to setup java on your host, if you are fine with using docker and Dojo. This is actually how our GoCD builds the plugin:
dojo "gradle test jar"
Assuming you already have a working docker, you can install dojo by placing the executable anywhere on the PATH.
DOJO_VERSION=0.6.0
# On Linux
wget -O dojo https://github.com/kudulab/dojo/releases/download/${DOJO_VERSION}/dojo_linux_amd64
# On OSX
# wget -O dojo https://github.com/kudulab/dojo/releases/download/${DOJO_VERSION}/dojo_darwin_amd64
sudo mv dojo /usr/local/bin
sudo chmod +x /usr/local/bin/dojo
Then enter a docker container with java and gradle pre-installed, by running following command at the root of the project:
dojo
We use semantic versioning.
If you are submitting a new feature then please run a major version bump by
./tasks.sh set_version 0.X.0
If you are submitting a fix, then do not change any versions as patch bump is made right after each release.
There are examples of yaml partials and their resulting json to be sent to GoCD server. If something is not working right we can always add a new case covering exact yaml that user has and json that we expect on server side.
Copyright 2019 Tomasz Sętkowski
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.