qarik-group / concourse-tutorial

Learn to use https://concourse-ci.org with this linear sequence of tutorials. Learn each concept that builds on the previous concept.

Concourse Tutorial

Learn to use https://concourse-ci.org with this linear sequence of tutorials. Learn each concept that builds on the previous concept.

Read the tutorial at https://concoursetutorial.com

Thanks

Thanks to Alex Suraci for inventing Concourse CI, and to Pivotal and VMWare for sponsoring him and a team of developers to work since 2014.

At Stark & Wayne we started this tutorial as we were learning Concourse in early 2015, and we've been using Concourse in production since mid-2015 internally and at nearly all client projects.

Thanks to everyone who has worked through this tutorial and found it useful. I love learning that you're enjoying the tutorial and enjoying Concourse.

Thanks for all the pull requests to help fix regressions with some Concourse versions that came out with "backwards incompatible change".

Getting Started

Read the tutorial at https://concoursetutorial.com

Local development of tutorial

This tutorial is built using mkdocs. Please make sure you have python3 and pip3 installed before running mkdocs and they are refenced as python and pip respectively. . Once installed, you can continuously build and serve the tutorial locally with:

pip install mkdocs
pip install pymdown-extensions
pip install mkdocs-material

mkdocs serve

Manual deployment

mkdocs build
cd site
gsutil -m cp -r . gs://concoursetutorial-com-website
gsutil -m rsync -r -x '\.git.*' . gs://concoursetutorial-com-website

View the site and live changes at https://localhost:8000.

concourse-tutorial's Issues

fly save-target does not exist in 0.66.1

Thanks for this tutorial!

You now use fly login to set aliases for targets but when running on a local Virtualbox Concourse setup there isn't any auth set up so running fly -t tutorial login -c 'http://192.168.100.4:8080/' is a no-op.

Solutions:

• use the IP:port of the concourse server - this is pretty verbose
• set up auth on the server - this is pretty advanced right out of the gate
• create another Vagrant box with auth set up - could be time consuming to configure.

The specified host network collides with a non-hostonly network!

While trying to run the tutorial, I got this error

The specified host network collides with a non-hostonly network!
This will cause your specified IP to be inaccessible. Please change
the IP or name of your host only network so that it no longer matches that of
a bridged or non-hostonly network.

when I ran the
vagrant up

How can I resolve this issue?

Thanks!

always not authorized. run the following to log in

fly -t tutorial login -c http://10.3.11.17:31868/ -u atc -p atc

but i can't login

Parameterized pipelines Readme typo

The Readme gives the example of parameterized a private key with the code:

github-private-key: |-
  -----BEGIN RSA PRIVATE KEY-----
  MIIEpQIBAAKCAQEAuvUl9YUlDHWBMVcuu0FH9u2gSi83PkL4o9TS+F185qDTlfUY
  fGLxDo/bn8ws8B88oNbRKBZR6yig9anIB4Hym2mSwuMOUAg5qsA9zm5ArXQBGoAr
  ...
  iSHcGbKdWqpObR7oau2LIR6UtLvevUXNu80XNy+jaXltqo7MSSBYJjbnLTmdUFwp
  HBstYQubAQy4oAEHu8osRhH1VX8AR/atewdHHTm48DN74M/FX3/HeJo=
  -----END RSA PRIVATE KEY-----

This causes an error when setting the pipeline. The first line should be github-private-key: | instead of github-private-key: |-

Can't find and start virtual box.

I can't seem to get started... I don't understand where to look for the box. I thought it would be pulled down by clone this git repository.
Can anybody interpret for me?
Rex

$ vagrant box add concourse/lite --box-version 2.2.1
Found VERSION file, using concourse/lite version 2.2.1
The box 'concourse/lite' could not be found or
could not be accessed in the remote catalog. If this is a private
box on HashiCorp's Atlas, please verify you're logged in via
vagrant login. Also, please double-check the name. The expanded
URL and error message are shown below:

URL: ["https://atlas.hashicorp.com/concourse/lite"]
Error:

Confusing benefit/downside section in tutorial 05

There is a benefit and a downside to abstracting tasks into YAML files outside of the pipeline.

One benefit is that the behavior of the task can be kept in sync with the primary input resource (for example, a software project with tasks for running tests, building binaries, etc).

One downside is that the pipeline.yml no longer explains exactly what commands will be invoked. Comprehension of pipeline behavior is potentially reduced.

But one benefit of extracting inline tasks into task files is that pipeline.yml files can get long and it can be hard to read and comprehend all the YAML. Instead, give tasks long names so that readers can understand what the purpose and expectation of the task is at a glance.

But one downside of extracting inline tasks into files is that fly set-pipeline is no longer the only step to updating a pipeline.

At first it says there's "a benefit and a downside" but then lists two of each. Another confusing part is that the second benefit reads like a downside. Am I reading this correctly?

Finish lesson 60

Need to confer with @drnic to clarify his intent, update manifests for current concourse, and write the README.

Git tag with concourse version

It might be a little hard to accomplish this, but I think it would be really useful every time we do a through pass on the concourse-tutorial we create a git tag that matches concourse's version.

That way anyone following the tutorial will have an idea what version of concourse to use.

Set up CI

Right now the only way to check the the tutorial is running through it manually. Let's practice what we preach and use Concourse.

Work is underway in a branch.

concourse api example with curl does not work

the example to trigger the build from curl [1] does not work:

$ curl http://192.168.100.4:8080/pipelines/main/jobs/job-hello-world/builds -X POST
404 page not found

maybe @vito knows why? the concourse api does not seem to be documented on the concourse.ci pages that i could find.

[1] https://github.com/starkandwayne/concourse-tutorial/blob/master/README.md#05---trigger-a-job-via-the-concourse-api

Example 05 fails to do resource check on Docker

I tried Example 5 [05_pipeline_task_hello_world] and it fails with following error;

resource script '/opt/resource/check []' failed: exit status 1
stderr:
failed to ping registry: 2 error(s) occurred:

• ping https: Get https://registry-1.docker.io/v2: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
• ping http: Get https://registry-1.docker.io/v2: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

11 - Job never executed

The directions never say to run the job, and it doesn't start automatically. Not sure if it's necessary, but I spent a few minutes debugging this since all previous jobs have been executed.

Paramaterise task files

Do you have any examples of how to pass params to task files to use them within the task?

# task_file.yml
image_resource:
  type: docker-image
  source:
    repository: {{docker-base-image}}
    tag: {{docker-base-version}}

This works when tasks are inline but when you split it into a file I cant find any docs on how to do this.
The task docs (https://concourse.ci/running-tasks.html) state you can define a params section but I cant see an example of how to use them.

all fly -t commands seem broken with 0.54.0

with vagrant you don't need to specify, but you may want to consider using the --target http://192.168.100.4:8080 syntax unless @vito or @xeobus explain what happened.

05 Pipeline Fails with "file not found" after the step "using version of resource found in cache."

I am running the Linux version of the tutorial. After reinstalling multiple times (assuming I had made an error), I still get "file not found" after the step "using version of resource found in cache."

Looking at the output, I notice this:
3172f4 Merge pull request #106 from starkandwayne/windows-users

It says "windows-users." Perhaps somewhere in the pipeline there is a "hidden" reference to "windows?" I searched through the text of all the example files and I did not find any occurrence of "windows" and I did verify that each of the tutorial yml files does contain linux. Since I cannot complete this step, I do not know how to proceed with the rest of the tutorial.

If anybody has seen a similar issue, please reply.

Thanks,

Jo3

Do you have a tutorial job to dry-run a task and waiting for unblock?

I need to run several tasks to apply the terraform changes.

The tasks will be:

terraform plan ( a dry-run)
block (waiting for unblock, y/n)
terraform apply (apply the changes)

So every time, when the resource (git) on branch master detects a new commit, it will trigger the job terraform plan. Then wait (block).

I need to read the output from the job terraform plan if the configuration changes are fine. Then I can click a button (or input y/n) to unblock the pipeline and allow the next job run terraform apply.

I have the same request to run ansible job with --check as dry-run.

For your reference, you can read this document (https://buildkite.com/docs/pipelines/block-step) , it introduces the block function in buildkite

10 deploy app readme?

What does a "credentials.yml" looks like?

04 create basic step 4 tutorial.

it show long time to perform the "checking pipeline is not paused" step, no error, just take long time to perform

02 task inputs inputs_required

Thanks for sharing this tutorial, it's really really cool !!!

There's a little problem with inputs_required in 02_task_inputs though:

07/03 2016 11:48:20 jpic@concourse ~/concourse-tutorial/02_task_inputs  (master)
$ fly -t tutorial execute -c no_inputs.yml
targeting http://localhost:8080

executing build 21
initializing with docker:///busybox
running ls -al
total 8
drwxr-xr-x    2 root     root          4096 Mar  7 11:48 .
drwxr-xr-x    3 root     root          4096 Mar  7 11:48 ..
succeeded

07/03 2016 11:48:28 jpic@concourse ~/concourse-tutorial/02_task_inputs  (master)
$ fly -t tutorial e -c inputs_required.yml -i some-important-input=.
targeting http://localhost:8080

executing build 22
container: start: exit status 2
errored
^C
aborting...

I'm a newbie and if at least you could give me some clues to debug this it would help a lot !

Thanks in advance

#44 BOSH Stemcells & Releases how do i get the latest version of bosh-stemcell-aws

hi friends,
I had a problem while getting the latest version of bosh-stemcell-aws.
I excepted 3312.38 latest version but I got 3263.28 version of bosh-aws-xen-ubuntu-trusty-go_agent.

I didn't modify anything in pipeline.yml in 44_bosh_io path.
resources:

• name: bosh-stemcell-aws
  type: bosh-io-stemcell
  source:
  name: bosh-aws-xen-ubuntu-trusty-go_agent

Is there a way to get the latest version 3312.38 ?
I'm using fly CLI 3.5.0 version.
Thank you for helping me in advance.

12 - golang-test-web-app fails to find martini package

golang-test-web-app
main.go:7:2: cannot find package "github.com/go-martini/martini" in any of:
    /usr/src/go/src/github.com/go-martini/martini (from $GOROOT)
    /go/src/github.com/go-martini/martini (from $GOPATH)

strange character in the terminal

I use git-for-windows(1.9.5), and the shell is based on mingw32. When I run the tutorial, strange character appears like below. Do you have any ideas? thanks!

Same thing happened, when I use cmd of windows:

Memory being eaten up

Hello,

Thanks for this awesome tutorial, really like it!

I'm currently thinking about using concourse as a tool for our company, but for some reason the vagrant machine eats up all my memory when running the tasks about Publishing Output. I doubt this is the expected behaviour.

I think the version of my softs are pretty up-to-date:

# Linux kernel
> uname -a
Linux dragonfly 4.5.5-1-MANJARO #1 SMP PREEMPT Thu May 19 17:20:19 UTC 2016 x86_64 GNU/Linux

# Go, Vagrant & Chromium
> go version
go version go1.6.2 linux/amd64
> vagrant version
Installed Version: 1.8.1
Latest Version: 1.8.1

# Regarding the latest commit I have & version
> git show | head -n1
commit 39fee8cd012b36b6bcc0fa826894186d2d36a17a
> cat VERSION
1.2.0
> chromium --version
Chromium 51.0.2704.63 

If you have already faced this issue, or know where this is coming from it would be appreciated :)
Didn't see anything matching memory problem on the repository's past issues.

It might eventually be similar to: concourse/concourse#334
I think that if it's a known issue in the tutorial, it might be good to warn the users about it.

Cheers!

vagrant up stalls

when typing vagrant up, the machine hangs trying to ssh in to the box.

Found VERSION file, using concourse/lite version 2.2.1
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'concourse/lite'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'concourse/lite' is up to date...
==> default: Setting the name of the VM: concourse-tutorial_default_1480446309370_92086
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
    default: Adapter 2: hostonly
==> default: Forwarding ports...
    default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key

01 - Hello World task(permission problem)

when I do this task:
$ fly -t tutorial execute -c task_show_uname.yml

What can I do next?

10 "the host is taken"

Seems like anyone using PWS is going to run into the same issue I did, that simple-go-web-app.cfapps.io is already taken. Not sure what the right solution is, I know manifest files used to support {random-word} or something in them for this purpose. I ended up creating a gist to host a manifest file with a different app name and adding it as another resource: https://gist.github.com/benmoss/ca6e66a63bbbdc5e8748

better use long form of the command line parameter

take this for example
set-pipeline --config pipeline.yml --pipeline my-pipeline
instead of
set-pipeline -c pipeline.yml -p my-pipeline

or you have to check the source to find out the meaning
(https://github.com/concourse/fly/blob/master/commands/set_pipeline.go)

PS. the document site (https://concourse.ci/fly-cli.html ) use long form.

09 - use of aggregate

Is there a reason you're using aggregate in pipeline-ls-abc-xyz.yml but not the other two pipelines? From all I can understand about aggregate you'd want to to use it for fetching all the resources. It also incidentally makes all the resources show up as dependencies in the UI, whereas when you just list them directly under the plan: it seems to only visualize the first one.

I'd be happy to update it, just not sure if you had some intention with it or were just messing around. I really appreciate the repo, even these minimally documented ones are easier to grok than the docs 😸

02_task_inputs fails (ubuntu/docker installation)

Hello,

Thanks for this tutorial. I am facing this issue.

Joe@SEVEN64 /c/Concourse/concourse-tutorial/02_task_inputs (master)
$ fly -t tutorial e -c inputs_required.yml -i some-important-input=.
executing build 12
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0curl: (7) Failed to connect to 192
.168.0.9 port 80: No route to host
gzip: invalid magic
tar: Child returned status 1
tar: Error is not recoverable: exiting now
exit status 2
�[31mfailed�[0m
upload request failed: Put http://192.168.0.9/api/v1/pipes/0711287c-1dd7-4db7-58a7-8c23fb7b6762: dial tcp 192.16
8.0.9:80: i/o timeout

I have no clue what's going on here. Can you help please?

Add tutorial section on credential management

Human intervention

Is there any way to include human intervention in alle pipeline, e.g. for approvals before deploying etc?

docker image is not pulled in helm chart

i installed concourse/stable in kubernetes cluster using helm charts.

when i try hello-world
i get this error

annoy
resource script '/opt/resource/check []' failed: exit status 1
stderr:
failed to ping registry: 2 error(s) occurred:

• ping https: Get https://registry-1.docker.io/v2/: dial tcp 184.168.221.96:443: i/o timeout
• ping http: Get http://registry-1.docker.io/v2/: dial tcp 184.168.221.96:80: i/o timeout

Ensure README links to all lessons

12 - can't parameterize booleans

After I resolved the issue with martini and got the tests to run, I was getting error reading request from stdin: json: cannot unmarshal string into Go value of type bool from the deploy task.

I figured out that it was coming from skip-cert-check in the CF configuration being turned into a string by fly. It turns out you can't parameterize anything in your YML file that's not just a string.

<alexsuraci> yea we force it to be a string to make marshalling weird values like private keys safer

Not sure if you want to remove this, hard code it, or what.

Do you have a tutorial job to create a service on cf

Currently, the cf resource is only used to put the app (deploy)

How do I create a service and bind it to an app and then tear it down after testing ?

accessing output jar file

Hi,
I'm configuring a simple concourse pipeline to build a spring-boot app with maven and then deploy it to pivotal cloudfoundry. The problem is, that when deploying to CF I am unable to access the jar builded during the execution of maven package. Here is the pipeline and tasks files.
Pipeline:
**jobs:

• name: package-deploy
  serial: true
  plan:
  • get: source-code
    trigger: true
  • task: package
    privileged: true
    file: source-code/ci/tasks/package.yml

  • put: resource-deploy-web-app
    params:
    manifest: source-code/manifest.yml
    path: project-package

    resources:
• name: source-code
  type: git
  source:
  uri: MY_GIT_RESOURCE
  branch: master
• name: resource-deploy-web-app
  type: cf
  source:
  api: https://api.run.pivotal.io
  username: MY_USER_NAME
  password: MY_PASS
  organization: MY_ORG
  space: MY_SPACE
  skip_cert_check: true**

and the task yaml file:
**platform: linux

image_resource:
type: docker-image
source:
repository: nitram509/debian-openjdk8-maven-cmake-gcc
tag: latest

inputs:

• name: source-code

outputs:

• name: project-package

run:
path: mvn
args: ["-f", "source-code/pom.xml", "-DskipTests=true","-DoutputDirectory=project-package","package"]**

Here is the error message that pops when trying to PUT the jar file to the CF resource :
"error invalid path: found 0 files instead of 1 at path: /tmp/build/put/project-package/

I think there is some configuration that need to be done to be able to access the output of the package task during the PUT step but I couldn't find my way around it.
Thank you.

08_pipeline_jobs

For some reason the "job-run-task" is not triggered automated. I am a bit confused. Is this behaviour expected? I was expecting after running "job-fetch-resource" the second task will start but nothing is happen...

Vagrant Box Add fails on windows

I've gotten the same problem with Vagrant v. 1.9 and 2.0.1 - running the command

vagrant box add concourse/lite --box-version $(cat VERSION)

gives me this message & error:

Found VERSION file, using concourse/lite version 3.2.1
==> box: Box file was not detected as metadata. Adding it directly...
You specified a box version constraint with a direct box file
path. Box version constraints only work with boxes from Vagrant
Cloud or a custom box host. Please remove the version constraint
and try again.

so I can't get the vagrant box even running.

See if we can replace spiff example with spruce or bosh v2 style

Broken run.sh outside of 01-03?

update Vagrantfile to use latest-known working box

this is a commented out line [1] that should probably be uncommented and modified to be a specific known-to-work box.

i found certain tutorial examples don't work with 0.54.0 because fly c -c pipeline.yml no longer works without naming the pipeline. there are other changes too.

[1] https://github.com/jbayer/concourse-tutorial/blob/master/Vagrantfile#L16

Requesting example for Section 12 showing https protocol for clone/push

The current example shows usage of the git protocol for both operations

Memory leak

Hi!

I started implementing Concourse but I've hit a big problem - memory is leaked and it kills the machine that it's running on. It seems that memory is leaked by the Docker containers that are re-built every time a build plan is executed or something.

I'm using OSX.

Anybody else facing this problem?

no workers satisfying: resource type 'archive'

Hi,

I have installed concourse and fly using the following tutorial on mac:
https://gist.github.com/kevin-smets/f20afd45a24ab3f88d01b2049ce7744f
and had to change the yml file like the following:
platform: darwin run: path: echo args: [hello world]

which worked, but now I am getting following error with 02_task_inputs

$ fly --target tutorial execute --config inputs_required.yml --input some-important-input=.
executing build 17
no workers satisfying: resource type 'archive'

available workers:

• platform 'darwin'
  errored

concourse/concourse-ci no longer exists

A few steps of this tutorial depend on concourse/concourse-ci - that was never meant to be a publicly used image, as it's only meant for our CI pipeline. It no longer exists - this tutorial should probably refer to a specific image built for the tutorial instead.

05 - The gif is out of date

There is one step missing from the three-step description below the gif animation, not big deal, but I guess it is not too hard to change that. :)

12 - Publishing outputs: Host key verification failed.

Walking through the tutorial step-by-step. Made a gist, added

 - name: resource-gist
   type: git
   source:
     uri: [email protected]:71c94aad09a4adc12ebec5ac2794bab4.git
     branch: master

into 12_publishing_outputs/pipeline.yml. A build attempt fails in resource-gist with

Cloning into '/tmp/build/get'...
Host key verification failed.
fatal: Could not read from remote repository.

Note this is pull, we're not at the push stage yet.

The host key is certainly not in ~/.ssh/known_hosts inside the task container. How / where can I run ssh-keyscan or push `known_hosts' there?

examples for output passing between plans

i desperately need to pass output from one plan to another, wonder if that's duable, or any way of passing parameters between plans, im stuck

Storage Driver AUFS vs Overlay2

Thanks for the tutorial!
Unfortunately, I am running into a technical issue on the first exercise. I am trying to run through your tutorial using docker to run concourse and not virtualbox/vagrant. The problem I am encountering is that (after getting concourse running using the concourse/concourse docker-compose.yml) when I try to run the very first task from the tutorial (task_hello_world.yml), it fails because it is trying to use the aufs storage driver as opposed to the default "Docker for Mac" storage driver, overlay2.
Is there a way to fix this without forcing my Docker daemon to use aufs as the storage driver? All other images I've ran use overlay2, so I'd like to stick with it, if possible.

 fly -t tutorial execute -c task_hello_world.yml 
executing build 1
initializing
Pulling busybox@sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f...
sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f: Pulling from library/busybox
7520415ce762: Pulling fs layer
7520415ce762: Verifying Checksum
7520415ce762: Download complete
7520415ce762: Pull complete
Digest: sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f
Status: Downloaded newer image for busybox@sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f

Successfully pulled busybox@sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f.

docker: Error response from daemon: error creating aufs mount to /var/lib/docker/aufs/mnt/66174d331c8ee6e48312a93b9bfe8912031752c38b5e4d292383046c59c18493-init: invalid argument.
See 'docker run --help'.
resource script '/opt/resource/in [/tmp/build/get]' failed: exit status 125
errored

11_merge_config_with_spiff - is this still relevant post-spiff?