This repo holds the Concourse Pipelines, Jobs, and Tasks to setup a Concourse environment with:
- Terraform scripts to provision the environment
- Strict security groups: e.g. the DB port is only accessible from the ATC VM
- SSH traffic disabled by default: The SSH port is opened automatically by Concourse tasks to perform deployments and closed after.
- VMs configured with no public IPs: Only natbox and jumpbox have external IPs.
We'll start by deploying a secondary "Upgrader" Concourse VM.
This Concourse will be used to setup the main Concourse environment on GCP as well as perform upgrades later on.
These steps assume you'll deploy the Upgrader to a local vSphere environment.
Alternatively you can vagrant up
a Concourse instance on your workstation.
- Create a DNS record for the Upgrader VM pointing to a valid vSphere IP.
- Register Upgrader Concourse as an OAuth application with GitHub: https://github.com/settings/applications/new
- Callback URL:
https://YOUR_UPGRADER_URL/auth/github/callback
- Copy the contents of
./upgrader/upgrader.vars.tmpl
to a LastPass note or some other safe location, filling in the appropriate values. - Deploy the Upgrader VM:
cd ./upgrader
bosh create-env ./upgrader.yml -l <( lpass show --notes "bosh-concourse-upgrader-create-env" )
git add ./upgrader-state.json
git commit && git push
The upgrader vm must be configured with the pipelines that can deploy the main Concourse deployment.
- Read
./scripts/provision-gcloud-for-concourse.sh
to make sure you're not blindly running an untrusted bash script on your system - Set up the required variables and run the provision scripts:
TERRAFORM_SERVICE_ACCOUNT_ID=concourse-deployments \
DIRECTOR_SERVICE_ACCOUNT_ID=concourse-director \
PROJECT_ID=my-gcp-project-id \
CONCOURSE_BUCKET_NAME=concourse-deployments \
./scripts/provision-gcloud-for-concourse.sh
- for debugging purposes you can also set
TRACE=true
to show all commands being run.
- Generate a set of Google Cloud Storage Interoperability Keys as described here.
- Create a GitHub access token to avoid rate limiting as described here.
- Register main Concourse as an OAuth application with GitHub: https://github.com/settings/applications/new
- Callback URL:
https://YOUR_CONCOURSE_URL/auth/github/callback
- Generate the Director CA Cert by running
./scripts/generate-director-ca.sh
. - Generate the jumpbox ssh keys by running
./scripts/generate-jumpbox-ssh-key.sh
. - Add the jumpbox key as a project-wide SSH key with the username
vcap
as described here. - Copy the contents of
./ci/pipeline.vars.tmpl
to a LastPass note or some other safe location, filling in the appropriate values. - Log in using the fly cli to the newly deployed upgrader Concourse vm
- Set the Concourse pipeline on the upgrader vm.
fly -t upgrader sp -p concourse -c ~/workspace/bosh-concourse-deployments/ci/pipeline.yml -l <(lpass show note YOUR_LASTPASS_NOTE)
- Configure external worker pipeline: The CPI Core team needs a few external workers and deploys them with this pipeline. If you'd like to deploy external workers yourself you can use this pipeline as an example.
fly -t upgrader sp -p concourse-workers -c ~/workspace/bosh-concourse-deployments/ci/pipeline-cpi-workers.yml -l <(lpass show note YOUR_LASTPASS_NOTE)
- Seed empty statefiles:
gsutil cp -n <( echo '{}' ) gs://${CONCOURSE_BUCKET_NAME}/asia/natbox-state.json
gsutil cp -n <( echo '{}' ) gs://${CONCOURSE_BUCKET_NAME}/asia/jumpbox-state.json
gsutil cp -n <( echo '{}' ) gs://${CONCOURSE_BUCKET_NAME}/worker/vsphere-v6.5-worker-state.json
gsutil cp -n <( echo '{}' ) gs://${CONCOURSE_BUCKET_NAME}/worker/vcloud-v5.5-worker-state.json
gsutil cp -n <( echo '{}' ) gs://${CONCOURSE_BUCKET_NAME}/worker/google-asia-worker-state.json
- Manually trigger
concourse/prepare-concourse-env
job. - Manually trigger
concourse/update-director
job. - Manually trigger
concourse/update-cloud-config
job. - Manually trigger
concourse/update-concourse
job.
If you have deployed optional external workers you must follow a slightly modified order:
- Manually trigger
concourse/prepare-concourse-env
job. - Manually trigger
concourse/update-director
job. - Manually trigger
concourse/update-cloud-config
job. - Manually trigger
concourse-workers/prepare-asia-env
job.
- the
concourse/update-concourse
job will place a file inconcourse-update-trigger
resource. This file is used to automatically trigger the external worker jobs across pipelines.
- Manually trigger
concourse/update-concourse
job. This should trigger the external worker jobs (i.e. you don't need to manually trigger the worker jobs).
Thanks to the distributed model of the CF Foundation many teams from many companies can share this CI environment to run builds against their CPIs. Currently we have created a credentials to allow the Openstack CPI team to deploy and use an external worker in one of their Openstack environments.
In this example, we are adding a new team 'DigitalOcean CPI'
The DigitalOcean CPI team has provided following:
- a worker public key
- a GitHub organization
- a GitHub team within that organization that will be able to authenticate against the Concourse environment.
- a Concourse team name, no space, no special characters, all lowercase, (e.g. "digitalocean")
The BOSH CPI team does the following:
- Shares the TSA host public key (search for
concourse_tsa_public_key
in LastPass) with the DigitalOcean CPI team (e.g. "ssh-rsa AAAAB3NTSAHostPublicKey...") - Add the worker public key entry to the list under
concourse_teams
on the secure note saved on LassPass. Example:
[{"name": "digitalocean", "github_team": "DigitalOcean/BOSH CPI", "worker_public_key": "ssh-rsa AAAAB3DigitalOceanWorker..."}]
- Trigger the update-concourse job, making sure there are no running jobs first.
Let the DigitalOcean CPI team know when the deploy has finished so that they can rock.
The BOSH CPI team has provided following:
- TSA host public key
- TSA URL (e.g. https://bosh-cpi.ci.cf-app.com)
Do the following:
- Generate a key for your worker. The following command will create a keypair; don't use passphrase:
ssh-keygen -N '' -b 4096 -f /tmp/openstack-cpi-worker -C team_name
- Transmit the public portion to the BOSH CPI team (e.g. "ssh-rsa AAAAB3DigitalOceanWorker...").
- Let the BOSH CPI team know your GitHub organization (e.g. "DigitalOcean") and team handle (e.g. "DigitalOcean CPI").
- Pick a display name for your team and let the BOSH CPI team know. (e.g. "digitalocean")
- Create the manifest for your worker and make sure to set the following properties:
team: ((worker_team_name))
tsa:
host: ((concourse_tsa_hostname))
host_public_key: ((concourse_tsa_public_key))
private_key: ((worker_private_key))
- worker_team_name, e.g. "digitalocean". This is the team name provided to BOSH CPI
- concourse_tsa_hostname, e.g. https://bosh-cpi.ci.cf-app.com, provided by BOSH CPI
- host_public_key: e.g. "ssh-rsa AAAAB3NTSAHostPublicKey...", provided by BOSH CPI
- worker_private_key: the private key generated for the worker
You can find a sample of a worker manifest here.
After deploying the worker, authenticate with Concourse and confirm worker has registered:
- Browse to the Concourse URL and download the
fly
client - Log into Concourse:
fly -t cpi login -c https://bosh-cpi.ci.cf-app.com -n digitalocean
- Confirm worker has registered:
fly -t cpi workers
Refer to the Troubleshooting document under docs/.