cloudfoundry / cf-deployment-concourse-tasks

cf-deployment-concourse-tasks

This repo contains concourse tasks for use with cf-deployment. If you are trying to deploy to IAAS, you may wish to check the Deployment Guide. Each task is in its own directory. A task will generally be composed of a task.yml to be referenced in pipelines, a supporting task file, and a Dockerfile. The Dockerfile is built and pushed to Dockerhub regularly in CI maintained by the CF Release Integration team here.

It should be clear how to use each task from the task.yml and the description below. If you find that it is not, please contact the Release Integration team in our Slack channel dedicated to supporting users of cf-deployment. Alternatively, you can open an issue.

Versioning of this repo

Development updates to the repo are made to the master branch, so untested or backwards incompatible changes may be present there. Once changes have been tested and all stories accepted, we add new version tags such as v1.6 to the approprate commit.

We use a bare-bones type of semantic versioning for this repo. Backwards incompatible changes warrant a major version bump (e.g. v1.6 to v2.0), while other changes will simply add a minor version bump (e.g. v2.0 to v2.1).

In Concourse, you can pretty easily lock to a major version, meaning that your pipeline will take minor (i.e. backwards compatible) changes only. Here's an example from our nats release pipeline:

- name: cf-deployment-concourse-tasks
  type: git
  source:
    branch: main
    uri: https://github.com/cloudfoundry/cf-deployment-concourse-tasks.git
    tag_filter: v3.*

When you're ready to take the backwards incompatible changes, you can take any necessary manual steps to upgrade, and then change the major version in your pipeline configuration.

Tasks

Tasks are listed here alphabetically, along with a brief description meant to be used alongside the task.yml within each task directory to understand the tasks' purpose, interface, and options. Each title is also a link to the appropriate task.yml.

bbl-destroy

This destroys the director and infrastructure created by bbl. Debug output is written to bbl_destroy.txt to help debug failures in this task.

bbl-up

This uses bbl to create your infrastructure and deploy a BOSH director. Debug output is written to bbl_plan.txt and bbl_up.txt to help debug failures in this task. This task requires a certificate and key (unless you are bbling up a bosh-lite environment) which can be generated using the commands specified here.

bosh-cleanup

This performs a BOSH cleanup which is necessary from time to time to avoid running out of space.

bosh-delete-deployment

This deletes a BOSH deployment. If you want to delete all of the available BOSH deployments you can set the DELETE_ALL_DEPLOYMENTS flag to true.

bosh-deploy

This performs a BOSH upload-stemcell and BOSH deployment. Optionally, operations files may be applied to the deployment manifest.

It's also configurable to regenerate deployment credentials on each deployment though this is not the default behavior. This is helpful for testing changes to variable generation, but is only expected to work with fresh deployments. This also automatically uploads the stemcells present in deployment. If you're deploying to bosh-lite environment you need to set the BOSH_LITE flag to true so the task uploads the correct stemcell(s).

bosh-deploy-with-created-release

This creates and applies an additional operations file to cf-deployment.yml, which causes BOSH to create, upload, and use a dev release from the provided release folder in place of the version specified in cf-deployment.yml. This is useful for testing an upstream component. Otherwise identical to the bosh-deploy task above.

bosh-deploy-with-updated-release-submodule

This takes as input a concourse resource for the submodule version bumped when creating a dev release from the provided release folder. Otherwise identical to the bosh-deploy-with-created-release task above.

bosh-upload-stemcells

This uploads stemcell(s) associated with the manifest and/or ops files provided. This task can be used to upload stemcells within jobs that do not contain a bosh-deploy* task (which handles uploading stemcells as well as executing the bosh deployment).

collect-ops-files

This collects two sets of operations files. The first set is the "base" set, to which the second ("new") set is added.

If there is a name conflict, the operations file from the second ("new") set wins.

run-cats

This runs CF Acceptance Tests against a CF environment specified by the CATs integration file. If desired, you can use the optional cf-cli input to provide your own CF CLI binary.

run-errand

This runs a bosh errand with the provided deployment and errand name.

set-feature-flags

This will toggle the specified feature-flags based on their boolean values.

update-integration-configs

This updates integration files to be consumed by CATs and RATs with credentials drawn from CredHub.

open-asgs-for-bosh-instance-group

This opens Application Security Groups for BOSH instance groups.

cf-deployment-concourse-tasks's Issues

Update or remove the `notify-bbl-updates` job

Currently the job tries to post a new chore to a tracker project whenever it recognizes that there's a new bbl version. This working group does not use a tracker, so arguably this job is currently useless and could be removed.

It might be worth going back through git history to try and determine why this job exists though. Maybe there's a valid reason that the working group should be notified about a new bbl version?

collect-ops-files task should create $BASE_OPS_FILES_DIR

In the collect-ops-files task, we fail if the specified dir does not exist:

cf-deployment-concourse-tasks/collect-ops-files/task

Lines 4 to 7 in 53c79a6

	if [ ! -d ${ROOT_DIR}/base-ops-files/${BASE_OPS_FILE_DIR} ]; then
	echo "[ERROR] base-ops-files/${BASE_OPS_FILE_DIR} must exist in order to collect operations files."
	exit 1
	fi

In the collect-ops-files task.yml, there is a comment that this path/directory will be created if it does not exist:

cf-deployment-concourse-tasks/collect-ops-files/task.yml

Lines 24 to 25 in 53c79a6

	# - If the specified path is not present,
	# it will be created.

I wanted to understand if the comment should be updated or the directory should be created if it does not exist.

Support --skip-drain in bosh-delete-deployment

It would be nice, in situations where you don't care about draining, to be able to supply a SKIP_DRAIN flag to the bosh-delete-deployment task.

bosh-deploy-with-created-release task fails if submodule history has changed

We have found that rebasing our submodule branch causes the bosh-deploy-with-created-release task to fail.

It appears that the pull triggers a merge conflict because of the rebase which causes the failure:

+ git pull
remote: Counting objects: 189, done.        
remote: Compressing objects: 100% (45/45), done.        
remote: Total 189 (delta 155), reused 178 (delta 144), pack-reused 0        
Receiving objects: 100% (189/189), 20.33 KiB | 0 bytes/s, done.
Resolving deltas: 100% (155/155), completed with 14 local objects.
From https://github.com/cloudfoundry-incubator/cloud_controller_ng_sapi
 + 38b8577...17b9ca2 service-instance-sharing -> origin/service-instance-sharing  (forced update)
 + 3e66cdc...bb1b6b3 pr-service-instance-share-summary -> origin/pr-service-instance-share-summary  (forced update)
 + 14ad5a2...fa669e0 pr-service-instance-sharing-bind -> origin/pr-service-instance-sharing-bind  (forced update)
 * [new branch]      tmp-rebase -> origin/tmp-rebase

*** Please tell me who you are.

Run

  git config --global user.email "[email protected]"
  git config --global user.name "Your Name"

to set your account's default identity.
Omit --global to set the identity only in this repository.

Can we do a fetch and reset instead?

Thanks,
Alex & @Samze

Bump to go1.22

go1.22 was just released and we should upgrade to it at some point in our images / modules.

Ability to use credhub to store generated vars

The currently implementation passes a var-store argument to bosh which effectively disables the use of credhub. I'd like to have a way of disabling this.

bbl 8.4.40 is unable to fetch terraform openstack provider

bbl 8.4.40 (from cf-deployment-concourse-tasks commit: 62ad0d4) is unable to fetch the terraform openstack provider:

Initializing provider plugins...

- Checking for available provider plugins on https://releases.hashicorp.com...

Error installing provider "openstack": openpgp: signature made by unknown entity.

Terraform analyses the configuration and state and automatically downloads
plugins for the providers used. However, when attempting to download this
plugin an unexpected error occured.

This may be caused if for some reason Terraform is unable to reach the
plugin repository. The repository may be unreachable if access is blocked
by a firewall.

If automatic installation is not possible or desirable in your environment,
you may alternatively manually install plugins by downloading a suitable
distribution package and placing the plugin's executable file in the
following directory:
    terraform.d/plugins/linux_amd64

Terraform manager init: Executor init: Run terraform init --upgrade: exit status 1

This might be related to cloudfoundry/bosh-bootloader#520.

bumping release is necessary for deployment

Hi!

We have uaa and uaa-release projects. Uaa is a submodule of uaa-release.
When we have a new version of UAA (on develop branch) we have to bump a submodule (uaa-release) in order to deploy newest code, and this is not currently supported by 'bosh-deploy-with-created-release' task. We had to have another task to manually copy uaa files into uaa-release, and output that for 'bosh-deploy-with-created-release' task to use.

Any plans on implementing the task to also bump submodule when creating the release?

As a user of the bbl-up task I would like to provide GCP credentials as a task param instead of as a path to a file

If the task only accepts a file, that leads me to check it into GitHub or put it in an S3 bucket. I would prefer to provide the credentials in the pipeline directly (like the AWS credentials) so I can store them in a more secure way with my other pipeline secrets.

run-errand task should support the `when-changed` flag

Hi Relint friends,

We are using the run-errand task in our pipeline and would like to use the --when-changed flag for some errands. Please could we add a param to the task? If you would like, we can PR in the change as well 😺

Chunyi and @jamesjoshuahill

Bump to ruby ~3.3

The current dockerfile ships with ruby ~2.7, which is EOL. We should upgrade to the current stable version of ruby, ~3.3.

Feature Request: support ops-files from multiple repositories in bosh-deploy

Not sure whether concourse will fully be able to support this, but it'd be nice to allow teams to have their own ops files. Currently, in order to write our own, we had to "vendor" the ops-files in cf-deployment.

release notes incorrect for v7.15

release notes have old version and new version of bbl & CF CLID the wrong way around. Please correct as follows

BBL Old version should be 6.10.18, new version 6.10.46
CCF CLI old version should be 6.40.1, new version 6.41.0

bash trap for "update-integration-configs" task not respecting current directory

We had the following error in our CI:

extracting cf admin password from mulan/deployment-vars.yml...
/tmp/build/ddaabd0b/integration-configs /tmp/build/ddaabd0b
updating CATs integration config file: mulan/cats_integration_config.json...
parse error: Expected another key-value pair at line 21, column 1
+ pushd integration-configs
cf-deployment-concourse-tasks/update-integration-configs/task: line 19: pushd: integration-configs: No such file or directory

The root cause was a trailing comma in our mulan/cats_integration_config.json, but the trap function commit_integration_configs in the update-integration-configs tried to navigate the filesystem using relative paths. I think the trap method specifically needs to be using absolute as it won't know what pwd might be.

The task run-cats failed with "Not logged in."

Environment

We are using Concourse v3.12 and latest cf-deployment-concourse-tasks(this repo) to setup our pipeline. The Concourse cluster is on Azure. One task in our pipeline is run-cats, its setting is:

- name: run-cats
  serial_groups: [main-group]
  serial: true
  plan:
  - aggregate:
    - {get: updated-integration-configs, trigger: true, resource: store, passed: [sleep]}
    - {get: cf-deployment-concourse-tasks}
    - {get: cf-acceptance-tests}
    - {get: cf-pipeline}
  - task: run-cats
    file: cf-deployment-concourse-tasks/run-cats/task.yml
    input_mapping:
      integration-config: updated-integration-configs
      cf-acceptance-tests: cf-acceptance-tests
    params:
      CONFIG_FILE_PATH: {{CONFIG_FILE_PATH}}
      NODES: 2
      SKIP_REGEXP: "transparently proxies both reserved characters and unsafe characters"

Issue

The run-cats task fails with many Not logged in. Use 'cf login' to log in. . Complete running log can be found here. In the running log, I found one item:

[2018-08-20 05:20:28.51 (UTC)]> cf auth admin [REDACTED] 
API endpoint: https://api.greentongyao.com
Authenticating...
Post https://login.greentongyao.com/oauth/token: dial tcp: lookup login.greentongyao.com on 168.63.129.16:53: read udp 10.254.0.122:46020->168.63.129.16:53: i/o timeout
FAILED

After this item, many Not logged in. Use 'cf login' to log in. are followed, which cause a single test case fails. "168.63.129.16" is Azure's DNS server, I use fly -t target intercept -j my-pipeline/run-cats to get into the container, running dig login.greentongyao.com manually, and I got:

; <<>> DiG 9.9.5-3ubuntu0.17-Ubuntu <<>> login.greentongyao.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40874
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1280
;; QUESTION SECTION:
;login.greentongyao.com.                IN      A

;; ANSWER SECTION:
login.greentongyao.com. 5       IN      A       104.210.59.119

;; Query time: 1 msec
;; SERVER: 168.63.129.16#53(168.63.129.16)
;; WHEN: Mon Aug 20 06:48:07 UTC 2018
;; MSG SIZE  rcvd: 67

Seems the DNS server can resolve the domain correctly.
We have seen this issue for several times, and cannot find the root cause of it. Could you please have a look?

Request: BBL_STATE_DIR calculates full(er) path

BBL is considering adding a BBL_STATE_DIR flag to mirror the functionality --state-dir flag. We actually tried shipping it, and then noticed in pipelines that this broke pipelines because of the process you follow, simplified like so:

cd $BBL_STATE_DIR
bbl up

Sure: we could pick a different name, or you could pick a different name, but I think in the long run it's a little more intuitive for people looking at the concourse task if all BBL_ are environment variables that BBL understands. For instance, I can check out my bbl-states repo, cd to that repo, set the vars I use in my concourse YML, and just bbl up. So my team has proposed this change as an example of a way that you can retain the name BBL_STATE_DIR and have a process that would be compatible with BBL before the change and after the change. Then long after the change you can elect at some time in the future to just cd bbl-states and allow BBL_STATE_DIR to be implicitly passed to bbl.

  local root_dir
  root_dir="${1}"
  export BBL_STATE_DIR=${root_dir}/bbl-state/${BBL_STATE_DIR}

...

+  mkdir -p "${BBL_STATE_DIR}"
+  pushd "${BBL_STATE_DIR}"
-  mkdir -p "bbl-state/${BBL_STATE_DIR}"
-  pushd "bbl-state/${BBL_STATE_DIR}"
  bbl up

Inconsistent concourse param for "VARS_STORE_{PATH/FILE}" between "set-feature-flags" and "update-integration-configs" tasks

As title tries to say, we are using VARS_STORE_PATH in one task, and VARS_STORE_FILE in another.

null bug in delete-deployment task

When attempting to run the delete-deployment task with DELETE_ALL_DEPLOYMENTS set to true, and if your environment doesn't have any deployments, you run into a null error:

++ jq -r '.Tables[].Rows[].name'
jq: error (at <stdin>:7): Cannot iterate over null (null)

Which appears to be coming from this line of code.

I don't know if this ever worked, or if a recent change to the bosh CLI made it break recently.

Here is a fuller log of our task output:

+ '[' true = true ']'
+ bosh_delete_all_deployments
+ local deployments
++ bosh deployments --json
++ jq -r '.Tables[].Rows[].name'
jq: error (at <stdin>:7): Cannot iterate over null (null)
+ deployments=

bosh-upload-stemcell-from-cf-deployment task does not upload windows stemcell

We have an environment that requires a windows diego cell, however the bosh-upload-stemcell-from-cf-deployment/task does not upload the windows stemcell.

Is there a way to get around this that doesn't involve uploading it manually each time there is an updated stemcell?

CF-CLI input in CATS task is not used correctly

The CATS tasks allows to provide a cf-cli input. However in the task itself it expects that CF CLI is provided as binary which the CF CLI Github release does not provide.
Additionally it's unclear if the PATH is set correctly so that CATS would use the provided CF-CLI.

Feature request: Support bbl's global debug flag

We found ourselves in a position where bbl create-lbs was failing with no helpful output. We believed this was the terraform apply failing, and since that output is scrubbed for security reasons, we didn't know why it was failing. The problem was only visible when we manually ran the command from our own machine and turned out to be a simple resource conflict.

Using `commit_bbl_state_dir` encourages committing secrets

There are various credentials in bbl-state.json, .tfstate and deployment vars files. Having commit_bbl_state_dir in bbl-up and bbl-destroy encourages folks to use the git resource for these files. git was not designed to be a secret store. Being one push away from leaking director credentials is... risky.

Support for ops-files in bbl-up

Hey,

It would be lovely to support bosh ops-files in the bbl-up task. We specifically are interested in this for the credhub opsfile.

One side-effect of this is that vars-store becomes optional for the other tasks. As far as I know, concourse does not allow optional inputs, so this might be in support of the discussion happening in #13.

No good suggestions on how to take in the list of ops-files either, as that answer overlaps a bit with #2

bbl-up task doesn't work with specified Docker image

We tried to use the bbl-up task with the Docker image it specifies, and got a huge error message when bbl drove bosh-init to do something involving Ruby.

Adding:

RUN apt-get install -y build-essential zlibc zlib1g-dev \
        ruby ruby-dev openssl libxslt-dev libxml2-dev \
        libssl-dev libreadline6 libreadline6-dev libyaml-dev \
        libsqlite3-dev sqlite3

to the Dockerfile and using that custom image fixed things. That apt-get command matches the bosh-init installation docs.

Since the pipeline jobs in runtime-ci are not triggered by anything, including daily or weekly resources, it appears this error may never have been caught.

/cc @fushewokunze-pivotal @dsabeti @anEXPer

Allow submodule resource to be input of bosh-deploy-with-created-release task

At the moment the bosh-deploy-with-created-release task takes a release, a submodule path and a branch name.

The flaw of this strategy is that the task takes the latest HEAD of the branch and does not allow the author to control the exact commit that the release is built with.

We want to commit to a submodule, run tests, build a release from the submodule and deploy it.

Example problem scenario:

1. Pair A commit & push to the branch of the submodule.
2. The pipeline runs unit tests against the submodule, they pass.
3. The pipeline triggers the bosh-deploy-with-created-release task.
4. Pair B also commit & push to the branch of the submodule.
5. The pipeline job fetches the branch from git, building the release with Pair Bs commit, not Pair As.

In an ideal world, create-release could take the submodule as a input resource, and then instead of checking out from a branch, it creates the release from the input. This would allow us to use concourse passing of resources to control this flow.

Thanks,
Sam & @ablease

build-docker-image job runs twice every time

Every time the build-docker-image job is triggered by a dependency update, the pipeline will push an updated dockerfile to the repo which will actually re-trigger the pipeline again.

run-errand task should allow for keep-alive flag option

It would be great to have a param KEEP_ALIVE that takes a boolean and adds the --keep-alive flag to the bosh run-errand command.

Cf deployment error

Using BOSH-Lite on UBUNtu 16.04 LTS the below commands throws error

bosh -e vbox -d cf deploy ~/workspace/cf-deployment/cf-deployment.yml \

-o ~/workspace/cf-deployment/operations/bosh-lite.yml
--vars-store ~/deployments/vbox/deployment-vars.yml
-v system_domain=bosh-lite.com
-o ~/workspace/cf-deployment/operations/experimental/enable-instance-identity-credentials.yml
-o ~/workspace/cf-deployment/operations/experimental/trust-route-ca.yml
-o ~/workspace/cf-deployment/operations/use-compiled-releases.yml
Using environment '192.168.50.6' as client 'admin'

Using deployment 'cf'

Task 13
Task 17
Task 14
Task 16
Task 15
Task 14 | 17:25:43 | Downloading remote release: Downloading remote release
Task 13 | 17:25:43 | Downloading remote release: Downloading remote release
Task 15 | 17:25:43 | Downloading remote release: Downloading remote release
Task 13 | 17:26:23 | Downloading remote release: Downloading remote release (00:00:40)
L Error: Downloading remote release failed. Check task debug log for details.
Task 13 | 17:26:23 | Error: Downloading remote release failed. Check task debug log for details.

Task 13 Started Fri Mar 30 17:25:43 UTC 2018
Task 13 Finished Fri Mar 30 17:26:23 UTC 2018
Task 13 Duration 00:00:40
Task 13 error

Task 17 | 17:26:23 | Downloading remote release: Downloading remote release
Task 15 | 17:26:23 | Downloading remote release: Downloading remote release (00:00:40)
L Error: Downloading remote release failed. Check task debug log for details.
Task 15 | 17:26:23 | Error: Downloading remote release failed. Check task debug log for details.

Task 15 Started Fri Mar 30 17:25:43 UTC 2018
Task 15 Finished Fri Mar 30 17:26:23 UTC 2018
Task 15 Duration 00:00:40
Task 15 error

Task 16 | 17:26:23 | Downloading remote release: Downloading remote release
Task 14 | 17:26:23 | Downloading remote release: Downloading remote release (00:00:40)
L Error: Downloading remote release failed. Check task debug log for details.
Task 14 | 17:26:23 | Error: Downloading remote release failed. Check task debug log for details.

Task 14 Started Fri Mar 30 17:25:43 UTC 2018
Task 14 Finished Fri Mar 30 17:26:23 UTC 2018
Task 14 Duration 00:00:40
Task 14 error

Task 17 | 17:27:03 | Downloading remote release: Downloading remote release (00:00:40)
L Error: Downloading remote release failed. Check task debug log for details.
Task 17 | 17:27:03 | Error: Downloading remote release failed. Check task debug log for details.

Task 17 Started Fri Mar 30 17:26:23 UTC 2018
Task 17 Finished Fri Mar 30 17:27:03 UTC 2018
Task 17 Duration 00:00:40
Task 17 error

Task 16 | 17:27:03 | Downloading remote release: Downloading remote release (00:00:40)
L Error: Downloading remote release failed. Check task debug log for details.
Task 16 | 17:27:03 | Error: Downloading remote release failed. Check task debug log for details.

Task 16 Started Fri Mar 30 17:26:23 UTC 2018
Task 16 Finished Fri Mar 30 17:27:03 UTC 2018
Task 16 Duration 00:00:40
Task 16 error

Creating and uploading releases:

• Uploading release 'capi/1.52.0':
  Uploading remote release 'https://storage.googleapis.com/cf-deployment-compiled-releases/capi-1.52.0-ubuntu-trusty-3541.10-20180327-184105-035528106.tgz':
  Expected task '13' to succeed but state is 'error'
• Uploading release 'cf-mysql/36.11.0':
  Uploading remote release 'https://storage.googleapis.com/cf-deployment-compiled-releases/cf-mysql-36.11.0-ubuntu-trusty-3541.10-20180327-190814-579497032.tgz':
  Expected task '15' to succeed but state is 'error'
• Uploading release 'cf-networking/1.12.0':
  Uploading remote release 'https://storage.googleapis.com/cf-deployment-compiled-releases/cf-networking-1.12.0-ubuntu-trusty-3541.10-20180327-183501-747828017.tgz':
  Expected task '14' to succeed but state is 'error'
• Uploading release 'binary-buildpack/1.0.18':
  Uploading remote release 'https://storage.googleapis.com/cf-deployment-compiled-releases/binary-buildpack-1.0.18-ubuntu-trusty-3541.10-20180327-183158-018354568.tgz':
  Expected task '17' to succeed but state is 'error'
• Uploading release 'cf-smoke-tests/40.0.1':
  Uploading remote release 'https://storage.googleapis.com/cf-deployment-compiled-releases/cf-smoke-tests-40.0.1-ubuntu-trusty-3541.10-20180327-183241-534645275.tgz':
  Expected task '16' to succeed but state is 'error'

Exit code 1

bosh-deploy task should take in a deployment name as an optional param

Currently the bosh-deploy task gets the deployment_name by interpolating it from the manifest.

So for deployments such as kubo-deployment, we are unable to deploy multiple CFCR clusters using the same director.

It would be super helpful if we could override the manifest's deployment_name with a DEPLOYMENT_NAME param, just like the run-errand task.

Why do we install and then remove software-properties-common in cf-deployment-concourse-tasks dockerfile?

cf-deployment-concourse-tasks/dockerfiles/cf-deployment-concourse-tasks/Dockerfile

Line 40 in ab469cf

apt-get remove -y --purge software-properties-common

Incorrect IaaS used when "required" param is missing for `bosh-upload-stemcell-from-cf-deployment`

When the INFRASTRUCTURE param is omitted, this param defaults to google which seems like an unsafe default. This should raise an error instead if it is actually required.

I spent quite a bit of time debugging as this only presented itself in the subsequent job that failed.

https://github.com/cloudfoundry/cf-deployment-concourse-tasks/blob/64f432d/bosh-upload-stemcell-from-cf-deployment/task.yml#L24

Bump to go1.21

The ARD WG discussed in our meeting today and think we should be able to cut a major release of cf-d-c-t with go1.21. We should announce it in all the usual places, and cut a release beforehand to give the community something to lock to with the latest resources on it.

Require Linux-ARM64 release for cloudfoundry/cf-deployment-concourse-tasks

Hi Team, 

I am building cloudfoundry/cf-deployment-concourse-tasks image for both AMD64 and ARM64 platforms and it’s failing on ARM64 due to below dependencies unavailable on arm64.

List of dependencies which are required on arm64:

1. cloudfoundry/cli  
2. cloudfoundry/bosh-cli
3. cloudfoundry/log-cache-cli
4. cloudfoundry/bosh-bootloader

Do you have any plans for releasing it for Linux/arm64? 

It will be really helpful if the above binaries and images are released for arm64 platform.

There is no specific release process is mentioned in the above projects to add arm64 builds and raise a PR. If interested, please share the release process and I will add and raise a PR.

Discussion: Merge `upload-stemcells` and `deploy` tasks

We were recently trying to use cf-deployment-concourse-tasks to upload stemcells in our pipeline, and we came across a few pain points:

• We needed to provide a list of ops-files to bosh-upload-stemcells in order to get our experimental and windows stemcells uploaded.
• To do this, we tried to use a YAML anchor to provide the exact same list of ops-files to both the deploy and upload-stemcells tasks.
• However, the upload-stemcells task rejects ops-files that don't modify the stemcells section of the manifest, so our YAML anchor didn't help. And we had to hand-curate the list of ops-files for the upload-stemcells task.

Originally, we were planning to give feedback about the errors raised in the upload-stemcells task and request that the task be modified so that you can just provide the same list of ops-files to both the deploy and upload-stemcells task.

But, as we thought about it more, we started to wonder if it made more sense for the upload-stemcells task to be subsumed into the deploy task.

The two tasks have very similar interfaces. They have same inputs, except that the deploy task has one additional input called vars-files. All of the parameters for the upload-stemcells task are also included in the deploy, except for INFRASTRUCTURE. The reason the inputs and parameters are so similar is because the two tasks are kind of similar work: generating a manifest (both tasks run bosh interpolate), and then performing a bosh command (deploy or upload-stemcells) using the completely generated manifest. This would solve our original problem (managing to distinct lists of ops-files between the two tasks) by consolidating the logic into one task.

Would it make sense to combine these two tasks? Or, at least, to add the upload-stemcells logic to the deploy task?

cc @flawedmatrix @jvshahid

Please configure GITBOT

Pivotal uses GITBOT to synchronize Github issues and pull requests with Pivotal Tracker.
Please add your new repo to the GITBOT config-production.yml in the Gitbot configuration repo.
If you don't have access you can send an ask ticket to the CF admins. We prefer teams to submit their changes via a pull request.

Steps:

• Fork this repo: cfgitbot-config
• Add your project to config-production.yml file
• Submit a PR

If there are any questions, please reach out to [email protected].

Discussion: Thoughts on converting some of these task files to resources

Hi all,

Has been any discussion around converting some of these tasks to a first-class Concourse resource? While the tasks work, the interface seems not as convenient or flexible as it could be due to the limitations on Concourse tasks. For example, this issue is caused by tasks having to explicitly declare a set number of inputs while a resource put can have an arbitrary number of inputs. We also have to duplicate params for each task (e.g. VARS_STORE_FILE) where a resource could have a single source.vars_store field.

Specifically I would really like to see a bbl-resource and some of the deploy tasks can already be replaced with the bosh-deployment-resource.

And thanks for sharing these tasks! CAPI team uses them heavily in CI.

cf tail command not found in docker image.

Our CATs tests started failing recently with the error:

'tail' is not a registered command. See 'cf help -a'

After some tracing, what I believe has happened is that CATs set the use-log-cache option to default to true whereas it used to be false by default.

cloudfoundry/cf-acceptance-tests@63c12af

This may be a reasonable change. But for users who are using the docker image defined in cf-deployment-concourse-tasks, it seems to be missing the appropriate command. Is it possible to get https://github.com/cloudfoundry/log-cache-cli installed in the cf-deployment-concrouse-tasks docker image?

As a work around for anybody else running into this, you can turn off the use-log-cache in your cats config and the old behaviour will be restored. at least up until cf 4.4, I haven't checked past that.

`update-integration-configs` should fail fast when `admin_password` is not found

update-integration-configs should fail fast when admin_password is not found. It's required for CATs, and I suspect WATS and RATS.

We've had a few experiences in the UAA pipelines where update-integration-configs does not find an admin_password for CATs and then updates cats_integration_config.json with admin_password: ''. update-integration-configs shows as "passed" in Concourse status, which indicates no problems. That being said, we do see some output from the Credhub CLI in update-integration-configs that may indicate a problem.

+ check_fast_fails
+ '[' '!' -f integration-configs/concourse/uaa-acceptance-gcp/state/cats_integration_config.json -a '!' -f integration-configs/rats_integration_config.json -a '!' -f integration-configs/wats_integration_config.json ']'
+ setup_bosh_env_vars
+ set +x
/tmp/build/7f3ba2bf/bbl-state/concourse/uaa-acceptance-gcp/state /tmp/build/7f3ba2bf
/tmp/build/7f3ba2bf
+ set +x
Usage:
  credhub [OPTIONS] get [get-OPTIONS]

Get a credential value by name or ID

Application Options:
      --http-timeout=    Http timeout for http-client. Needs to have unit
                         passed in (i.e. 30s, 1m) [$CREDHUB_HTTP_TIMEOUT]
      --version          Version of CLI and targeted CredHub API
      --token            Return your current CredHub authentication token

Help Options:
  -h, --help             Show this help message

[get command options]
      -n, --name=        Name of the credential to retrieve
          --id=          ID of the credential to retrieve
          --versions=    Number of versions of the credential to retrieve
      -j, --output-json  Return response in JSON format
      -q, --quiet        Return value of credential without metadata
      -k, --key=         Return only the specified field of the requested
                         credential
updating CATs integration config file: concourse/uaa-acceptance-gcp/state/cats_integration_config.json...

CATs then fails with error Invalid configuration: 'admin_password' must be provided.

Note that we're still able to retrieve this password via Credhub CLI on our local machine, and it appears from the timestamp that it hasn't updated in a while (version_created_at: "2019-08-16T22:14:20Z"). I'm not sure why update-integration-configs does not retrieve this password.

Bump to latest patch of golang in dockerfile

Don't know why dependabot hasn't done this...

director created by the tasks is open to the world

we are running bbl-up task to create our ENVs, we discovered that the director is accessible from everywhere. We would like to restrict this to a whitelisted set of IPs, is that a parameter that can be added to this task?

bbl up should update dns records

On GCP when bbl up creates a new cloud DNS managed zone X (after a bbl destroy), it doesn't update the X record-set in managed zone Y that uses it.

See bbl issue that suggests fixing this through a concourse task.

Setting the committer name and email

We want to use the tasks in this repo, but we'd also like to set the commits with our own teams email address and bot username. Are there any plans to parameterize git config such as:

cf-deployment-concourse-tasks/shared-functions

Line 26 in 7c86b2a

git config user.email "[email protected]"

cc @zankich

FEATURE: bosh-cleanup task optionally uses --all

The bosh-cleanup task currently uses --all flag. We would like it if the --all flag was optional.

Our use case is a couple jobs in our pipeline that deploy two different deployments (it tests upgrading from one to the other). It would be nice if we could run bosh cleanup without the --all option so that the releases for the not-currently-deployed deployment would not get deleted.

Thanks,
Angela & @mcwumbly

[Feature request]Ability to specify flakeAttempts in run-cats

Sometimes the environment might be fragile, it would be useful to be able to manually set flakeAttempts in run-cats to make ginkgo try more times.

bosh-deploy-with-created-release task should default to `name` if `final_name` is unavailable

It doesn't look like releases can be named with final_name or name (but not both) in bosh release. See these two commits on the bosh-cli:

• cloudfoundry/bosh-cli@6668d19
• cloudfoundry/bosh-cli@112f7a2

It would be great if this task could support either convention in bosh releases:

Relevant code

cf-deployment-concourse-tasks/bosh-deploy-with-created-release/task

Line 38 in b9c1ad4

bosh_interpolate "${root_dir}" "$(grep final_name release/config/final.yml | awk '{print $2}')"

Delete Deployment / Destroy Director flag for no fail

Would y'all be open to a PR that allowed the user to specify a "no-fail" type of scenario? We sometimes have situations where deletes have already been triggered and they end up getting triggered again (thus resulting in a pipeline failure).

I'd love to make it so we do a quick check of if the bbl-state exists and if it does not (and no fail is set) we skip the destroy / delete but return success.

Does this sound ok?

bbl-up task should rename "IS_BOSH_LITE" flag

The bosh team is using this task in one of our pipelines and found that we could prevent load balancer creation by setting IS_BOSH_LITE to true. This is very useful for us, so we'd like to keep using it. However, the IS_BOSH_LITE name suggests more than just skipping load-balancer creation, so we are a bit worried about relying on this functionality. Can you please provide a dedicated way to skip load-balancer creation instead? Thank you!

SYSTEM_DOMAIN sometimes incorrectly derived from "toolsmiths-env" input

The current tasks assume that envs associated with the "toolsmiths-env" input are always under the domain cf-app.com.

This is no longer always the case. For example, the "toolsmiths-env" we pass to the deploy task is api.xxx.yyy.shepherd.lease where the correct system domain should be xxx.yyy.shepherd.lease.

Pushing creds to credhub rather than git

It is my understanding that we should try to have our concourse credentials in credhub and not Git/github. This task.yml and its task commiting of yours suggest we should be putting the resource after this task.

In our pipeline, we regenerate the 3 files before each rats, cats etc. run by pulling the creds from credhub, this prevents us from pushing creds to git.

Removing the "put" related comment from the task.yml and removing all git commit related things in the script should prevent users from pushing sensitive information to git.