cloudfoundry / capi-ci

capi-ci

Hello, this is the CAPI team's CI repo. It houses Concourse configuration settings for our CI environments.

Check it out! https://concourse.app-runtime-interfaces.ci.cloudfoundry.org/teams/capi-team/pipelines/capi

📌 This repository has gone through a complete overhaul. The old artifacts can still be found on the legacy branch.

Environments

See pipeline.yml for more details.

   ________________________________________________________________________
 / \                                                                       \
|   |  Elsa: biggest and most "real" environment                           |
 \_ |          · Long-lived                                                |
    |          · HA / Multi-AZ                                             |
    |          · Windows cell                                              |
    |          · Encrypted database                                        |
    |          · Clustered database                                        |
    |          · Runtime CredHub (assisted mode)                           |
    |          · Database: MySQL                                           |
    |          · Platform: GCP                                             |
    |          · Blobstore: GCP blobstore                                  |
    |                                                                      |
    |  Kiki: used for testing that db migrations are backwards compatible  |
    |          · Short-lived                                               |
    |          · Database: PostgreSQL                                      |
    |          · Platform: GCP                                             |
    |          · Blobstore: WebDAV                                         |
    |                                                                      |
    |  Asha: used for testing CATS and CAPI-BARA tests on MySQL            |
    |          · Short-lived                                               |
    |          · Database: MySQL                                           |
    |          · Platform: GCP                                             |
    |          · Blobstore: WebDAV                                         |
    |                                                                      |
    |  Olaf: used for running CATS and CAPI-BARA tests on AWS with MySQL   |
    |          · Short-lived                                               |
    |          · Database: MySQL                                           |
    |          · Platform: AWS                                             |
    |          · Blobstore: S3                                             |
    |                                                                      |
    |  Scar: used for testing CATS and CAPI-BARA tests on PostgreSQL       |
    |          · Short-lived                                               |
    |          · Database: PostgreSQL                                      |
    |          · Platform: GCP                                             |
    |          · Blobstore: WebDAV                                         |
    |   ___________________________________________________________________|___
    |  /                                                                      /
    \_/______________________________________________________________________/

What's Up with Kiki

Kiki starts with an older version of cf-deployment. It then runs the new migrations, but keeps the old Cloud Controller code. This catches any backwards-incompatible migrations. This is important because Cloud Controller instances do rolling upgrades. For example: if you write a migration that drops a table, old CC instances that depend on that table existing will crash during the rolling deploy.

Pipelines

capi

This pipeline is responsible for testing, building, and releasing capi-release.

capi-release

This is where the majority of testing for capi-release components live.

• Runs unit tests for Cloud Controller and bridge components
• Builds capi-release release candidates and deploys to Elsa, Kiki, Asha, Olaf, and Scar
• Runs appropriate integration tests for each environment
• Bumps the ci-passed branch of capi-release
• Updates release candidate in v3 docs every time ci-passed branch is updated.

bump-dependencies

Automatically bumps golang version for capi-release components every time a new golang-release is available. Also bumps Redis.

ship-it

Jobs responsible for cutting a capi-release.

• Bump API versions
• Update API docs
• Release capi-release

bbl-up

Updates the bosh deployments for all the pipeline environments (using bbl up).

bbl-destroy

Theoretically useful for destroying broken bosh deployments for all the pipeline environments. Often doesn't work because the directors are in such bad state. There are also jobs to manually release pool resources for the following environments: Elsa, Asha, and Scar.

bosh-lites

Pipeline responsible for managing the development bosh-lite pool.

• Create new bosh-lites if there is room in the pool
• Delete released bosh-lites

Using Pooled Environments

There are a number of helpful scripts in capi-workspace for using the bosh-lite pool. Most notably, claim_bosh_lite, unclaim_bosh_lite, and print_env_info. See the commands list for a full list of useful commands for interacting with the pool.

capi-ci's Issues

Kiki is not actually testing that migrations are backward compatible

In this commit, the kiki apply_latest_migrations.sh script was switched from using proxychains to apply migrations to using bosh ssh.

This unfortunately defeats the purpose of the kiki environment. From capi-ci README.md:

Kiki starts with an older version of cf-deployment. It then runs the new migrations, but keeps the old cloud controller code. This catches any backwards-incompatible migrations. This is important because cloud controller instances do rolling upgrades. For example: if you write a migration that drops a table, old CC instances that depend on that table existing will crash during the rolling deploy.

The script previously used proxychains (or tsocks in the original commit) as a way of running migrations from the concourse VM's local version of capi-release (which is newer than what's deployed) against the foundation's database, so that newer migrations are run WHILE the foundation is still using code from an older version.

Using bosh ssh just runs the migrations that are already deployed to the VM (i.e. no new migrations are run). So since this change kiki, has not been actually testing that new migrations are backward compatible.

What to do

Make the apply_latest_migrations script actually apply latest migrations.

I think we have a few options:

1. We can revert back to using proxychains. This may be hard because we apparently can't edit /etc/hosts with this change to using registry-image resource instead of docker-image (and is how we stumbled upon this issue)

2. Maybe we can look into using bosh scp to copy the new capi-release (perhaps in /tmp) to the vm and then using bosh ssh.

3. Some other way?

DRATS seems to be using an older image which contains an older version of the CLI

The DRATS jobs for Gabriel and Xena are currently failing due to a docker image which uses CLI v6. We should be using a more updated version (7 at the time of this writing).

We should evaluate the best version to use at this time, as well as the best way to establish the best version in a rolling manner.

Consider replacing outdated docker image from `relintdockerhubpushbot` with new image from `cloudfoundry`

Highly recommend that y'all consider switching from the unmaintained relintdockerhubpushbot/cf-deployment-concourse-tasks image to the maintained cloudfoundry/cf-deployment-concourse-tasks image. The image location was moved around two years ago, and the old image hasn't been updated since then.

I see that the old image is used in many tasks within this repo, for example:

capi-ci/ci/bosh/upload-release.yml

Line 6 in d96b8d8

repository: relintdockerhubpushbot/cf-deployment-concourse-tasks

remove `api-docs-production-deploy` step from `update-and-push-docs-v2`

Currently the update-and-push-docs-v2 pipeline job is paused. We are now hosting the v2 docs on a cloud foundry deployment that lives behind the VMware VPN. Setting up a concourse worker that lives behind the VPN and can automatically push the v2 docs felt more complicated than was necessary given how infrequently we are making v2 docs changes.