concourse / hush-house

Concourse k8s-based environment

HUSH-HOUSE

A hush house is an enclosed, noise-suppressed facility used for testing aircraft systems, including propulsion, mechanics, electronics, pneumatics, and others.

This repository contains the configuration of hush-house.pivotal.io, metrics-hush-house.concourse-ci.org, and any other Kubernetes (K8S) deployments using the hush-house Kubernetes cluster available in the shared Concourse Google Cloud account.

Table of contents

• Repository structure
• Dependencies
• Gathering acccess to the cluster
• IaaS
• Deployments
  • Creating a new deployment
  • Without any credentials setup
  • With credentials
  • Visualizing metrics from your deployment
• SSHing into the Kubernetes node VM
• k8s cheat-sheet
  • Contexts
  • Namespaces
  • Nodes
  • Kubernetes for credential management
• Bootstrapping the cluster
  • Getting the GCP credentials
  • Applying the Terraform
  • Creating the base Kubernetes objects

Repository structure

.
│
├── deployments 		# Where you can find all deployments that get
│   │				# continuously deployed by `hush-house` when changes
│   │				# to configurations in this repository are made.
│   │
│   │
│   ├── with-creds		# Deployments that require credentials that do not exist
│   │   │			# in this repository (as they're not public)
│   │   │
│   │   ├── Makefile		# Scripting related to the deployments
│   │   │
│   │   ├── hush-house  # The web nodes of the `hush-house` Concourse deployment
│   │   ├── worker		  # The deployment of the kgeneric pool of workers that connect to `hush-house`
│   │   └── metrics		  # The `metrics` deployment
│   │
│   │
│   │
│   └── without-creds		# Deployments that require NO credentials, i.e., that rely
│       │			# solely on public values from this repository.
│       │
│       ├── Makefile		# Scripting related to the deployments.
│       │
│       └── bananas		# Fully functioning example deployment.
│
│ 
├── helm			# Scripts to automate the provisioning of Helm and its
│				# server-side component (tiller)
│
│
├── Makefile			# General scripting for getting access to the cluster and
│				# setting up the pipelines.
│
│
└── terraform			# Terraform files for bringing up the K8S cluster and other
    │				# configuring other infrastructure components necessary
    │				# for the deployments (addresses + cloudsql).
    │
    ├── main.tf			# Entrypoint that makes use of modules to set up the IaaS
    │				# resources.
    │
    ├── address			# Module to create addresses ...
    ├── cluster			# Module for creating GKE clusters w/ node pools
    │   └── vpc			# Module for setting up the VPC of the cluster
    └── database		# Module for setting up CloudSQL

Dependencies

• LastPass CLI (lpass-cli)
  • brew install lastpass-cli
• Terraform CLI (terraform)
  • brew install terraform
• Helm (helm)
  • brew install kubernetes-cli
  • brew install helm && mv /usr/local/bin/helm{,3} (for ci)
  • brew install helm@2 (for the other deployments)
• Kapp (kapp) (for ci)
  • brew tap k14s/tap
  • brew install kapp
• Helm diff plugin (helm diff)
  • helm plugin install https://github.com/databus23/helm-diff --version master
• Google Cloud CLI
  • brew cask install google-cloud-sdk

ps.: if you're creating your own environment based on an existing k8s cluster, you'll probably only need helm.

Gathering acccess to the cluster

0. Install the dependencies

1. Configure access to the Google Cloud project

gcloud config set project cf-concourse-production
gcloud config set compute/zone us-central1-a
gcloud auth login

2. Retrieve the k8s credentials for the cluster

gcloud container clusters get-credentials hush-house

3. Initialize the Helm local configuration

Note.: this is only needed if you've never initialized helm locally.

helm init --client-only

4. Retrieve the Helm TLS certificates and the CA certificate

# fetch the creds to lpass
make helm-creds

# copy the credentials to $HELM_HOME
make helm-set-client-creds

5. Run Helm command against Hush-House requires --tls.For example,

helm install ... --tls

IaaS

As hush-house is a complete environment for deploying Concourse and any other Helm charts, it requires few infrastructure pieces to be in place.

All of that is provisioned using terraform, having its configuration under the ./terraform directory.

Make sure you DON'T change the IaaS parameters in the Google Cloud Console - modifications MUST be made through terraform.

Deployments

In the hush-house cluster, there are currently a few Helm charts deployments running.

As mentioned in the repository structure section, these all live under ./deployments.

Check the deployments README to know more about them.

Creating a new deployment

To create a new deployment of your own, a Chart under ./deployments/(with|without)-crekds needs to be created (given that every deployment corresponds to releasing a custom Chart).

There are two possible types of deployments we can create:

1. without any credentials setup, and
2. with credentials.

Without any credentials setup

tl;dr: copy the ./deployments/without-creds/bananas directory and change bananas to the name of the deployment you want.

0. Create a directory under ./deployments/without-creds, and get there:

mkdir ./deployments/without-creds/bananas
cd ./deployments/without-creds/bananas

1. Populate the repository with the required files for a Helm Chart, as well as metadata about itself (Chart.yaml):

# Create the required files
touch ./{Chart.yaml,requirements.yaml,values.yaml}


# Populate `Chart.yaml` file with some info about it
echo '---
name: bananas
version: 0.0.1
description: a test deployment!
maintainers:
- name: ciro
' > ./Chart.yaml

2. Add the concourse release candidate as a dependency

echo '---
dependencies:
- name: concourse
  version: 0.0.15
  repository: https://raw.githubusercontent.com/concourse/charts/gh-pages/
' > ./deployments/bananas/requirements.yaml

ps.: the version can be retrieved from concourse/charts.

pps.: the upstream version of the Chart could be used too! See helm/charts for instructions.

With that set, hush-house is ready to have the deployment going.

You can either trigger the deployment from your own machine if you have Helm already set up, or make a PR to hush-house so that the pipeline does it all for you.

Once the process is completed, you should be able to see your resources under the deployment namespace:

kubectl  get pods --namespace=bananas
NAME                                  READY   STATUS    RESTARTS   AGE
bananas-postgresql-7f779c5c96-c8f4v   1/1     Running   0          2m
bananas-web-78db545cc9-xrzd9          1/1     Running   1          2m
bananas-worker-78f6cddccb-brvm9       1/1     Running   0          2m
bananas-worker-78f6cddccb-qd6zn       1/1     Running   0          2m
bananas-worker-78f6cddccb-xv7p5       1/1     Running   0          2m

With credentials

A deployment that requires credentials that can't be publicly shared involve all of the steps above, including some few more. Once those steps were finish, proceed with the following :

1. Create the values.yaml file with public configurations

echo '---
concourse:
  worker:
    replicas: 3
  concourse:
    web:
      prometheus:
        enabled: true
' > ./deployments/bananas/values.yaml

2. Populate the .values.yaml file with credentials

echo '---
concourse:
  secrets:
    localUsers: test:something
' > ./deployments/bananas/.values.yaml

ps.: this can be left blank

3. Populate the hush-house-main namespace with your credentials

Having kubectl configured (see gathering access to the cluster) with access to hush-house-main, create the secret using the hush-house-creds-secrets-$DEPLOYMENT target from ./deployments/with-creds/Makefile:

# go back to `./deployments/with-creds`
cd ..
make hush-house-creds-secrets-bananas

Visualizing metrics from your deployment

When using the Concourse Helm chart, metrics get scrapped and graphed by default under https://metrics-hush-house.concourse-ci.org if Prometheus integration is enabled.

To do so, make sure you have concourse.web.prometheus.enabled set to true and the prometheus.io annotations added to concourse.web:

concourse:
  web:
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "9391"
  concourse:
    web:
      prometheus:
        enabled: true

With that set, head to the Concourse dashboard under the metrics address provided above and change the Namespace dropdown to the one corresponding to the name of your deployment.

SSHing into the Kubernetes node VM

As the worker nodes created by worker pools declared in the main Terraform file are just regular GCP instances, these can be accessed using the regular ways of accessing VMs through gcloud.

# The name of the instance can be retrieved from the
# command that lists nodes connected to the k8s cluster:
# - `kubectl get nodes`.
NODE_NAME="gke-hush-house-test-workers-1-46b1d860-65mf"


# Use `gcloud` to facilitate the process of getting the
# right credentials set up for SSHing into the machine.
#
# ps.: you must have `gcloud` credentials set up before
#      proceeding - check out the section `Gathering acccess to the cluster`
#      in this README file.
gcloud compute \
	ssh \
	--zone us-central1-a \
	$NODE_NAME

k8s cheat-sheet

Here's a quick cheat-sheet that might help you get started with kubectl if you've never used it before.

Contexts

These are the equivalent of Concourse targets, storing auth, API endpoint, and namespace information in each of them.

• Get all configured contexts:

kubectl config get-contexts

• Change something in a context (for instance, the namespace to a default one):

kubectl config set-context $context \
	--namespace=$new_default_namespace

• Set the context to use:

kubectl config use-context $context

Namespaces

A virtual segregation between resources in a single cluster.

It's common to have environments associated with namespaces such that their resources get isolated too. In this scenario, you can think of namespaces as BOSH deployments - they're all managed by the same director, but they get their instances and other resources isolated from each other.

The namespace to target is supplied via the --namespace flag, or having a default namespace set to the context (see #contexts).

Nodes

Similar to bosh vms, it's possible to gather the list of instances that compose our cluster.

• Retrieve the list of all registered k8s nodes:

kubectl get nodes

• Describe (get events and extra information of) a particular node:

kubectl describe node $node_name

Kubernetes for credential management

Just like you can tie Vault or CredHub to your Concourse instances in order to have secrets suport, you can also make use of Kubernetes secret for that, with some specialties:

• Can't make use of _ in the names (a limitation from k8s secrets)

For instance, the Secret something_a is invalid:

metadata.name:
  Invalid value: "something_a":
    a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.',
    and must start and end with an alphanumeric character (e.g. 'example.com', regex used
    for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')

• Names must not be longer than 63 characters

• for interpolating ((mything)):

kubectl create generic mything \
  --from-literal=value=$value \
  --namespace $prefix$team

• for interpolating nested structures ((mything.foo)):

kubectl create generic mything \
  --from-literal=foo=$foo \
  --namespace $prefix$team

Bootstrapping the cluster

Creating the hush-house cluster on GKE from the ground up requires:

1. having GCP credentials,
2. applying the Terraform definition under ./terraform, then
3. creating few objects in the Kubernetes cluster.

Below you find instructions to how to do those steps.

Getting the GCP credentials

Access to the GCP credentials for hush-house can be granted through a GCP JSON key stored in LastPass.

The Makefile at the root of this repository contains a target for retrieving that key and placing it at the right place:

make gcp-key

Applying the Terraform

With the credential obtained, we can follow up creating the underlying resources in the IaaS (GCP), using the defitnions under the ./terraform directory.

cd ./terraform
terraform apply

Creating the base Kubernetes objects

A fully working hush-house Kubernetes clusters requires few components: a Tiller deployment (the server-side compoennt of Helm), and a custom StorageClass (so we can create PersistentVolumeClaims based of SSD storage).

To configure Tiller, first get the Helm certificates and keys from LastPass and then run the script that bootstraps it.

Note.: the script is meant to be run with the current working directory pointing to cluster-bootstrap.

make helm-creds
cd ./cluster-bootstrap
./bootstrap-tiller.sh

To finish the bootstrapping, we now need to create the StorageClass. From the root of this repository, run the following:

cd ./cluster-bootstrap/storage
./ssd-storage-class.sh

hush-house's Issues

Move metrics Github to the Concourse org

[updated by @cirocosta ]

When logging into metrics-hush-house.concourse-ci.org making use of the GitHub login, the user is prompted to sign into an application that is currently owned by @cirocosta's GitHub account.

What we should have, instead, is an application under the Concourse org (we might have a bot account or something that manages metrics.concourse-ci.org which we could leverage).

metrics: panels regarding container & volume garbage collection

Hey,

I noticed that in our metrics dashboard we don't have any panels that give us stats regarding GC activity in terms or either volumes or containers, despite their usefulness.

For instance, having a problem in prod we could quickly spot that, for some reason, containers were being found for deletion but not really deleted:

Looking at the Prometheus endpoint of a specific instance indeed revealed that there are no gc metrics there, meaning that we'd probably need to PR a solution to concourse/concourse.

Task inheriting parent cluster GCP account info

The results of our task can be seen here: https://hush-house.pivotal.io/teams/PE/pipelines/kibosh/jobs/delete-gke-cluster-and-registry-images/builds/1. Our pipeline creates and deletes a GKE cluster using our service account key provided. On our delete step, we forgot to add the --project parameter. The result was trying to delete a GKE cluster in the cf-concourse-production project.

hh: configure ingress so we can serve multiple deployments through single address

As a way of dogfooding concourse/concourse#2408, we could have web's default 8080, metrics and all other stuff on it and for those few deployments that might require external workers, configure TCP for 2222 separately.

hh: per-team credential management

Hey,

With the intent of letting teams manage their credentials, whenever a new team is on board, we could create a namespace for that team, which ATC could then use for credential retrieval.

This would make hush-house more compelling than Wings as this would allow teams to not have their credentials in plain-text in their pipelines.

As I see, the whole flow would look like:

1. team signs up for participating in hush-house
2. a new team configuration is created under hush-house/teams
3. a namespace is created for that team to add secrets to
4. a service account that has permissions just to create/read/update/delete secrets in that namespace is created
5. access to that service account is granted somehow

Internally, this would have the following effect:

                  person from team_a
                        |
                        |
 k8s cluster -----------+----------------------------------
 |                      |
 |                      | (auth w/ serviceaccount that has access
 |                      |  to `team_a namespace`)
 |                      |
 |    team_a namespace -+----------------------------------
 |    |                 |
 |    |                 |
 |    |                 *------CREATE_SECRET (mysecret)
 |    |                              |
 |    |                              |
 |    |                            mysecret
 |    |                              ^
 |    |                              |
 |    |                              GET_SECRET
 |    |                              |
 |    *------------------------------+----------------------
 |                                   |
 |                                   |
 |    hush-house namespace ----------+----------------------
 |    |                              |  
 |    |                              |   (access to all team namespaces)
 |    |                              |
 |    |             ATC ---get_cred--*
 |    | 
 |    *-----------------------------------------------------

Acceptance Criteria

• have the current teams that we already have (see https://github.com/concourse/hush-house/tree/master/deployments/with-creds/hush-house/teams) with their own secrets namespaces, and
• have a way for the users from those teams to set secrets in those namespaces.
• ensure that Concourse is able to consume the secrets set in those namespaces.

Thanks!

hh: thoughts on consequences of updating team names

We just recently saw how frustrating it is for a team to have their team name changes and then forgetting to change the name used in team authorized keys.

It seems to me that there are two ways that we could solve this:

• leaving only ourselves as owners and centralizing the set-team like we used to do, or
• having a checklist for those who get to perform operations like renaming teams for other teams.

Thoughts?

Thanks!

metric: keep track of Prometheus Server statistics

Right now, there's no way of knowing how much is being consumed from the Prometheus server instance or the PD associated with it.

A follow up to this would be to have alerts for it too, so that we can proactively react to problems that might arise (like disks getting full), which could be based on the work done for the panel where we graph such utilization (but that's for another issue).

ps.: in terms of disk usage, we might be able to use the same logic we have for Postgres - using kube-state to retrieve that.

metrics: add `cloudsql` statistics to grafana

With our bump to Grafana 6, we're now able to make use of the Stackdriver integration and get metrics from there.

What's even better is that we can leverage the automatic provisioning of datasources: http://docs.grafana.org/features/datasources/stackdriver/#configure-the-datasource-with-provisioning.

metrics: set up alerting

Hey,

Despite the fact that we already have metrics being collected by Prometheus, and dashboards being displayed from Grafana, we're still in the need of having alerting set up in order to not need to constantly look at dashboards to know when things go wrong.

From my understanding, we have at least two choices here:

• go with grafana alerts (like we do for Wings), or
• go with alertmanager, which would be notified by Prometheus when a particular query goes beyond a certain level.

While the second case seems interesting to me from the standpoint that we can decouple the visualization from the alerting, the second case allows us to visualize how our thresholds look like, as well as consume multiple data sources (e.g., not only Prometheus but Stackdriver as well).

Acceptance criteria:

• Can we use Prometheus for all alerting needs?
• have the full alerting system working: having the observed metrics going up, we really get paged.

Thanks!

Research Linux OOM killer behavior for cgroups

Hey,

We've been seeing some of our workers going away after a certain memory profile gets reached due to the kernel's OOM killer getting started and destroying our container.

It seems like k8s does not act as an intermediary for the OOM killer when it comes to cgroups, making the whole process very ungraceful.

It'd be interesting to understand:

• how does k8s deal with limits when you reach memory limits (does it live to the kernel to terminate the parent process in the cgroup?), and
• what is the best metric to look for when it comes to understanding if an OOM eviction is going to take place: should we look 1 - available? Should we consider cache?

This is very impactful for workloads like strabo that might generate a huge in-memory cache for the files it access.

metrics: un-expose Grafana's HTTP port in favor of the HTTPS one

Hey,

Right now it's possible that someone goes directly to http://metrics-hush-house.concourse-ci.org and face a long interval between a failure being returned by the browser.

Having the port unexposed in the first place would allow us to make such requests fail in a quicker manner.

metrics: add panel descriptions

We could do the same as we did for https://metrics.concourse-ci.org in terms of annotating every panel with a useful description (see concourse/concourse#2711) as that has proven to be very useful for folks that recently got exposed to such metrics.

hh: pods getting evicted due to being low on ephemeral-storage

Hey,

Despite now specifying a limit in the size of emptyDir, GKE has been evicting our workers:

status:
  message: 'The node was low on resource: ephemeral-storage. Container hush-house-worker
    was using 73712Ki, which exceeds its request of 0. '
  phase: Failed
  reason: Evicted
  startTime: "2019-02-15T20:18:59Z"

It'd be good to see if there's anything configuration-wise that we did wrong 🤔

Configuration endpoint for Credhub secrets provider

For credentials management, I'd like a configuration endpoint for credhub.

Although there doesn't appear to be a helmchart to deploy credhub on Kubernetes, I'd like to configure our Concourse helm chart to leverage our existing credhub deployed via BOSH running within the same VPC.

Currently there are values.yml endpoints for vault, awsSsm, and awsSecretsManager; could one be opened up for Credhub as well?

Road to a supported installation

Hey,

Below is a list of items that we will need to complete/confirm before we can start moving/adding workloads onto hush house:

• Decide between having the workloads on top of GKE or PKS
  • at the moment, deploying hush-house in PKS wouldn't give the team much more data than we'd get from GKE, where we already have it running, allowing us to not have to learn any details of PKS and just move directly to what we already have.
• Monitoring
  • see the metrics deployment
  • #26
• SLOS
  • see the datadog dashboard
• Domain
  • @scottietremendous - I sent out an ask request for this.
  • #25
• Pager Duty - Alerts
  • #26
• Setup Monitors on New Screens 📺
• Set-up Slack Channel
• Flight Attendant
  • https://github.com/topherbullock/flight-attendant/issues/2
• move Dwayne's pipelines to hush-house. His teams:
  • https://wings.pivotal.io/?search=team:cf-notifications
    • https://hush-house.pivotal.io/?search=team:cf-notifications
  • https://wings.pivotal.io/?search=team:cf-tools
    • https://hush-house.pivotal.io/?search=team:cf-tools
• Mailing List
• Set-up Wiki

Thanks!

---

cc @scottietremendous

hh: enforce hard antiaffinity on web nodes

Hey,

Having jobs that stream in and out a lot, we might end up having problems with the web pods if those end up being scheduled to run in the same machine as the whole network bandwidth of the machine might be consumed by them, while in reality, those two could be serving the full bandwidth of two VMs.

For instance, consider the following case:

There we have two web nodes in the same VM consuming the whole 1Gib of network bandwidth that the instance has (2* 250Mbit TX + 2*250MBit RX).

If those workloads were split in two instances, we could go even higher.

Upgrade terraform to 0.12

We want all the new awesome features they released.

metrics: correlate Concourse worker metrics to k8s worker node

Hey,

Right now it's fairly easy to correlate the metrics that we get from cadvisor with a given node. This makes the process of jumping from one dashboard to the other fairly easy.

That's not true for metrics scraped by kubernetes-pods though - any node-specific information gets dropped during relabelling. As a result, we can't easily filter information by node.

Some possible actions:

• update the relabelling scheme for kubernetes-service-endpoints and go with it, including pod information there (already has node info, misses pod though); or
• update relabelling scheme for kubernetes-pods, including node information there (it already has pod info).

Thanks!

metrics: add panel for worker container file descriptors

Hey,

Despite the various panels that we have, one that we miss for the worker containers is "Number of open files" and "Number of processes".

Right now, we have a similar one for the web pods given that the golang Prometheus client library exposes that through process_*, but given that the worker does not have an exporter, we'd need to rely on cAdvisor for that.

Unfortunately, with the current version of k8s in GKE (v1.12.5), the latest cAdvisor is not used - what we have there is 0.30.1 see release notes, while the metric we need is only in 0.32.0.

metrics: get rid of alertmanager

Hey,

We currently have alertmanager deployed, but we're not making use of it.

After we tacked #26, we end up going with Grafana alerts, invalidating the use of Aertmanager for now.

Thanks!

metrics: alert based on SLIs

Hey,

At the moment, we have our SLIs (see Datadog SLI Dashboard) being displayed in a monitor in the office, but this does not necessarily mean that the operators (Test Pilot Pair) end up acting on problems that happen there, as the pair would need to keep looking at it.

By tying those SLIs with PagerDuty, we're then able to treat degradations in those numbers as triggers for action.

To better reflect the fact that an alert is generated, we should also update the way that the colors are set there (such that it's immediate for someone to see that there's an active incident).

In terms of "which thresholds to use", let's start with an SLO of daily 95 (which means 1h12m of downtime), and adjust that accordingly.

Acceptance criteria

• With a daily SLI being hurt, a page is generated to PagerDuty
• Have Datadog colors representing our objectives.

Thanks!

Update CloudSQL to PostgreSQL 11

Hey,

GCP just got support for PostgreSQL 11 out

https://cloud.google.com/blog/products/databases/enterprise-databases-managed-for-you

We’ve heard you want the latest version, so we are pleased to announce PostgreSQL version 11 support, which includes useful new features like partitioning improvements, stored procedures, and more parallelism.

so we should be able to upgrade hush-house's database to leverage that.

Because all of our IaaS resources are created using terraform, we'd need to update that in our module that brings the DB up:

hush-house/terraform/database/main.tf

Lines 5 to 8 in d2fefbe

	resource "google_sql_database_instance" "main" {
	name = "${var.name}-${random_id.instance-name.hex}"
	region = "${var.region}"
	database_version = "POSTGRES_9_6"

From https://cloud.google.com/sql/docs/postgres/create-instance#create-2ndgen-curl, it appears that the version should then be specified as POSTGRES_11_1.

Thanks!

hh: remove `configuring teams` section

Hey,

As pointed out by @pivotal-bin-ju , as teams are able to change their configuration, it doesn't make a lot of sense to keep those configs in this repo - https://github.com/concourse/hush-house/tree/master/deployments/with-creds/hush-house#configuring-teams - (as people will change them).

My bad!

Thanks!

Ability to leverage certs via cert-manager

(Background: using PKS 1.13.6 on vSphere leveraging NSX)

Hey there!

I'd like to leverage cert-manager to manage generating and rotating TLS certs via the letsencrypt issuer rather than manually generating them & having to pass it into to the deployment (via a secret .secrets.yml file or something). This reduces operational overhead and makes our Concourse helm deployment yaml's more declarative.

Having the certs associated to the Ingress resource is doable in this configurable annotation block, which can be configured with something like certmanager.k8s.io/cluster-issuer: letsencrypt-prod (in the default values.yml, an example of kubernetes.io/tls-acme: 'true' is used, which is what cert-manager used to use as it's "activation annotation", apparently.) This can automatically generate and apply a TLS cert for the hostname specified on the Ingress, but, that's only on the Ingress resource.

I'm under the impression we need ATC itself to leverage the cert, which requires setting concourse.web.tls.enabled-- which means the webTlsCert and webTlsKey must be provided as literal values.

With that said:

1. Does using cert-manager/let's-encrypt seem like a reasonable use-case?
2. If so, is there a way the cert values generated via cert-manager on the Ingress could be referenced by webTls?
3. More generally, is there a way for keys to be dynamically generated using some resource within k8s rather than generating them "externally" and providing them?

The goal I had in mind was to reduce "moving parts" involved in managing Concourse via helm, e.g. generating/acquiring certs, storing them in credhub or lastpass, monitoring their expiration, etc.

Thoughts?

thanks for your time 👍

hh: add instructions for adding team external workers

Hey,

After the "Easier way to add team external workers" PR got in, we should update the documentation around adding team external workers to hush-house.

secrets:
  teamAuthorizedKeys:
    - team: foo
      key: |-
        ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OMDG...

Thanks!

metrics: increase prometheus retention

Hey,

At the moment, the metrics Prometheus server is configured with a retention policy that allows us to store only 15d of data:

Which, having a default of 15d when 0 (see https://prometheus.io/docs/prometheus/latest/storage/), means that we end up being able to go back a max of 15d.

Even though we didn't have a big need of going way back, it might be useful for some more high-level data like what flight_recorder or an SLI runner could give us.

--

Regarding disk, at the moment we're at a constant 20% utilization:

My guess is that going up in terms of retention would proportionally make this grow.

hh: understand why system CPU usage is so high

Hey,

Looking at the CPU utilization of a node that runs the worker that got assigned the final step of the strabo pipeline, we can see that the CPU utilization gets to a quite interesting place:

From the perspective of the container, we can see similar results, indicating that at least a great part of that usage indeed comes from processes within that worker cgroup:

Doing a pidstat on the pid of concourse worker inside the container:

12:17:09      UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
12:17:10        0         8    0.00   14.85    0.00    0.00   14.85     6  concourse
12:17:11        0         8    0.00    1.00    0.00    0.00    1.00     6  concourse
12:17:12        0         8    0.00    0.00    0.00    0.00    0.00     6  concourse
12:17:13        0         8    0.00    0.00    0.00    0.00    0.00     6  concourse
12:17:14        0         8    0.00    1.00    0.00    0.00    1.00     6  concourse
12:17:15        0         8   26.00  100.00    0.00    0.00  100.00     6  concourse
12:17:16        0         8   26.00  100.00    0.00    0.00  100.00     6  concourse
12:17:17        0         8    0.00  100.00    0.00    0.00  100.00     6  concourse
12:17:18        0         8    0.00  100.00    0.00    0.00  100.00     6  concourse
12:17:19        0         8    0.00  100.00    0.00    0.00  100.00     6  concourse
12:17:20        0         8    0.00   67.00    0.00    0.00   67.00     6  concourse
12:17:21        0         8    0.00   74.00    0.00    0.00   74.00     6  concourse
12:17:22        0         8    0.00   96.00    0.00    0.00   96.00     6  concourse
12:17:23        0         8   10.00   67.00    0.00    0.00   77.00     6  concourse
12:17:24        0         8    0.00   57.00    0.00    0.00   57.00     6  concourse
12:17:25        0         8    0.00  100.00    0.00    0.00  100.00     6  concourse
12:17:26        0         8    0.00  100.00    0.00    0.00  100.00     6  concourse
12:17:27        0         8    0.00  100.00    0.00    0.00  100.00     6  concourse
12:17:28        0         8    0.00  100.00    0.00    0.00  100.00     6  concourse
12:17:29        0         8    0.00  100.00    0.00    0.00  100.00     6  concourse
12:17:30        0         8    1.00   91.00    0.00    0.00   92.00     6  concourse
12:17:31        0         8   32.00   44.00    0.00    0.00   76.00     6  concourse
12:17:32        0         8    0.00   89.00    0.00    0.00   89.00     6  concourse
12:17:33        0         8    0.00   87.00    0.00    0.00   87.00     6  concourse
12:17:34        0         8    0.00  100.00    0.00    0.00  100.00     6  concourse
12:17:35        0         8    0.00   76.00    0.00    0.00   76.00     6  concourse
12:17:36        0         8    0.00   70.00    0.00    0.00   70.00     6  concourse
12:17:37        0         8    0.00   64.00    0.00    0.00   64.00     6  concourse

To help troubleshoot that, it might be useful to generate flamegraphs from the perspective of the host to see from the traces where it's spending all of that time.

hh: document how to allow Concourse to gather secrets for teams other than `main` using k8s as cred mgmt

Add documentation around workloads possibly falling under the preemptible pool

As we don't make use of taints for the preemptible pool, it'd be good to advise people using this that they should explicitly tag their resources when consuming hush-house if they don't want their workloads to fall into such pool.

hh/metrics: update TLS certificates

With the wildcard certificate that we use very close to its expiration and having a new one already generated, let's just update them!

hh: explicitly target the preemptible node pool for workers

Hey,

RIght now the Concourse workers for deployments/hush-house end up falling under the preemptible node pool just because the other pool can't fit the requests.

It'd be better if instead we explicitly targetted it like we used to do before.

Road to supported Installation - round 2

Hey,

At the moment, hush-house has pretty much feature parity with
wings, which was accomplished through our first iteration (see
#27).

In this second iteration, we plan to refine it more, aiming at two goals:

• making hush-house compelling for internal teams, and
• making it more maintainable.

To achieve those, here are some of the steps to get us there:

• Add panel descriptions - #15
• Allow teams to leverage credential management
  • Verify if we can use k8s secrets as team credential management - #40
    • if so, go with it
    • otherwise, evaluate Vault
• Deploy hush-house from main Concourse pipeline - #28
• Alert based on SLIs - #39
• document k8s credential management - concourse/docs#96
• Remove hush-house.concourse-ci.org domain from concourse-ci-org zone
  • 7735302
• Improve documentation about the terraform definitions
  • 3d1da46
• fix terraform deprecation warnings - #44
• get rid of AlertManager - #43

---

Not included in this iteration, but soon to be:

• Allow the provisioning of team-specific workers
  • some teams might have the desired of having their own team-specific workers;
    what if we could just manage their workers for them?
• Continuously deploy metrics
• Continuously terraform apply
  • whenever the terraform scripts are updated.

Thanks!

PagerDuty Notifications incorrectly show affected service as concourse-wings

When a PagerDuty notification goes off it shows the affected service is concourse-wings when it should be hush-house. When I click the Grafana link or dig into the details of the incident I find the error is coming from hush-house.

The pager duty alert should initially say the affected service is hush-house

hh: move external url from `concourse-ci.org` to `pivotal.io` and update certificates

Hey,

With hush-house graduating to "being a thing" for Pivotal folks, we're going to update the external URL to hush-house.pivotal.io.

Once we get the domain mapped to this address, we can then:

• update the external url that we set for the Chart

  hush-house/deployments/with-creds/hush-house/values.yaml

  Line 71 in d2fefbe

  externalUrl: https://hush-house.concourse-ci.org

• update the certificates in our hidden .values.yaml file.
• update URL in Datadog SLI variable
• Update URL in README

Thanks!

hh: deploy from the main Concourse pipeline

Hey,

The way that hush-house gets updated at the moment requires someone manually setting a version in the values.yaml file and letting the deployment happen.

Just like we update wings (whose configuration sits in https://github.com/concourse/prod) from the main pipeline, hush-house could be updated in a similar manner too.

• see the job that deploys wings,
• see the ops files used, and
• see variables consumed.

Moving towards that, we'd need to:

• get our secrets out of lpass, putting them into credhub (such that we can use variable interpolation for the .values.yaml file that contains our secrets).

Regarding how often to deploy: same frequency as Wings.

Thanks!

hh: make use of taints

Hey,

Given that the worker-1 node pool is specially tailored for hush-house workers and we're not using taints there, it'd be nice if we had a higher priority for our workers there.

hh: Leverage 5.2 features

Hey,

With the new features that are coming to v5.2 (see https://raw.githubusercontent.com/concourse/release-notes/master/lit/v5.2.0.lit and helm/charts#13295), we're able to some cool things like:

• naming the hush-house cluster as hush-house
• enabling cacheing for our secrets lookups
• having an audit trail of user interaction through logs

Regarding the latter, we should be able to get a nice vis of those actions by crafting some metric queries against our logs in stackdriver 🤔

Thanks!

metrics: perform automatic redirect from HTTP to HTTPs

Hey,

RIght now, requests to http://metrics-hush-house.concourse-ci.org are not getting redirect to https://metrics-hush-house.concourse-ci.org. It'd be nice to have that redirection in place as making a request to the HTTP one makes it feel like the service is down.

Thanks!

metrics: alerts dashboard

Hey,

In order to better utilize the real state that we have for quickly assessing whether we're close to hitting problems, we could leverage the https://grafana.com/docs/features/panels/alertlist/ panel type, or perhaps, do something similar to what we do for our regular CI: https://metrics.concourse-ci.org/dashboard/db/alerts?refresh=1m&orgId=1.

Thanks!

hh: make use of local disks

Hey,

Right now we're using a patch that makes everything ephemeral by only using emptyDir, which is ok, but that's been painful for workloads that require a lot of disk IO as the disk throughput ends up being shared between the various pods in the machine that make use of ephemeral storage.

It's also known that the pd-ssd that we use throttles us at a not very high bandwidth:

With the use of local SSDs we can have a much better perf, but that'd require us making helm/charts#9668 more versatile in terms of having the option to use statefulsets (so we can leverage the local provisioner to give us access to the local SSDs via regular PVCs).

hh: add a title to our cluster

Hey,

With concourse/concourse#3736 getting in (which should come in the next version of Concourse - post 5.1), we could add a hush-house name to our cluster.

Wdyt?

Thanks!

Add a job to deploy to hush-house in the main pipeline

since we have wings deploy job in the concourse main pipeline. It would be nice if we have a similar job that deploys to hush-house whenever we want to deploy there before releasing.

Add pg_stat_statments to hush-house cloudsql terraform.

Background:

• In order to be able to get query statistics, we need to have the pg_stat_statments extension enabled on Google Cloud SQL.
• @zoetian and I were able to accomplish that by running the SQL statement: CREATE EXTENSION pg_stat_statements.
• The idea behind this issue is to be able to either automate that in our database terraform, or at least document it somewhere so we can run it when recreating the environment.

hh: Improve our streaming throughput

Hey,

I've been noticing that even though we recently got a better utilization of our worker nodes when it comes to system CPU usage (see "Why our system CPU usage is so high?"), our throughput when streaming between one worker to another is still pretty disapointing.

ps.: those bits going from workers to workers are compressed (gzip) - still, there's a lot of bandwidth left to be utilized

ps.: in the image above, the thin pink line at 400Mbps is a web instance piping data from one worker to another, thus, 200 + 200 Mbps

In order to test what would be the theoritical maximum amount of throughput that we could get on GCP, I started experimenting with stressing some of the edges in terms of streaming performance.

Here are some results.

Table of contents

• Machines used
• Streaming directly from one node to the other without any disk IO
• Streaming from one node to another, writing to disk when receiving
  • 1TB non-SSD persistent disk
  • 1TB SSD persistent disk
• Extra - Local SSDs
  • 1 Local SSD (375GB)
  • 3 Local SSDs (3 x 375GB) on Software RAID 0 EXT4
  • 3 Local SSDs (3 x 375GB) on Software RAID 0 XFS

Machines used

In the following tests, the same machine type was utilized (2 instances) with varying disk configurations:

• 16 vCPU Intel(R) Xeon(R) CPU @ 2.20GHz
• 32 GB Mem
• Ubuntu 16.04 / 4.15.0-1026-gcp

Streaming directly from one node to the other without any disk IO

In this scenario, all that we need to do is make the receive side discard all of the input that it receives.

We can do that by taking all of the contents that come from the listening socket and writing it all down to /dev/null.

What can be observed in this case is that GCP indeed delivers what it promises, giving the consistent ~10Gbps of network throughput.

This is specially true knowing that we're able to get a worker-to-worker bandwidth of consistent 10Gbs without much effort:

Receiver:

Transmitter:

With the results above, we can then establish that without any diks IO, we can get the promised 10Gbps of network throughput.

Streaming from one node to another, writing to disk when receiving

Now, changing the receiver side to write to a disk that gets attached to the node, we're able to see how the story changes.

1TB non-SSD persistent disk

GCP promise

Operation type	Read	Write
Sustained random IOPS limit	750	1,500
Sustained throughput limit (MB/s)	120	120

ps.: 120 MB/s ~ 960Mbps

Actual:

• 460 IOPS
• 1Gbps (~125 Mbps)

Looking at the results, seems like we're hitting the byte upper limit:

Now, we can clearly see:

• throttling hapenning at ~460 IOPS
• 1Gbps of network throughput limited by the 1Gbps disk throughput
• iowait times pretty high due to throttling

1TB SSD persistent disk

GCP Promise:

Operation type	Read	Write
Sustained random IOPS limit	25,000	1,500
Sustained throughput limit (MB/s)	480	480

ps.: 480 MB/s ~ 3.8Gbps

Actual:

• 1560 IOPS
• 3.3Gbps (~ 415MBps)

Now, instead of being throttled on the bytes upper limit, we're throttled at the IOPS limit, even though the bytes number is quite similar, anyway.

Extra - Local SSDs

I was curious about how much perf we could get using Local SSDs.

Without any particular tuning applied to the filesystems on them, I went forward three configs, all using the
same machine type.

1 Local SSD (375GB)

3 Local SSDs (3 x 375GB) on Software RAID 0 EXT4

3 Local SSDs (3 x 375GB) on Software RAID 0 XFS

Next steps

It's interesting to see how the software-based RAID didn't help much, and that the default configs for a 1TB
pd-ssd was the configuration that achieved the highest throughput.

Could that be a flaw in the way that I'm testing these setups? Very likely!

With some baseline of what's some of the best throughput we can get in terms of disk IO combined with network throughput, we can go ahead with some more testing in terms of either adding alternate compression algorithms, or perhaps even being more aggressive in terms of streaming by doing that with more concurrency (it's currently done in serial).

Thanks!

terraform: address deprecation warnins

Hey,

Using the latest version of the google Terraform provider, we've been getting few deprecation notices:

Warning: module.cluster.google_container_cluster.main: "zone": [DEPRECATED] Use location instead

Warning: module.cluster.google_container_node_pool.main[0]: "zone": [DEPRECATED] use location instead

Warning: module.cluster.google_container_node_pool.main[1]: "zone": [DEPRECATED] use location instead

Let's address that! 😁

Thanks!

ci: include chart validators as a step before deploying

Right now, it's possible that the deployments might fail due to minutiae in the chart configurations.

With something like helm lint we can do part of that static check, and with kubeval, verify if the generated k8s objects match the expectations of Kubernetes before pushing them through tiller.

Ref: https://github.com/helm/chart-testing

ci: make use of `--atomic` in deployments once Helm 2.13.x is released

With the use of the feature provided by helm/helm#5143, we can more securely make sure that the deployments get to a usable state even if something goes wrong by automatically applying a proper rollback if applicable.

See https://github.com/helm/helm/releases for available releases.

ci: make use of a Helm resource instead of tasks for deploying the deployments

Right now, deployments get performed by using the deploy task that manages the idempotency via helm-diff, but it'd be more interesting to use what's out there in terms of resource types so that we can both leverage the functionality and also check out what's been built by the community.

A candidate that I've been considering is Linkyard's Helm Resource, but we can check others out as well 😁

ci: automate check of chart's values documentation

Pretty much described under helm/charts#11300, the idea is to make sure that we don't end up with large discrepancies between README.md and values.yaml.

ci: make `reconfigure-pipeline` validate credential parameters before setting pipeline

Right now reconfigure-pipelines ends up reconfiguring the pipelines without validating the credentials beforehand.