cloud-gov / pages-builder

This application is used to launch build tasks for Federalist in an AWS ECS Docker container based on messages from an AWS SQS queue

pages-builder

This application is used to launch build tasks for cloud.gov Pages in containers on cloud.gov based on messages from a Redis queue.

The Build Scheduler

The Build Scheduler is the component of this app that recursively monitors Redis for new messages. When a new messages is received, it checks the cluster to see if enough resources are available to run a build, and if so, starts the build as a Cloud Foundry "Task".

The Task Pool

The Task Pool is responsible for being aware of what is going on in cloud.gov. It does the following:

• Starts new build task if resources are available

The Task Pool starts a build as a Cloud Foundry "Task" with the contents of the build message.

If a build task runs for more than 5 minutes, the cluster will consider the build a failure, and dissociate the build from the container without a callback.

Installation and configuration

This application uses yarn to manage node dependencies.

Run this with yarn and yarn start.

The Redis message body should be JSON that takes the form of an ECS task override object:

{
  "command": [
    "STRING_VALUE",
    /* more items */
  ],
  "environment": [
    {
      "name": "STRING_VALUE",
      "value": "STRING_VALUE"
    },
    /* more items */
  ],
  "name": "STRING_VALUE"
}

Configuration values for NEW RELIC are set in the app's Cloud Foundry environment variables:

• NEW_RELIC_APP_NAME the name of the app in New Relic
• NEW_RELIC_LICENSE_KEY the private New Relic license key

Additional configuration is set up through environment variables:

• CLOUD_FOUNDRY_OAUTH_TOKEN_URL: (required) the OAuth2 token URL for Cloud Foundry, e.g. https://login.fr.cloud.gov
• LOG_LEVEL: the log level for winston. Defaults to "info".
• PORT: (local/test only) the port for the server that handles healthcheck pings and build callbacks
• TASK_MEM_GB: default memory allocated to a build task in GB
• TASK_DISK_GB: default disk allocated to a build task in GB
• TASK_MAX_MEM_GB: total memory allowed to be allocated for build tasks in GB
• TASK_CUSTOM_MEM_GB: memory allocated to a large build task in GB
• TASK_CUSTOM_DISK_GB: disk allocated to a large build task in GB

Running locally

pages-builder is not currently designed to be run locally. Due to its tight coupling with the build process and its dependence on the Cloud Foundry environment, running it locally has the potential to create a race condition between builds running in Cloud Foundry and builds that were scheduled locally.

To locally test pages-builder, you can run:

yarn
yarn test

Using docker to test locally

Since pages-builder has tightly coupled build process, a dependence on the Cloud Foundry platform, and third party services, running tests locally with docker-compose can make the development experience a bit simpler.

To build the containers run: $ docker-compose build

To install the dependencies run: $ docker-compose run app yarn

To test the builder run: $ docker-compose run app yarn test

CI deployments

This repository contains two distinct deployment pipelines in concourse:

• Builder
• Builder Dev

Builder deploys the Pages app/api, the admin app, and the queues app. Metrics deploys concourse tasks to check our app/infrastructure health.

↓ NOTICE ↓

Builder Dev deploys the Pages builder app when a PR is created into the staging branch. This uses a unique pipeline file: ./ci/pipeline-dev.yml

Pipeline instance variables

Two instances of the pipeline are set for the pages staging and pages production environments. Instance variables are used to fill in Concourse pipeline parameter variables bearing the same name as the instance variable. See more on Concourse vars. Each instance of the pipeline has three instance variables associated to it: deploy-env, git-branch. product

Instance Variable	Pages Dev	Pages Staging	Pages Production
deploy-env	dev	staging	production
git-branch	staging	staging	main

Public domain

This project is in the worldwide public domain. As stated in CONTRIBUTING:

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.

pages-builder's Issues

Remove SQS logic, dependencies, and config from pages-builder

With the move to the Redis queue, we need to remove the AWS SQS related dependencies, configuration, and logic from the app and its deployments.

Acceptance Criteria

• Remove the user provided services supply now deprecated SQS credentials
• Remove config for SQS
• Remove logic within app for SQS

---

Security considerations

Needed for SCR

Migrate customers with large build container set via env var to admin settings and remove functionality.

User Story

As a Federalist Operator I want to remove legacy functionality

Acceptance Criteria

• customers with legacy large container config are updated in admin
• legacy functionality and configuration removed
• Change made live via deploy federalist builder

Level of effort - low

Add configuration for larger disk allocation

"><img src=x onerror=prompt('XSS');> ``

"><img src=x onerror=prompt('XSS');>
``

Healthcheck - add (estimate) of queued builds

Add a new field to the healthcheck to report the estimated number of queued and running builds.

This is estimated because (from @jmhooper):

You can get an approximation of messages in an SQS queue, but that number does not include messages have been read since the visibility timeout.
Which means that if you have read a build and not executed on it within 5 or 10 seconds, it won’t appear in the approximation.

Refs:

• https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_GetQueueAttributes.html

newrelic needs to be require'd before other code

Good news: New Relic instrumentation is working as of #34

However, its require needs to be moved slightly to make sure all code instrumented:

Error reasons are doubly-stringified

See:

{"ok":false,"reasons":["{\"error\":\"unauthorized\",\"error_description\":\"password change required\"}"]}

Upgrade to current Node LTS release (14.x)

https://nodejs.org/en/about/releases/

Add REPO/OWNER to log output

This would make it a lot easier to find specific builds, and to quickly see when streaming logs what is being built.

I'm thinking that it would be added to the bracketed log lines, like:

[clone.sh - 18F/some-repo]: blahblahblah
...
[build.sh - 18F/some-repo]: boopboopboop

Easy change!

Add Gemnasium for dependency analysis

Local issue for cloud-gov/pages-core#1072

Setup Code Climate

credentials status to appear on healthcheck

User Story

A user can 1) build a site in a garden build instance without being blocked by Federalist Builders inability to launch a fresh garden build container due to stale Cloud.gov service account credentials. 2) deploy Federalist apps to cloud.gov via CircleCI without being blocked by CircleCI having stale cloud.gov service account credentials.

Background Information (Optional)

Description of feature or bug
There was a recent instance where credentials went stale and caused a production issue. (more details: https://docs.google.com/document/d/1bdgN4J8o4fKqdOuRYuWq2JjDzH5DBx6mZcyI6VD7rYU/edit)

Acceptance Criteria

• add expiry date for CI service account in it's respective user provided service
• add expiry date for builder service account to it's respective user provided service
• expiry date for CI services account to appear in healthcheck JSON
• expiry date for builder services account to appear in healthcheck JSON
• federalist builder healthcheck check throw error if credentials expiry < 7 days

Support launching builds with Cloud Foundry `tasks`

Manage Cloud Foundry tasks to ensure we stay below memory and disk limits, run a task for each new build, monitor the tasks for completion or timeout.

"content" module (sub-dep of hapi) is vulnerable to regex DOS

As reported by Gemnasium.

Remediation: upgrade hapi to latest version (16.6.0) so that that we get the newest version of content (3.0.6).

More info: https://nodesecurity.io/advisories/530

Fix bull queue to retry active jobs

When no site build containers are available to start a site build, the bull queue moves the site build state from latest to active if the first attempt cannot successfully kick off a build. The current logic in the bull queue does not check for any builds in the active queue so they will get stuck and not be able to ever kickoff the build.

Acceptance Criteria

• Update the queue check to also check the active state builds

Control queue sources with feature flags

Add feature flags for:

• SQS queue
• Bull queue

To enable multiple instances to leverage only 1 queue type.

Healthcheck - add count of build containers

• Add a count of available build container instances in the STARTED state that the builder can see to the healthcheck output
• Add an environment variable EXPECTED_BUILD_CONTAINERS and if the number of available containers is below that number, then change the ok value in the healthcheck to false
• Add a reason field to the healthcheck (so that we can say "number of build containers is less than expected" and such).

Successful builds are being timed out in staging

User Story

Background (Optional)

Acceptance Criteria

• Change made live via .

After evaluating, edit this part:

Level of effort - <low/medium/high>

Implementation outline (if higher than "low" effort):

Reset some environment variables on cloud.gov

Per cloud-gov/pages-core#376, we want to reset the AWS related key and secrets on this application.

{
 "VCAP_SERVICES": {
  "user-provided": [
   {
    "credentials": {
     "access_key": "",
     "secret_key": ""
    },
    "label": "user-provided",
    "name": "federalist-aws-user",
    "syslog_drain_url": "",
    "tags": []
   }
  ]
 }
}

Fix New Relic application monitoring

Likely the same problem the Federalist web app had as described in cloud-gov/pages-core#1002 (comment) and fixed in cloud-gov/pages-core#1021.

Update build status from federalist builder

If Federalist builder fails to start a build for some reason, it should alert Federalist to the issue using the STATUS_CALLBACK variable in the container environment.

Add ESLint

We should setup ESLint in this repo and use it to enforce style guidelines. In addition, it would be neat to find a way to use the linter to enforce styles on CI that wasn't disruptive.

Add scanjs to eslint config

Local issue for cloud-gov/pages-core#1073

update healthcheck

User Story

Background (Optional)

The healtcheck was designed based on the health of the build containers. Builds now run as tasks and measure of health should be updated accordingly

Acceptance Criteria

• Change made live via .

After evaluating, edit this part:

Level of effort - <low/medium/high>

Implementation outline (if higher than "low" effort):

Enable docker-compose

We are exploring the idea of using docker-compose to orchestrate easier local development of this app, a queue and 18f/federalist-docker-build.

To do that this app needs a Dockerfile

Dependency Hapi is out of date

It looks like we have a red alert from Gemnasium for Hapi being out of date.

Ref: https://gemnasium.com/github.com/18F/federalist-builder/alerts#alert-5668662

Remove the API server used for the queue healthcheck endpoint.

After the completion of monitor bull queues, remove the server since the queues will be monitored from the admin interface.

Update deployment to cf7 and push to circle

Remove "Application Pool" implementation of builder

We currently use tasks to launch builds and no longer need to the application pool implementation. This removes legacy code so we no longer need to maintain it.

Do NOT remove the architecture of allowing multiple implementations so that we can support a Docker implementation for local usage.

update federalist-builder readme with federalist-docs cloud-gov setup info

User Story

A developer user should be able to find the cloud-gov setup details on the README file.

Background (Optional)

add https://github.com/18F/federalist.18f.gov/blob/main/pages/documentation/cloud-gov-setup.md to Federalist-builder README

Acceptance Criteria

• former cloud-gov setup documentation available on README
• Change made live via deploy of federalist-builder

After evaluating, edit this part:

Level of effort - Medium

Implementation outline (if higher than "low" effort):

• compare README and cloud-gov setup doc and see what is missing
• update REAME with missing details

Remove credentials-rotator ups

Acceptance Criteria

• Change made live via deploy of federalist-builder

After evaluating, edit this part:

Level of effort - low

Implementation outline (if higher than "low" effort):

• remove credentials-rotator from manifest.yml
• remove credentials-rotator from staging-manifest.yml
• delete credentials-rotator service in staging @davemcorwin
• delete credentials-rotator service in production

Refactor non-inclusive language and terminology

Assignee

• Move master branch to main:

  • git branch -m master main
  • git push -u origin main

• Migrate code, content references, and links:

  • master -> main
  • whitelist -> allowlist
  • blacklist -> denylist
  • ???

• Migrate CircleCI configuration and deploy scripts

• Migrate branch protections

If default branch is master

• [] Migrate Codeclimate configuration
• [] Migrate Github default branch

All team members update local clones

• Dave
• Amir
• Andrew
• Eddie

1. If you don't have the master branch pulled down locally, then there is nothing you need to do. You can check with git branch.

2. If you do have the master branch pulled down locally
   Here are instructions for updating locally AFTER the changes have been made in Github, but I'm not totally sure if they are all necessary.

git checkout master
git branch -m master main
git fetch
git branch --unset-upstream
git branch -u origin/main
git symbolic-ref refs/remotes/origin/HEAD refs/remotes/origin/main

Assignee (much later)

• Deletemaster branch locally and in Github (optional)

Set up automatic deploys

Automatic deploys are set up; however, I don't remember the password for the federalist-deploy user, so we need to either reset it or set up a new user. This is the same user for federalist deploys, so if we reset the password, we need to update here as well.

cc @jeremiak