mozilla-services / dockerflow

Cloud Services Dockerflow specification

Dockerflow

Dockerflow is a specification for automated building, testing and publishing of docker web application images that comply to a common set of behaviours. Compliant images are simpler to deploy, monitor and manage in production.

The specification is this README.md file. This repo contains a reference application that will change with the specification. See the Contribution document for details on suggesting changes and providing feedback.

Automated Creation Pipeline

  +-(1)--+         +-(2)-------------+        +-(3)-------+        +-(4)--------+
  | Code |         |  CI builds and  |        |   Docker  |        | Container  |
  | Push | ------> | tests container | -----> |    Hub    | -----> | Deployment |
  +------+         +-----------------+        +-----------+        +------------+

1. A Code push triggers automated image building
2. CI builds and tests the image
3. Tested images are published to Docker Hub.
4. Images are pulled from Docker Hub to be used

Specification

The specification has requirements that a container must comply with. Recommendations are optional but encouraged if they are suitable for the application.

Building and Testing

Dockerflow requires an automated build and test tool that meets these requirements:

1. Able to trigger a build from a code change like a pull request or a merge.
2. Able to run tests on the code and the application in the container.
3. Able to manually trigger a previous build and test.
4. Able to publish container to Docker Hub or Google Artifact Registry.
5. Able to provide a build and test log.
6. Secure and keeps secrets from being exposed.

Within Mozilla, we support the use of CircleCI, Taskcluster, or Github Actions.

Containerized App Requirements

When the application is ran in the container it must:

1. Accept its configuration through environment variables.
2. Listen on environment variable $PORT for HTTP requests.
3. Must have a JSON version object at /app/version.json.
4. Respond to /__version__ with the contents of /app/version.json.
5. Respond to /__heartbeat__ with a HTTP 200 or 5xx on error. This should check backing services like a database for connectivity and may respond with the status of backing services and application components as a JSON payload.
6. Respond to /__lbheartbeat__ with an HTTP 200. This is for load balancer checks and should not check backing services.
7. Send text logs to stdout or stderr.
8. Serve its own static content.

Dockerfile requirements

1. Sources should be copied to the /app directory in the container.
2. Create an app group with a GID of 10001.
3. Create an app user with a UID of 10001 and is a member of the app group.
4. Have both ENTRYPOINT and CMD instructions to start the service.
5. Have a USER app instruction to set the default user.

Optional Recommendations

1. Each instance of a container should only run one process. If you need to run two processes, just create another instance of the container & run a different command.
2. Log to stdout in the mozlog json schema.
3. Containers should be optimized for production use.
4. Listen on port 8000.

Docker Hub Credentials

Internal processes for managing DOCKER_USER and DOCKER_PASS per-project are documented in hiera-sops/misc/dockerhub/.

Contributing

• Contribution Guidelines

dockerflow's Issues

Dockerflow apps shouldn't have to serve static files.

There isn't a good story around the serving of static files that are part of a Dockerflow app.

Write Recommendation for an ENTRYPOINT shell script

A pattern that has emerged is a container is good for more than just one thing, like the main web app and a background worker process. Often they share all the same libraries and differ in the way they are run.

I tested something out with a run.sh for mozilla/universal-search-recommendation. This shell script is the ENTRYPOINT and depending on the CMDs given to it can run a background worker, another task, etc.

Another example is Dockerflow itself. Where the ENTRYPOINT is npm and the CMD tells it what to do.

All of this should be captured in a recommendation document.

Unencrypted password warning

While running the Deploy to Dockerhub job, I get an unencrypted password warning:

WARNING! Your password will be stored unencrypted in /root/.docker/config.json

Wondering if it's a credible vulnerability? Is there a better way to run this job/step, and should the Dockerflow repo be updated accordingly?

Add recommandations about version prefix?

Should the application have a version prefix /v1 ? Or should the application serve on / and the prefix be set during the deployment?

Using wellknown for health checks

Why not use .wellknown name space for health checks?

https://en.wikipedia.org/wiki/List_of_/.well-known/_services_offered_by_webservers

_{Sent with GitHawk}

Remove Mozilla CloudOps specific information

The original doc was very much Mozilla CloudOps specific. These should be removed and the spec should focus on:

• simplifying deployment of a service
• simplifying identifying service version
• health checks for monitoring
• conventions for logging and metrics

CODE_OF_CONDUCT.md file missing

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

1. Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
2. Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please reach out to [email protected].

(Message COC001)

Redundant login to Dockerhub during build job

I'm trying to understand the need for Login to Dockerhub in the build job. It doesn't appear that anything is posted to Dockerhub until the deploy job, where a second login is initiated. Is the first login a redundant step?

How are services managed?

Modern apps require lots of services that typically require additional containers running alongside the application container: databases, message queues, etc. After discussion with @mostlygeek, I've gathered that:

• The intent is for the application container to exclude those services and only serve the app itself.
• In production deployments, ops will manage those services manually using CloudFormation templates.
• For local development, it's up to developers to determine the sanest way to manage those services.

Open questions:

• If there are services that require developer configuration, how should that be managed?

mozlog-formatted `Fields` can't be nested objects.

According to the pipeline team, we aren't able to nest object literals within the Fields object; just "values" and arrays. This documentation is wrong. See this mana article for the full schema.

List supported CI tools in README

In the README it says that any CI tool that meets certain requirements can be used for Dockerflow, but apparently builds from Travis CI are not supported (e.g., only Taskcluster and Circle?)

Might be good to include the supported list in the README, and maybe how support for others can be contributed?

how to do endpoints for non-webapp services

For services that are not webapps, what does Dockerflow recommend we do for healthcheck endpoints?

For example, the Socorro processor is not a webapp and doesn't have anything to respond to HTTP, so there's nothing to implement healthchecks with.

Is it the case that all services must implement a webapp to handle Dockerflow healthcheck enpoints? Should we have something else for non-webapp services?

Improve clarity of the app user in the container

From mozilla/testpilot/issues/398, the usage of the app user in the container is a little ambiguous. It should be clarified that:

• root can be used in the container to install packages and other system level dependencies
  • the app user can also be used to run commands like npm
  • the /app directory in the container should be owned by app
• the app only needs to be used to actually run the service
• this can be accomplished with the USER command in the Dockerfile

Add open source software license

This Mozilla repository has been identified as lacking a license. Consistent with Mozilla's Licensing Policy an open source license should be applied to the code in this repository.

Please add an appropriate LICENSE.md file to the root directory of the project. In general, Mozilla's licensing policies are as follows:

• Client-side products created by Mozilla employees or contributors should use the Mozilla Public License, Version 2.0 (MPL).

• Server-side products or utilities that support Mozilla products may use either the MPL or the Apache License 2.0 (Apache 2.0).

In special cases, another license might be appropriate. If the repository is a fork of another repository it must apply the license of the original. Similarly, another license might be appropriate to match that of a broader project (for example Rust crates that Firefox depends on are published under an Apache 2.0 / MIT dual license, as that is the dual license used by the Rust programming language and projects).

Please ensure that any license added to the LICENSE.md file matches other licensing information in the repository (for example, it should match any license indicated in a setup.py or package.json file).

Mozilla staff can access more information in our Software Licensing Runbook – search for “Licensing Runbook” in Confluence to find it.

If you have any questions you can contact Daniel Nazer who can be reached at dnazer on Mozilla email or Slack.

OPENLIC-2023-01

How do we verify container integrity?

Dockerflow introduces the notion of a third party that takes source code, puts that into a container that we mostly treat as binary, and publishes it for us to use.

Assuming the third party is circleci (but it could be anyone really). We know that if circleci gets owned, someone can modify containers to inject bad code, which would be dramatic for us.

It's a bit different than trusting github, because modifying git history and source code without being detected by developers is very hard. It is similar to trusting AWS, but we justify that trust with contracts and years of proven security practices.

My question is: Do we, and should we, have a way to control what circleci produces, or do we blindly trust the service?

Logging to stdout in the mozlog json schema harmful for pre-forking servers in some situations

If your application isn't single threaded/process the optional recommendation will result
in corrupted log entries when there are frequent writes over 4k in size.

references

1. https://www.gnu.org/software/libc/manual/html_node/Pipe-Atomicity.html
2. https://www.notthewizard.com/2014/06/17/are-files-appends-really-atomic/
3. minimal gunicorn test case in https://gist.github.com/dylanwh/dbb7acc97151503ef061f0895ec0d137

work arounds

1. Prevent writes over 4k (or log additional warnings)
2. Use something to mux the input. Each worker thread/process should log to a socket and there should be a single process multiplexing over that socket and writing to the container's stdout
3. Run only a single process per container (viable for smaller services?)
4. Write your service in go, which seems to handle multiple goroutines writing to stdout just fine,

Convert to CircleCI 2.0

The 1.0 style configuration still used in this sample is supposedly retiring this August

https://circleci.com/blog/sunsetting-1-0/

Why should I use Dockerflow?

Configuration endpoint

In a few cases, we have implemented endpoints that expose the configuration of a running app. I think it would be useful to standardize how to do that, and document it as optional.

A few considerations:

• Expose ENV variables as key/value under an environment key ?
• Expose hard-coded config under configuration key ?
• Limit exposure to internal network and/or authenticated endpoints only (simple token should be enough)?
• Use the standard __config__ endpoint?

Add docker image recommendations to README

@ckolos volunteered to do this one.

Group and user creation arguments are ambiguous

Dockerflow/Dockerfile

Lines 5 to 6 in 09475d1

	RUN addgroup -g 10001 app && \
	adduser -D -G app -h /app -u 10001 app

Depending on the underlying distribution of the container, these options can be ambiguous. The options should use the full name format for readability.

What should we return if /app/version.json is missing?

• 500 ?
• Empty body?

Consider updating severity levels for MozLog

As more of our services have been migrated to GCP, MozLog's severity levels are causing issues as they're different from those of GCP Cloud Logging. Dockerflow-compliant services need to remap SLs in order for GCL to render log records properly.

Is it time to update MozLog's SLs to match against GCL now?

@jasonthomas @mostlygeek @jbuck WDYT?

/app/version.json in reproducible docker images

in mozilla-releng/services we are trying to become "dockerflow complient" (mozilla/release-services#1183) either we run services on dockerhub or elsewhere. Standards are good! :) We are not there yet but we are getting there.

Our docker images are build reproducible and the future we would like to continuously check for binary reproducibility.

The requirement JSON version object breaks the promise of reproducibility since you don't know at build time if the artifact is going to be released.

Sure we can rebuild docker image just before pushing dockerhub, but then this is not really the same artifact as we tested on staging environment.

I'm opening this ticket to get some advice how to proceed. In the worse case we can rebuild the docker image before pushing, but maybe there is a way to have this JSON version object be outside the docker image as a metadata of the running container.

Thank you in advance, Rok.

Add recommandation about default port?

use port 8000. It's more of a convention in ops.

mozilla-services/buildhub#248 (comment)

Where should the container build and test happen?

The current recommendation is to build and test the container in Circle CI. The motivation for this is:

• transparency in viewing build logs
• no in-house testing infrastructure to run
• simple integration into github pull requests
• common scriptable builds (via .travis.yml or circle.yml)

However, some questions were brought up around:

• verification of the build
• knowing exactly what we are deploying
• ability to re-build a container without dev support

@oremj @ckolos @jvehent please add anything I've missed

Where & How can I test my docker file or image?

I think I've made a dockerfile (and image) conforming to Dockerflow requirements. Where can I build/deploy it to test that it's working?

[proposal] Add an endpoint that crashes

Adding a crashing endpoint to our services could help us make that the service monitoring (logging and graph of 5XX responses) and error reporting (Sentry, slack notifs) is configured correctly.

@app.route("/__crash__")
def crash():
    logger.error("about to boom")
    raise ValueError("boom")