mdn / infra

(Deprecated) MDN Web Docs Infrastructure scripts and configuration

MDN infrastructure (deprecated)

Important In April 2023, MDN moved from AWS to GCP. This repository is no longer used.

This repo is maintained by the MDN Web Docs operations team, which includes MDN developers, Mozilla IT, and Mozilla Marketing Engineering and Operations (MozMEAO) site reliability engineers (SREs).

It was originally established and maintained by MozMEAO during the AWS update in 2017-2018. In 2018, MDN engineering moved to Emerging Technologies, and this repo was forked from mozmeao/infra to the MDN org.

Tools we use

Automation and infrastructure tools:

• Docker
  • software containerization
• Amazon Web Services
  • our primary cloud services platform
• Jenkins
  • including Groovy scripting
• Terraform
  • declarative infrastructure provisioning
• Python
  • including boto
• jq
  • json transformations and queries

Container technologies:

• Kubernetes
  • container orchestration in the cloud
  • see this page for more info.
• DockerHub
  • public image registry

Monitoring tools:

• New Relic
  • performance monitoring
• Papertrail
  • centralized logging
• FluentD
  • collecting logs from Kubernetes pods

How we manage our work

Our goal is to use the same principle as open source development for infrastructure management. We try to keep the planning, discussions, code, documentation, and processes open to the community. Operational access is limited to staff, but working in the open allows more staff members to take on tasks, and helps new staff get up to speed quickly.

Some data, such as API keys, are sensitive. These items are stored elsewhere and applied in production. Sensitive issues are discussed in confidential Bugzilla issues.

The MDN team uses 3-week sprints to break up work into manageable milestones. The MDN team work is tracked in GitHub issues and milestones using ZenHub. See the sprints wiki for more information.

Contributing

If you'd like to make a contribution, or you've found an issue with our work, please submit an issue and/or pull request. We're happy to take a look, however, a timeframe for review cannot be guaranteed.

Contacting us

We're in the #mdndev channel on IRC.

infra's Issues

Pick domain name for kubernetes cluster

Need to pick a few domain names before we can move forward. Here are some suggestions that we discussed in our meeting

• mdn.mozit.cloud
• mozmdn.{cloud,com,net} (This will need to be purchased)
• Something else open to suggestion

My suggestion for the layout of cluster name should be as follows: k8s.${REGION}.${DOMAIN_NAME}

Plan monitoring without Datadog

Plan on operational monitoring without Datadog, and see if Amazon Cloudwatch is a sufficient replacement.

Current Datadog usage:

• Repeats of AWS service monitoring, such as counts of ELB and CDN requests
• Kubernetes-level stats, such as bytes transmitted and pod memory usage
• The depth of Celery queues in Redis, 1 minute resolution.

Acceptance Criteria

• The tech team has a plan for monitoring in the future, which may or may not include Datadog
• Follow-on tasks are written and scheduled

Setup Standby (Disaster-Recovery) instance of MDN

Setup the standby (disaster-recovery) instance as a read-only (maintenance-mode) deployment.

User story

As an MDN developer, I want to ensure we have an instance of MDN available in another AWS region as a backup in case the production instance in the Oregon region experiences a catastrophic failure.

Acceptance Criteria

• Read-only instance of MDN running in Frankfurt-based Kubernetes cluster
• Automated tests / checklist run against this instance

Tasks:

• Create germany/prod.mm.sh (standby read-only) configuration
• Make the K8s namespace, shared-storage, and services
• Add the access URL's of the backing services (RDS, Redis) to mdn-secrets.prod
• Deploy the mdn-backup-secrets, mdn-newrelic-secrets, mdn-speedcurve-secrets.yaml, and mdn-secrets.prod
• Deploy the S3 EFS sync (make k8s-restore-cron) to pull (hourly) the EFS assets from the IT-owned S3 bucket into the IT-owned EFS volume
• Create DNS CNAME for standby.mdn.mozit.cloud that points to the new ELB for the web service created above
• Make the K8s deployments
• Request increase in the domain name limit on ACM certs (from 10 to 12) within the eu-central-1 region so I can add standby.mdn.mozit.cloud to the ELB cert
• Create new standby ELB cert
• Perform checklist

RDS module needs to be able to create a replica

At the moment rds.tf doesn't create a replication database, need to modify terraform module to be able to do this.

Setup prod database instance

Setup a prod RDS instance for MDN and make sure data from the current prod instance is copied over for testing

Datadog event collection

Collect kubernetes events using the datadog agent.

Docs indicate that I can set environment DD_COLLECT_KUBERNETES_EVENTS to true and it should work but it does not, so investigate more to figure out whats going on.

Read: https://docs.datadoghq.com/agent/kubernetes/event_collection/

Update ElasticSearch clients to 5.x

User story

As an MDN developer, I want to use ElasticSearch 5.x clients in Kuma, so I can use the 5.x API.

Acceptance criteria

• ElasticSearch 5.x libraries used in Kuma

References

• #8 - split from updating servers
• bug 1464093

Tasks

• Update ES client libraries to 5.x in development environment, test
• Ship new ES client libraries to production.

Update cloudfront terraform to use new ACM cert

We shouldn't be using wildcard certs for the prod resources, they should be using proper ACM certs

MDN Cutover plan

Define a detailed, step-by-step plan for switching from the MozMEAO-based to the MozIT-based infrastructure.

Acceptance Criteria

• Create a detailed, ordered list of steps required to perform the actual cut-over from the MozMEAO to the MozIT infrastructure
• Create a detailed, ordered list of steps required after cut-over has been completed
• Get approval from @jwhitlock, @limed, @metadave, and @jgmize

Set up a Disaster Recovery (DR) cluster and AWS resources

A Disaster Recovery (DR) cluster is a standby cluster that can rapidly be promoted if there is a region-level failure in AWS.

Acceptance Criteria

• A disaster recovery cluster is configured
• A read-only replica of the new production database is configured
• The standby-push branch updates the cluster applications

Setup an elastic.co account

This is a hosted elastic search service thats needed. Setup an an account here and get a corp card placed in on that account. Budget is around $320/month $271/month with an annual contract.

Terraform ACM module doesn't create SAN cert

The simple module I wrote doesn't do SAN certs, should modify this module to be able to do this. Backburner work since we can still create a cert manually for now.

Setup deadmans snitch account

Set an account up here https://deadmanssnitch.com/

Setup stage MDN instance

Once #25 and #20 is setup we can stand up a stage MDN instance

Cluster needs worf

Make it so

Create and test a K8s Job that pulls from an S3 bucket into EFS (Cutover Task)

User story

As an MDN staff developer, I want to ensure that we're ready for executing the portion of the cutover plan where we copy the EFS data from MozMEAO to MozIT, so create and test a K8s Job on the MozIT cluster (based on mdn-sync-from-s3-cron.yaml.j2) that pulls from the mdn-shared-backup-c2037ed87dd96008 S3 bucket into EFS.

Acceptance Criteria

• K8s Job created that pulls from an S3 bucket into EFS
• K8s Job has been successfully tested on the MozIT cluster

Tasks:

• Create K8s Job that pulls from an S3 bucket into EFS
• Create a Makefile target that runs the K8s Job
• Test the K8s Job on the MozIT cluster

Setup another kube cluster in same region but different AZ

Stand up another kubernetes cluster in a different AZ so that we can support a multi cluster single region cluster

Log cloudfront logs to an s3 bucket

Create an s3 log bucket for cloudfront logs

Create and test a K8s Job that pushes EFS data to an S3 bucket (Cutover Task)

User story

As an MDN staff developer, I want to ensure that we're ready for executing the portion of the cutover plan where we copy the EFS data from MozMEAO to MozIT, so create and test a K8s Job on the MozMEAO cluster (based on mdn-backup-cron.yaml.j2) that pushes EFS to the mdn-shared-backup S3 bucket.

Acceptance Criteria

• K8s Job created that pushes EFS data to an S3 bucket
• K8s Job has been successfully tested on the MozMEAO cluster

Tasks:

• Create K8s Job that pushes EFS data to an S3 bucket
• Create a Makefile target that runs the K8s Job
• Test the K8s Job on the MozMEAO cluster

Broken links in repo causes thinBackup to fail

Broken symlinks for whatever reason causes thinBackups for fail see:

https://issues.jenkins-ci.org/browse/JENKINS-18866

This in turn causes the backup plugin not delete and archive old jenkins backups. So we should avoid broken symlinks in the repo.

Setup CI instance

Need a jenkins server setup on the MDN account, depends on #20 and #19

Make CI instance data persistent

Currently in an ASG when the instance goes down it and comes back up again it does not reconfigure itself into a state that we can use.

One option we have right now is to use userdata to grab the ansible playbook on bootup to configure jenkins and nginx. And then have a backup of jenkins restored which allows a jenkins instance to terminate and come back up again and configured without human intervention.

There is already a POC that I wrote for this and just needs a PR to get this merged back in

Update jenkins role to upload to interactive-example bucket

Jenkins role needs to have permissions to upload files to the s3 bucket.

Setup papertrail account

Setup a papertrail account.

Tasks:

• Setup a free papertrail account
• Associate with shared email
• Configure logging to account
• Add credit card to account

Upgrade Elasticsearch Server from 2.4 to 5.6

User Story

As an MDN visitor, I want to use search after August 28, 2018, because I don't want to leave the site to use a better search engine.

Acceptance Criteria

• Elasticsearch 5.6 used in staging, production, etc.

Background

Elasticsearch 2.4 reaches end of life (EOL) on February 28, 2018, and since our Elastic Cloud service for MDN is currently running 2.4, we will need to upgrade. However, the Elastic Cloud service will support version 2.4 for 6 months longer, until August 28, 2018.

• bug 1464093 - More details
• mozmeao/infra#666 - Original issue

This is from an email I received from Elastic:

We highly recommend all customers to upgrade their 2.x clusters to Elasticsearch 5.6. Elasticsearch 5.6 has lots of goodness that will make your life a bit more zen and it is the perfect bridge to the new 6.0 release.

Usually, EOL means that we won’t provide support for clusters after this date. However, since this is the last version on the 2.x Elasticsearch branch, we understand that it can take time to test your application on a new major version, iron out any issues and complete the upgrade. We will therefore support Elasticsearch 2.4 clusters until August 28, 2018. If you have not upgraded your Elastic Cloud clusters by this date, Elasticsearch 2.4 will no longer be maintained and supported after this date.

Unlike the 0.x to 2.x update, 5.x requires that the client libraries are the same versions at the server.

Tasks

• Explore tasks to update to 5.x
• Update ES queries to 2.x usage
• Update ES client libraries to 2.x, update ES queries to 5.x
• Update ES server, client libraries to 5.x in development environment
• Create ES 5.x server in elastic.co
• Ship ES 5.x to production, recreate indexes

Setup ark on cluster

Set this up https://github.com/mozmeao/infra/blob/master/docs/k8s_ark_backup_restore_docs.md

Review Accessibility Checker Plugin for CKEditor

User story

As an MDN user, I'd like the ability to check my document contributions for accessibility via the CKEditor. Kuma PR #4989 has already been submitted for review, but needs to be reviewed.

References

• bug 1169350 - Add the Accessibility Checker plugin to CKEditor

Acceptance criteria

• Review/merge Kuma PR #4989

Setup resources for interactive examples

Interactive examples need a cloudfront distribution and an s3 bucket. Get this setup on the aws account

Move ip whitelist for ssh and https to security group

Move from NACL -> SG

kube2iam should be installed on the cluster

Something to look into at a later time but this is something that should be installed on the cluster

https://github.com/jtblin/kube2iam

Update Primary Stage/Prod CDN's to Support Contribution Feature

User story

As an MDN developer, I want to ensure that the primary stage and production CDN's support the new contribution feature.

Acceptance Criteria

• New dwf_contrib_beta cookie remains unchanged when switching among the landing page and/or the document pages on stage (developer.allizom.org)
• New dwf_contrib_beta cookie remains unchanged when switching among the landing page and/or the document pages on prod (developer.mozilla.org)

Tasks:

• New dwf_contrib_beta cookie is added to the cookie whitelist on all appropriate behaviors on the primary stage CDN as well as the Terraform
• New dwf_contrib_beta cookie is added to the cookie whitelist on all appropriate behaviors on the primary prod CDN as well as the Terraform

Grow kubernetes cluster size

Grow the cluster to reflect whats on prod at the moment.

Current cluster size

Desired: 10
Min: 10
Max: 15

Create and test multi-branch pipeline for the https://github.com/mdn/interactive-examples repo on new CI/CD service.

This is a follow-on task from #22.

User story

As an MDN developer, I want to ensure that the Jenkinsfile for https://github.com/mdn/interactive-examples runs on MDN's new IT-owned CI/CD service so that we can continue to build the site, send notifications, and sync to its S3 bucket.

Acceptance criteria

• The mdn/interactive-examples Jenkinsfile runs successfully for master and prod branches
• Notifications successfully sent to IRC #mdndev
• build stage runs successfully for the prod branch
• s3 sync stage runs successfully for the prod branch
• Follow-on tasks are documented

Tasks

• Configure Jenkins for multi-branch pipeline for mdn/interactive-examples
• Install awscli on Jenkins service
• Create AWS IAM role for syncing from Jenkins to the S3 bucket
• Install credentials for IAM role above on the Jenkins service
• Create new S3 bucket for interactive examples (add to Terraform?)
• Modify Jenkinsfile to handle both the IT-owned and the MozMEAO-owned Jenkins services
• Test as detailed in the acceptance criteria

Automate Daily RDS Backups

User story

As an MDN developer, I want to ensure we backup the production database on a regular basis via an automated process.

Acceptance Criteria

• Backups of the production database are automated to be made daily
• First of the backups has completed successfully
• Backup successfully decrypted

Tasks:

• Create and register a new mdn-rds-backup Docker image on DockerHub (mdnwebdocs)
• Create or ensure that an S3 bucket for RDS backups has been created and is defined in the Terraform
• Create an AWS IAM user for syncing from the K8s CronJob to the S3 bucket
• Create and deploy secrets for running the RDS backups
• Create a K8s CronJob from the current RDS backup tool
• Deploy the K8s CronJob
• Decrypt a backup file

Install route53 mapper addon to cluster

This will be a handy addon to have: https://github.com/kubernetes/kops/tree/master/addons/route53-mapper

This allows DNS records to be mapped to an elb when it gets created by kubernetes

Create kubernetes cluster using kops

Use kops to create a new cluster, kops will also create the VPC. From there we can deploy the relevant infra needed to support kubernetes

Test CDN cutover/rollback on stage (developer.allizom.org)

User story

As an MDN staff developer, I want to ensure that we're ready for executing a downtime-free cutover from the MozMEAO CDN's to the MozIT CDN's, so work with AWS to test a cutover from MozMEAO's primary stage CDN to MozIT's primary stage CDN, as well as a rollback.

Acceptance Criteria

• Downtime-free cutover from MozMEAO's to MozIT's primary stage CDN (developer.allizom.org)
• Downtime-free rollback from MozIT's to MozMEAO's primary stage CDN (developer.allizom.org)

Tasks:

• Schedule a date and time for the cutover/rollback with AWS
• Immediately prior to the cutover, run the headless tests against developer.allizom.org in a continuous loop
• Add a TXT record for developer.allizom.org whose value is the domain name of MozIT's primary stage CDN
• Request that AWS change the developer.allizom.org CNAME in CloudFront to point to the MozIT CDN
• Monitor New Relic and Papertrail to ensure there is no downtime
• Request that AWS change the developer.allizom.org CNAME in CloudFront to point back to the MozMEAO CDN
• Monitor New Relic to ensure there is no downtime

Setup a datadog account

Estimate: ~$300/month ($15/host)

Rename s3 buckets

Some of the names of the buckets can get confusing, renaming it to the following

mdn-backup-bucket-xxxx-> mdn-rds-backup-xxxxx
mdn-shared-backup-xxxx -> mdn-efs-backup-xxxxx

MDN Cutover Review

Before executing the MDN cut-over plan, we'd like to do a review of the MozIT infrastructure, so we don't miss anything and avoid surprises.

Acceptance Criteria

• Sanity check infrastructure within the following areas:
  • Interactive examples S3 bucket and CDN (prod) including SSL cert check
  • File attachments CDN (prod) including SSL cert check
  • Primary CDN (stage and prod) including SSL cert check
  • AWS ELB cert check (stage)
  • AWS ELB cert check (standby)
  • AWS ELB cert check (prod)
  • AWS Route53 DNS checks (stage, prod, and standby)
  • Sentry setup (stage, prod, and standby)
  • Dead Man's Snitch setups (stage, prod, and standby)
  • Elasticsearch setups (stage and prod)
  • New Relic setups (stage, prod, and standby) including deployment recording
  • Data-Dog setup (prod)
  • Papertrail setup (stage, prod, and standby)
  • PagerDuty setup (prod)
  • RDS backups (prod)
  • EFS backups/restore (stage, prod, and standby)

Post Cutover - AWS & Jenkins Tasks

User Story

As an MDN staff developer, after successfully executing the MDN cut-over plan, I'd like to make sure that updates to the MDN repos no longer trigger actions upon MozMEAO-related infrastructure, as well as that any we take care of any AWS tasks (e.g., the sample database downloads are handled from the MozIT AWS account rather than MozMEAO account).

Acceptance Criteria

• The mdn-downloads S3 bucket and its content have been moved from the MozMEAO account to the MozIT account
• The four (4) MDN pipelines within the MozMEAO Jenkins service have been paused/disabled so that we no longer trigger effects within the MozMEAO world (e.g., image pushes to quay.io as well as pushes to MozMEAO Kubernetes clusters)
• Temporary IAM users within the MozIT account that were created solely for the pre-cutover period have been deleted
• Unused S3 buckets have been removed from the MozIT AWS account

Tasks

• Copy the content of the MozMEAO mdn-downloads S3 bucket to a temporary S3 bucket in the MozIT account
• Delete the MozMEAO mdn-downloads bucket
• Create a new mdn-downloads bucket within the MozIT account (with public access)
• Transfer data from temporary S3 bucket to the mdn-downloads bucket
• Delete the temporary S3 bucket
• Pause/disable the mdn_multibranch_pipeline (Kuma) within the MozMEAO Jenkins service
• Pause/disable the kumascript_multibranch_pipeline (Kumascript) within the MozMEAO Jenkins service
• Pause/disable the mdn_interactive_examples pipeline within the MozMEAO Jenkins service
• Pause/disable the MozMEAO MDN backup pipeline within the MozMEAO Jenkins service
• Delete the migration-sync-user from the MozIT AWS account
• Delete the following unused S3 buckets from the MozIT AWS account and from the Terraform code in the Infra repo:
  • mdn-db-storage-anonymized-c2037ed87dd96008
  • mdn-db-storage-c2037ed87dd96008
  • mdn-db-storage-c2037ed87dd96008-logs
  • mdn-downloads-c2037ed87dd96008
  • mdn-downloads-c2037ed87dd96008-logs

Ensure "standby-push" branch works for the kuma and kumascript pipelines on new CI/CD service.

User story

As an MDN developer, I want to ensure our CI/CD code for the "standby-push" branch of the kuma and kumascript pipelines works within IT's CI/CD infrastructure so that we can deploy code to the new standby cluster.

Acceptance criteria

• The Kuma Jenkinsfile runs successfully (runs tests and deploys code to the new germany cluster) for the standby-push branch
• The Kumascript Jenkinsfile runs successfully (runs tests and deploys code to the new germany cluster) for the standby-push branch

Tasks

• Configure Jenkins with Kubernetes credentials for the new standby cluster in germany (@limed, @escattone)
• Modify the Groovy utils in Kuma such that get_region() returns germany instead of frankfurt for the IT-owned Jenkins pipeline (@escattone)
• Test both the Kuma and Kumascript Jenkinsfile (as detailed in the acceptance criteria) (@escattone)

Publish docker images from CI server

Moving issue from mozilla-itcloud/mdn-migration-project#15

This is copy and pasted from repo listed above:
Build docker images and push them to the repository from the CI server. The MozMEAO Jenkins will continue building and pushing its own images.

• MozMeao repository - https://quay.io/repository/mozmar
• IT Cloud repository - https://hub.docker.com/u/mdnwebdocs

Image	MozMEAO name	IT Cloud name
kuma_base	quay.io/mozmar/kuma_base	mdnwebdocs/kuma_base
kuma	quay.io/mozmar/kuma	mdnwebdocs/kuma
kumascript	quay.io/mozmar/kumascript	mdnwebdocs/kumascript

Tasks:

• Allow name to be configured from environment (devs)
• Configure kuma pipeline to use IT Cloud name (IT)
• Configure kumascript pipeline to use IT Cloud name (IT)
• Determine credentials needed to push images (devs)
• Provision and install push credentials on IT Cloud CI server (IT and devs)
• Push kuma_base and kuma images from kuma pipeline (IT and devs)
• Push kumascript images from kumascript pipeline (IT and devs)

Create TXT record to validate CDN's

Read this first, this is to allow us to prove we own the domain which will allow AWS to actually migrate the CDN cname from one CDN to the other.

Execute MDN Cut-Over Plan

User story

As an MDN developer, I want to move off of the MozMEAO-based infrastructure and onto the MozIT-based infrastructure.

Acceptance Criteria

• MDN Stage (developer.allizom.org) is running from MozIT-based infrastructure
• MDN Production (developer.mozilla.org) is running from MozIT-based infrastructure
• Interactive examples (interactive-examples.mdn.mozilla.net) are served from the MozIT-based infrastructure (S3 bucket and CDN)
• File attachments (mdn.mozillademos.org) are served from the MozIT-based CDN

Tasks:

• Execute the cut-over plan listed here (also see #43)

Cloudfront doesn't allow a same alias to be created when there is an existing one

Cloudfront will not allow you to create another alias that is associated to an existing distribution. In our case interactive-examples.mdn.mozilla.net exists in the MozMEAO AWS account, and we are therefore unable to create a distribution with interactive-examples.mdn.mozilla.net as an alias. In the interest of moving forward I have commented out that line out in terraform here

But when we cutover we need a plan to make the cutover seamless. I spoke to our AWS TAM and this is what he said:

15:14:04 ChrisAWS | Okay, I’ve got a possible solution.
15:14:45 ChrisAWS | The TXT record mentioned in that article is just to prove that you control the domain. Once AWS can validate that you control it, they will point traffic at the new distribution.
15:15:09 ChrisAWS | That article is more for people who have had their CNAME associated with a distro by someone not in their org.
15:15:20 ChrisAWS | In your case, here’s what you can do:
15:16:27 ChrisAWS | Let’s call the old distro 1 and the new distro 2
15:16:45 ChrisAWS | You set up distro 2 in the new account, then point the CNAME at distro 2.
15:17:06 ChrisAWS | Cloudfront will continue to route traffic to distro 1, because it’s the distro associated with the CNAME in our system.
15:17:46 ChrisAWS | You then open a support case and we validate that you control the CNAME by noticing that it points to distro 2.
15:17:58 ChrisAWS | And we swap the CNAME association to distro 2.
15:18:35 ChrisAWS | It takes about 15-20 minutes for that change to propagate across our edge locations, during which time both distro 1 and distro 2 receive traffic. After that, all traffic will route to distro 2
15:18:40 ChrisAWS | Would that work?

MDN Cutover EFS/RDS Transfer Dry-Run

Before executing the MDN cut-over plan, we'd like to do a dry-run of the RDS (DB) and EFS (file attachments and legacy files) content transfers, so we are ready for the actual cut-over on the morning of Oct. 29th and so we have an accurate idea how long the transfer operations will take.

Acceptance Criteria

• EFS content transferred from MozMEAO to MozIT (in the MDN Cutover Plan, document the commands required if they aren't already)
• Time estimate for the EFS content transfer when we do the actual cutover
• RDS content transferred from MozMEAO to MozIT (in the MDN Cutover Plan, document the commands required if they aren't already)
• Time estimate for the RDS content transfer when we do the actual cutover
• Document any follow-on tasks as new issues

Setup supporting infra

Will need to setup supporting infra

• EFS
• ACM
• Redis
• RDS
• Memcached

Post Cutover - Code Cleanup

User Story

As an MDN staff developer, after successfully executing the MDN cut-over plan, I'd like to cleanup/remove the code that's related to the MozMEAO infrastructure so we maintain less code going forward.

Acceptance Criteria

• Review Kuma repo for any code related to the MozMEAO infrastructure, and submit/merge any pull requests necessary to remove it.
• Review Kumascript repo for any code related to the MozMEAO infrastructure, and submit/merge any pull requests necessary to remove it.
• Review Infra repo for any code related to the MozMEAO infrastructure, and submit/merge any pull requests necessary to remove it.
• Review Private repo for any code related to the MozMEAO infrastructure, and submit/merge any pull requests necessary to remove it.

Setup Prod (Read-Only PoC) instance of MDN

Once stage is up and running setup the prod instance as a read-only, proof-of concept deployment.

Acceptance Criteria

• Read-only snapshot of production available at new MozIT cloud domain
• Automated tests / checklist run against new instance
• Follow-on tasks for go-live are captured in new issues

Tasks:

• Decide if we start with production-sized services (do it once, start paying) or stage-size services (pay less for a month or so) -- 9/10/2018 (@escattone) starting with stage-size services
• Combine oregon/stage.sh and portland/prod.mm.sh into oregon/prod.mm.sh (production read-only) and oregon/prod.mm.min.sh (production read-only but with stage-size replica counts)
• Provision prod CloudFront for the primary, attachments, and interactive-examples CDN's
• Provision prod database (production size)
• Provision prod redis (production size)
• Provision prod memcache (production size)
• Provision prod elasticsearch (production size)
• Download encrypted, compressed dump of the production RDS instance in MozMEAO
• Decrypt and load the compressed DB dump into the new prod RDS instance in IT
• (Optional) Anonymize the new prod RDS instance in IT
• Copy EFS backup (attachments) from prod S3 bucket into new S3 bucket
• Make the K8s namespace, shared-storage, and services
• Add the access URL's of the backing services (RDS, Redis) to mdn-secrets.prod
• Deploy the mdn-backup-secrets, mdn-newrelic-secrets, mdn-speedcurve-secrets.yaml, and mdn-secrets.prod
• Deploy the S3 EFS sync (make k8s-sync-from-s3-cron) to pull (hourly) the EFS assets from the IT-owned S3 bucket into the IT-owned EFS volume
• Create DNS CNAME for prod.mdn.mozit.cloud that points to the new ELB for the web service created above
• Create DNS CNAME for developer-prod.mdn.mozit.cloud that points to the new prod primary CDN
• Create DNS CNAME for demos.mdn.mozit.cloud that points to the new prod attachments CDN
• Create DNS CNAME for interactive-examples.mdn.mozit.cloud that points to the new interactive-examples CDN
• Create, validate, and attach new certificates for the new web ELB, the prod primary CDN, the attachments CDN, and the interactive-examples CDN (see comment below)
• Make the K8s deployments
• Login to the MDN admin console and change the Site to developer-prod.mdn.mozit.cloud
• Create, populate, and promote new search index via the admin console
• Make the humans and sitemaps (run ./manage.py make_humans and ./manage.py make_sitemaps on the web pod)
• Perform checklist