eclk / deployment

Kubernetes and overall deployment aspects

deployment

Kubernetes and overall deployment aspects

deployment's Issues

Deploy results distribution app to dev

The code is not yet committed but it will be soon :-).

The structure of the code will be like the backend of the Staffing app as its being written in Ballerina.

The system has a website for media people to login and access and also has a mechanism to push the data out (via registered webhooks). The webhook subscription URL has to be open to anyone to access (details will be given). We haven't yet decided whether to use IS for authentication or not but we can decide that later.

Lets set up mediaresults.ecdev.opensource.lk as the host please.

The build and deploy process will be same as for Staffing.

Fix tests in the hub to not use the same hub as that's set up with DB creds which may not be there at build time

Create a Playbook for Production and Staging Environments

Setup Staging MySQL Master Slave Solution

To research and find out the best solution for HA MySQL

Production K8s install cluster essentials

Setup Production Monitoring and Alerts

Setup according to monitoring and alert specs

Setup CI/CD on Staging

Setup nginx ingress on k8s dev cluster

Deploy results tabulation user admin app to dev

Repo location: https://github.com/ECLK/results-tabulation-admin-app.

Please deploy as: resultstabadmin.ecdev.opensource.lk.

Get EC DR site hardware from EC team

Production wso2-apim integration

Staging wso2-apim integration

Production Bastion Host

Bastion host should have

A public IP Address

NFS mount on /data
mysql client connected to prod mysql server
Kubectl connected to prod k8s server
Helm connected to prod k8s server

Staging K8s Cluster install essentials

1. Nginx Ingress
2. Cert Manager
3. K8s Dashboard
4. Metrics Server
5. Helm

Get a wildcard dns for Staging Environment

*.ec-staging.opensource.lk A record pointed to the Staging LB ip address. (Kosala to comment the IP after provisioning)

Production NFS Server

Setup Staging Load Balancer

HA Proxy to be used for it's simplicity and light weightedness

Finalise Staging and Production Infra specs

Production Test run

Run the tests that are in the test doc and confirm that the environment is in a production ready state

Get new SSDs for staging deployment from EC

Request made by Nilesh already

Integrate EFK Logging Solution for k8s dev cluster

Setup Staging Logging

Setup Production HA K8s Cluster

Deploy results app to dev

Repo URL: https://github.com/ECLK/Results/.

Desired URL: results.ecdev.opensource.lk.

Note however that production deployment will be in a totally different cluster due to need to scale separately.

(Creating issue for record purposes).

Setup Staging HA K8s Cluster

Staging Test run

Run tests against the test doc and make sure Staging is in production grade state

Setup Production CI/CD

Increase Memory and CPU of Worker Nodes in Dev Environment

Get DNS entries created with government for Production applications

Enable Production Audit logs

MySQL Audits
Kubernetes Audits

Setup CI/CD on K8s dev cluster

Setup DR site

Get data backup VM from EC

Deploy staffing system backend APIs for dev environment

The process is:

• repo: Staffing/backend
• build: ballerina build services
  will produce target/bin/services.jar
• build docker image using Alpine Linux OpenJDK image
• command to RUN is java -jar services.jar
• map port 80 to port 9090
• temporary (proper fix is to expose under API manager)
  • create a new k8s service called "staffing-backend" for this service

Integrate Monitoring for K8s Dev Cluster

Integrate wso2-apim for K8s dev cluster

Setup Staging NFS Server

Setup Production Loadbalancer

Setup cert-manager on k8s dev cluster

Production Backup Plan

Need to do research on this and a step by step guide on what exactly we are going to do with MYSQL and NFS Server data

Create DB user for results distribution

Results dist needs to store subscribers in the DB.

Production Alert Resources

Identify the alerts that are needed.
Get the list of emails
Get the list of Phone numbers
Get any other social messaging platforms required (Slack, Google chat etc)

Setup Production MySQL Master Slave

Complete dev K8S cluster

The cluster is to be setup on data center

Staging Bastion Host

Bastion host should have

A public IP Address

NFS mount on /data
mysql client connected to staging mysql server
Kubectl connected to staging k8s server
Helm connected to staging k8s server

Setup Staging Monitoring and Alerts

Setup CICD in QA Environment

Production Backup system implementation

Implement the system with CronJobs according to the document created on backups

Provision Staging VMs and Open Network Ports

Draft test cases for production and staging environments