microsoft / containerized-microservices-pipeline

Reference solution for Production micro-services pipeline built on VSTS, docker, kubernetes and full CI / CD pipeline.

Containerized Micro-services

This project is a reference solution for building out a full Production ready micro-services solution. This solution is hosted in Kubernetes running on Azure. All apps and services have a small (micro-service) footprint of responsibilities and are run in Docker containers. This solution is meant to be a best practice example of a micro-service development and hosting solution ready for Production.

Solution Overview

• Cross platform solution (runs on Linux, Windows and Mac)
• Source repositories hosted in Git
• Continuous Integration (CI) including unit & integration tests via VSTS (Visual Studio Team Services)
• Continuous Deployment (CD) via VSTS
• Containerized services hosted in Kubernetes (k8)
• Secrets management via Azure KeyVault, k8 secrets and Hexadite
• Azure Active Directory Service Principal identities and least privilege access policies
• Helm chart k8 package deployments
• Azure Traffic Manager DNS request routing
• Traefik ingress controller k8 traffic routing
• Canary deployments
• Azure k8 cluster deployment via ACS Engine
• Application Insights telemetry
• Log Analytics k8 cluster health dashboard

Included in this Solution

Deployment scripts

Deployment scripts

Deployment scripts are provided as part of the reference solution that will setup a new Azure Subscription with all of the necessary resources, create Azure Active Directory Service Principal service accounts, grant needed access and deploy out n number of k8 clusters. Documentation for CI / CD setup in VSTS is provided to enable you to get a full end-to-end pipeline micro-service pipeline setup in no time.

Example Angular app micro-service

https://github.com/Microsoft/containerized-microservices-pipeline-app

An example front end web app written in Angular is provided as part of the solution, including a suite of unit and integration tests that are run when a Pull Request is created for a robust CI process. Changes to the app micro-service is automatically deployed to the canary pods as part of the CD process.

Example .NET Core api micro-service

https://github.com/Microsoft/containerized-microservices-pipeline-service

An example cross platform middle tier api is provided as part of the solution, including a suite of unit and integration tests that are run when a Pull Request is created for a robust CI process. Changes to the api micro-service is automatically deployed to the canary pods as part of the CD process.

Development Prerequisites

1. Mac - Homebrew

   • Installing homebrew

2. Windows - Powershell & Linux Subsystem

   • Installing Windows Powershell
   • Installing the Linux Subsystem on Windows

3. Docker

   • Install on Mac
   • Install on Windows

4. Azure CLI

   • Install Azure CLI on Mac with Homebrew

     brew install
     brew install azure-cli

   • Install Azure CLI on Windows: Download and run AZ Installer
   • For more information on installing Azure CLI

5. kubectl

   • Install kubectl on Mac with Homebrew

     brew install kubectl

   • Install kubectl on Windows with PowerShell

     Install-Script -Name install-kubectl -Scope CurrentUser -Force
     install-kubectl.ps1 [-DownloadLocation <path>]

6. ACS Engine

   • Download and install ACS Engine

7. JQ

   • Download and install JQ

8. Visual Studio

   • Will be used for editing and executing web and load tests. There's an option to run the tests locally or in the cloud through VSTS. Web and load tests examples found here.
   • Download and install Visual Studio

Deploying Azure Resources and ACS Kubernetes Cluster

Deployment

Securing Secrets

Deploying secrets to production

Traffic Routing

Routing requests to Azure and within Kubernetes

Using VSTS to set up Continuous Integration and Continuous Deployment

Continuous integration/deployment

Log Analytics

Integrating Log Analytics

Repository Structure and Strategy

We have one repository for the front end solution, one for our middle tier, and one common repository for deployment scripts. Because we are dealing with microservices, this approach allows the project to be both modular and scalable. The smaller codebases help developers contribute with faster execution and innovate rapidly due to fewer merge conflicts and clear ownership. Additionally, it allows the project to take full advantage of helm charts, helping control Kubernetes objects and application versioning. Using this structure as a reference solution, it is recommended that each additional microservice uses separate repositories for each of its respective services.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

containerized-microservices-pipeline's Issues

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Rollout release to regions

Design a flow and pattern to do a release rollout by region

containerized-microservices-pipeline-service doesn't exist anymore

The provided example service repo doesn't exist (at least I get a 404):
https://github.com/microsoft/containerized-microservices-pipeline#example-net-core-api-micro-service

Could you link an alternative repository?

Service mesh to route, restrict and secure communications

• Is Istio ready as a service mesh in production?
  • If not, what is missing?

Examples using kube-advisor

https://github.com/Azure/kube-advisor/

Replace Hexadite with Key Vault Flex Volumes

https://github.com/Azure/kubernetes-keyvault-flexvol
https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md

• Follow up if Key Vault flex volumes and AAD pod identity is production ready
• Using AAD pod identity
• Traefik uses flex volumes
• Middle tier API uses flex volumes
• Documentation updated

Allow user the ability to use terraform for resources

Move OMS agent to kube-system

Review k8s best practices documentation

Here are a few k8s best practice repos we can review against our solution. This could possibly drive additional backlog items

https://github.com/shanepeckham/AKS_Security
https://github.com/Azure/k8s-best-practices
https://github.com/Azure/blackbelt-aks-hackfest

Evaluate AppGateway Ingress, Azure Front Door and Azure Firewall

https://github.com/Azure/application-gateway-kubernetes-ingress

Complete OMS example with containers workspace

As a user deploying a k8s cluster via acs-engine, I want the same OMS/Log Analytics setup that I can get from deploying a AKS cluster

Acceptance Criteria:

• OMS/Log Analytics parity with AKS when deploying via acs-engine

HA clusters

As a user of this reference architecture I would like to know how to deploy highly available k8s clusters.

Acceptance Criteria:

• Ability to deploy a highly available cluster(s)
• Ability to turn on/off highly available feature during deployment
• Documentation on HA best practices

POC acs-engine and VMSS

With the release of k8s 1.12, you can now deploy a cluster with Azure virtual machine scale sets using acs-engine. Cluster autoscaling was also added. More details here: https://kubernetes.io/blog/2018/10/08/support-for-azure-vmss--cluster-autoscaler-and-user-assigned-identity/

We should evaluate how well this works over availability sets.

• Requires k8s 1.12. If user tries to deploy VMSS where k8s < 1.12, script should fail with the appropriate error
• 1.12 k8s cluster with VMSS
  • VMSS node pool
  • VMSS masters?
• cluster autoscaler

Design interactive version of deploy resource and deploy cluster scripts

Design how we can redesign the deployment scripts to make them generic, interactive and reusable so there is no need to modify the scripts by hand for any project. We should also include CI/CD task deployment as part of the design.

First step is to get a meeting together and start a design doc to get in agreement on what the requirements are and how we will build it. Additional user stories should be created as a result of this.

Possible options to explore:

• Azure CLI extension
• VS Code extension
• kubectl plugin (requires 1.12 of CLI)
• Aure ARM template

Add ability to enable/disable RBAC

Both AKS and acs-engine support RBAC. When deploying a cluster, add the ability to switch RBAC on/off.

AKS currently has the ability to switch this on/off. RBAC is enabled by default in acs-engine

If RBAC is turned on, we should "wire up" the cluster with RBAC examples to give the user a good understanding of best practices using RBAC in a k8s cluster.

Acceptance Criteria:

• When deploying a cluster, user has ability to turn RBAC on or off
• Wire up and document RBAC policies for appropriate deployments, secrets etc.

Log collector examples

• Example(s) using Azure resources
• Example(s) using open source resources

Update ReadMe with documentation for PFX with no password

End to end testing

We should add some type of end to end testing that can be ran on PRs and PR merges.

Examples:

• Basic linting on deployment scripts
• Run scripts end-to-end and verify there are no errors
• Run basic operations on cluster to determine correct functionality
• Teardown cluster at the end of tests

Acceptance Criteria:

• On every PR and PR merge, I want the ability to run tests on the deployment scripts
• Documentation on end to end testing
• (Bonus) Azure DevOps pipeline YAML committed with steps

k8s VSTS pipeline yeoman integration

We have a pull request out for a yeoman generator that will setup CI/CD for a k8s microservice in Azure DevOps. We need to fix any issues on this PR, get it merged and integrate into this project.

PR:
DarqueWarrior/generator-team#22

Shell script character limit check for project and cluster name

Add documentation to configure Traefik routes after deploying Traefik ingress controller

Could we deploy the ingress object with routes as part of the CD process?

Add thorough documentation on how this should work.

Add ability to deploy an AKS cluster

Currently we only deploy a k8s cluster using acs-engine. Add the ability for the user to choose between AKS (managed) or ACS engine (self-managed).

• Add a flag in deployCluster.sh to switch on deploying acs-engine vs aks.
  • Deploying via acs-engine should be default ie AKS flag committed as turned off
• Update Deployment.md with details on how to deploy AKS vs acs-engine

Possible refactoring: Split acs-engine deploy and AKS deploy into separate shell files

Improve deployCluster.sh use az to get values instead of using output from inception.sh

Rotate Certificate for Ingress TLS

Applying the certificate is documented here. https://docs.microsoft.com/en-us/azure/aks/ingress-tls
Its suggested in this doc to use cert-manager but, the github for cert-manager clearly indicates its not production ready. Therefore, this issue should indicate how admins should rotate the certificate for ingress TLS before it expires.

Distributed tracing examples

vsts-npm to auth with vsts doesn't run cross plat

deployCluster.sh should move & rename the ACS engine local folders

Need to remove (rm -rf) or override folders and files created by ACS engine.

Examples of releases

CD, daily, weekly etc

Exit out of deployment scripts on error or failure

Implement helm install --name omsagent in deployCluster.sh