milesmatthias / gke-enterprise-demo

This demo aims to show what a fully-featured project running in Kubernetes Engine looks like. It includes Elasticsearch, a very popular open-source project for indexing and searching data, as well as some custom software to interface with it.

Kubernetes Engine Enterprise Demo

• Introduction
• Architecture
  • RBAC Setup
  • Build and Push Pyrios Docker Image
  • Elasticsearch Cluster HA Set Up With Regional Persistent Disks
• Prerequisites
  • Tools
• Configure gcloud
• Enable the GCP Services
• Run Terraform for Infrastructure Provisioning
• Configure
• Deploy Kubernetes Resources
• Expose the pyrios Pod
  • Load Sample Data to the Elasticsearch Cluster
  • Validation
  • Web Page User Interface
• Teardown
• Troubleshooting
• Relevant Material

Introduction

Kubernetes Engine has made the process of administering a k8s cluster extremely simple and allows teams to focus on what matters instead of pain points like upgrading etcd.

This demo aims to show what a fully-featured project running in Kubernetes Engine looks like. It includes Elasticsearch, a very popular open-source project for indexing and searching data. Also included are 2 demonstration applications that are used to show some advanced features in GCP and a fully operational Bazel pipeline and workflow.

Some of the highlights included are:

• Rolling upgrades
• k8s services
• Using Cloud VPN to connect disparate networks
• Terraform as IAC
• Using Google Cloud Build
• Using Stackdriver tracing
• Using Stackdriver monitoring

Architecture

The demo does not use an actual on-prem data center so we have to emulated an on-prem data center by creating a separate VPC network. We are using two different GKE clusters. The first GKE cluster hosts a deployment of an Elasticsearch database, and the second GKE cluster hosts an example application. The GKE cluster that hosts Elasticsearch emulates an GKE On-Prem cluster running in your data center.

The two different networks, that contain the GKE clusters, are connected via Cloud VPN. It's common practice to connect remote networks together with VPNs and the cloud is no exception. Other options exist including GCP Interconnect.

You can setup a Cloud VPN to connect your cloud VPC to your on-prem data center. We use forwarding rules to direct traffic sent to the public IP addresses to the gateways. The demo creates an IPSec tunnel between the gateways using a shared secret. A VPN gateway with subnet routes and traffic selectors, ensures the data center traffic egresses. All of this configuration can be found in the 'datacenter' module of this demo.

The Elasticsearch pods are accessed via their Cluster IPs which is within the Alias IP range of the VPC. Once the networks are connected the application is able to connect to the 'on-prem' Elasticsearch cluster.

The 'on-prem' datacenter has an eight pod Elasticsearch cluster running client, master, and data pods. The data pods are deployed as a StatefulSet. The nodes in the GKE cluster are type n1-standard-4. The manifests are based on the pires Elasticsearch project. We updated the memory and added pod disruptions budgets.

In order to expose the on-premise Elasticsearch as a service to the cloud Kubernetes Engine cluster, we need to expose it via an Internal Load Balancer (ILB). By doing so, the Elasticsearch cluster can be accessed by any application running in the cloud. The traffic will travel through the VPN tunnel between the cloud network and the on-prem network. es-svc.yaml shows how it is implemented by a kubernetes annotation, cloud.google.com/load-balancer-type: "Internal", which specifies that an internal load balancer is to be configured. Please refer to Creating an internal load balancer for details.

The cloud datacenter contains the second GKE cluster running a single node with a single pod in it. The pod is running a custom application called 'pyrios' It acts as a proxy to the on-prem Elasticsearch cluster.

pyrios is a minimalist proxy server for Elasticsearch. The key idea is to proxy the REST request from the cloud Kubernetes Engine cluster to the on-prem Elasticsearch cluster.

The demo includes a simple UI that is fed by data from Elasticsearch. It shows the same information that validate.sh shows. You can read more in the Web Page User Interface section.

RBAC Setup

Role-Based Access Control is used for authenticating Elasticsearch data node's graceful shutdown script. During the shutdown, the shutdown script (pre-stop-hook.sh in manifests/configmap.yaml) needs to access the stateful set's status in the on-prem Kubernetes Engine cluster. Hence, we need to create a service account, cluster role and cluster role binding for this to work. Under manifests folder

• clusterrole.yaml: a ClusterRole elasticsearch-data for reading statefulset
• service-account.yaml: a ServiceAccount elasticsearch-data
• clusterrolebinding.yaml: a ClusterRoleBinding elasticsearch-data to bind the cluster role and service account declared in the above.

Build and Push Pyrios Docker Image

This step is only necessary if you are interested in building and pushing your own pyrios docker image onto your own project's gcr.io image repository.

• The demo itself uses a public image lives at gcr.io/pso-example/pyrios.
• cd pyrios && make container uses the GCB(Google Container Builder) for building and pushing docker image to gcr.io/${PROJECT_ID}, i.e. your own project's gcr.io image repository.

Elasticsearch Cluster HA Set Up With Regional Persistent Disks

The Elasticsearch cluster uses regional persistent disks to improve its storage resiliency. These disks are replicated across multiple zones in a region. The Elasticsearch cluster in the demo uses the 'regional-pd' volume type for its data nodes. Once the clusters are setup you can see for yourself with the following command. Note that the LOCATION_SCOPE says 'region'.

Execute:

gcloud beta compute disks list --filter="region:us-west1"

Example output:

NAME                                                             LOCATION     LOCATION_SCOPE  SIZE_GB  TYPE         STATUS
gke-on-prem-cluster-f1-pvc-9cf7b9b3-6472-11e8-a9b6-42010a800140  us-west1  region          13       pd-standard  READY
gke-on-prem-cluster-f1-pvc-b169f561-6472-11e8-a9b6-42010a800140  us-west1  region          13       pd-standard  READY
gke-on-prem-cluster-f1-pvc-bcc115d6-6472-11e8-a9b6-42010a800140  us-west1  region          13       pd-standard  READY

Prerequisites

Tools

1. Terraform >= 0.11.7
2. Google Cloud SDK version >= 204.0.0
3. kubectl matching the latest GKE version
4. bash or bash compatible shell
5. jq (very common, can be found in any package manager)
6. bazel (can also be found in most package managers)
7. A Google Cloud Platform project where you have permission to create networks

This demo has been tested with macOS and Cloud Shell.

You can obtain a free trial of GCP if you need one

Install Cloud SDK

The Google Cloud SDK is used to interact with your GCP resources. Installation instructions for multiple platforms are available online.

Install kubectl CLI

The kubectl CLI is used to interteract with both Kubernetes Engine and kubernetes in general. Installation instructions for multiple platforms are available online.

Install Terraform

Terraform is used to automate the manipulation of cloud infrastructure. Its installation instructions are also available online.

Configure gcloud

Before running any commands, configure gcloud with the project you wish to use for this demo:

gcloud config set project <PROJECT_ID>

Enable the GCP Services

Please enable the GCP services by running make bootstrap. The command runs like the following:

make bootstrap

Example output:

gcloud services enable \
	  cloudresourcemanager.googleapis.com \
	  compute.googleapis.com \
	  container.googleapis.com \
	  cloudbuild.googleapis.com \
	  containerregistry.googleapis.com
Waiting for async operation operations/tmo-acf.e898de22-74cb-4f9c-9811-fd66c0f57173 to complete...
Operation finished successfully. The following command can describe the Operation details:
 gcloud services operations describe operations/tmo-acf.e898de22-74cb-4f9c-9811-fd66c0f57173
Waiting for async operation operations/tmo-acf.33b02d7e-6e5f-489f-a764-083faa63c367 to complete...
Operation finished successfully. The following command can describe the Operation details:
 gcloud services operations describe operations/tmo-acf.33b02d7e-6e5f-489f-a764-083faa63c367
Waiting for async operation operations/tmo-acf.99f6f42d-b98a-4ea2-a924-db6e219d3b62 to complete...
Operation finished successfully. The following command can describe the Operation details:
 gcloud services operations describe operations/tmo-acf.99f6f42d-b98a-4ea2-a924-db6e219d3b62
Waiting for async operation operations/tmo-acf.d46ed8d8-9c98-4e42-a931-56376d788104 to complete...
Operation finished successfully. The following command can describe the Operation details:
 gcloud services operations describe operations/tmo-acf.d46ed8d8-9c98-4e42-a931-56376d788104
Waiting for async operation operations/tmo-acf.a720d526-cec8-4b29-b38c-b197b9597de4 to complete...
Operation finished successfully. The following command can describe the Operation details:
 gcloud services operations describe operations/tmo-acf.a720d526-cec8-4b29-b38c-b197b9597de4

Run Terraform for Infrastructure Provisioning

Please run make terraform to provisions the infrastructure. The first time you run this code, make will initialize terraform in the project directory:

make terraform

Example output:

terraform fmt
terraform validate -check-variables=false
terraform init
Initializing modules...
- module.cloud
  Getting source "modules/datacenter"
- module.on-prem
  Getting source "modules/datacenter"

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "google" (1.14.0)...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Then it prompts you for a shared secret for VPN:

var.shared_secret
  Enter a value:

VPN tunnels need a shared secret so they can encrypt their communications. The shared secret is temporarily stored as a Terraform variable which is what creates the prompt. Due to Terraform handling the variable you will have to enter the same shared secret when tearing down the VPNs.

Finally it asks for confirmation to go ahead with the deployment and build out the two networks, each with a GKE cluster.

After the steps finished successfully, there will be one multi zone Kubernetes Engine cluster to simulate an on-prem data center, as well a single zone Kubernetes Engine cluster for cloud.

Configure

Run make configure, which will generate an environment specific k8s.env to be shared among the shell scripts.

make configure

Example output:

Fetching cluster endpoint and auth data.
kubeconfig entry generated for on-prem-cluster.
Fetching cluster endpoint and auth data.
kubeconfig entry generated for cloud-cluster.

Deploy Kubernetes Resources

Run make create, which invokes the scripts to create all Kubernetes objects.

You should see output similar to this:

clusterrolebinding.rbac.authorization.k8s.io/cluster-admin-binding created
service/elasticsearch-discovery created
service/elasticsearch created
deployment.apps/es-master created
Waiting for deployment "es-master" rollout to finish: 0 of 3 updated replicas are available...

Expose the pyrios Pod

We need to make the pyrios pod accessible so that we can load sample Shakespeare data and validate the Elasticsearch API via pyrios. Run make expose in a separate terminal. make expose will cause kubectl to use SSH to create a tunnel that forwards local traffic on port 9200 to the remote node running the pyrios pod on port 9200. The pyrios pod forwards the API request to the on prem Elasticsearch cluster and passes the results back.

make expose

Example output:

Forwarding from 127.0.0.1:9200 -> 9200
Handling connection for 9200

kubectl will not return on its own. The tunnel remains open until you end the process with Ctrl+C.

Load Sample Data to the Elasticsearch Cluster

Run make load loads the sample shakespeare.json into the Elasticsearch cluster via Elasticsearch Bulk API.

make load

Example output:

Loading the index into the Elasticsearch cluster
{"acknowledged":true,"shards_acknowledged":true,"index":"shakespeare"}
Loading data into the Elasticsearch cluster

Validation

Run make validate validates the Elasticsearch cluster by invoking its REST API. The scripts checks cluster version, health, sample data as well as a couple of types of queries (called query DSL in Elasticsearch term) on the sample data.

make validate

Example output:

Elasticsearch version matched
Elasticsearch cluster status is green
Shakespeare data has the expected number of shards
Shakespeare match_all query has the expected numbers of hits
Shakespeare match query on speaker LEONATO has the expected numbers of hits

Web Page User Interface

The web-based UI is equivalent to the validate.sh script. It queries Elasticsearch through the Pyrios proxy and verifies that all the data looks correct. The web-based UI also demonstrates usage of Stackdriver Tracing and custom Stackdriver metrics. You can view the UI by running make expose-ui. You will need to have port 8080 available on your machine before running make expose-ui. In your browser visit localhost:8080. Each time the UI page is refreshed it creates traces and metrics.

The custom metric is called custom/pyrios-ui/numberOfLeonatoHits and can be found in the global resource.

There are two traces:

1. pyrios-request - this is an actual Elasticsearch call through the pyrios proxy
2. pyrios-ui-request - this is the UI being loaded and making numerous calls to pyrios

Teardown

Teardown is fully automated. The teardown script deletes every resource created in the deployment script.

It will run the following commands:

1. cloud-destroy.sh - destroys the pyrios deployment
2. on-prem-destroy.sh - destroys the Elasticsearch deployments
3. terraform destroy - it prompts you for a shared secret for VPN and then destroys the all the project infrastructure

In order to teardown, run make teardown.

You should see output similar to this:

deployment.apps "pyrios" deleted
service "pyrios" deleted
deployment.apps "pyrios-ui" deleted
configmap "esconfig" deleted
networkpolicy.networking.k8s.io "pyrios-ui-to-pyrios" deleted
Switched to context "gke_jmusselwhite-sandbox_us-west1-a_on-prem-cluster".
clusterrole.rbac.authorization.k8s.io "elasticsearch-data" deleted
clusterrolebinding.rbac.authorization.k8s.io "elasticsearch-data" deleted
configmap "es-cm" deleted
deployment.apps "es-client" deleted
poddisruptionbudget.policy "elasticsearch-data" deleted
storageclass.storage.k8s.io "repd-fast" deleted
statefulset.apps "es-data" deleted
service "elasticsearch-data" deleted
service "elasticsearch-discovery" deleted

Troubleshooting

1. default credentials Error:

   * provider.google: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

   Set your credentials through any of the available methods. The quickest being:

   gcloud auth application-default login

2. make expose-ui is not working:

   Make sure that port 8080 is not being used by another process on your machine. It's a very common port for development servers, etc.

Relevant Material

• Creating an internal load balancer
• Get higher availability with Regional Persistent Disks on Google Kubernetes Engine
• pires Elasticsearch project
• ElasticSearch Cluster Sample data
• ElasticSearch bulk load API
• ElasticSearch Query DSL

This is not an officially supported Google product