aws-samples / eks-workshop

AWS Workshop for Learning EKS

Warning This repository is in maintenance mode and currently only accepting bug-fixes. PRs for new content should be adapted for the new EKS Workshop: https://github.com/aws-samples/eks-workshop-v2

eksworkshop

Setup:

Using GitPod.io:

This is how I set up my environment: (I am using gitpod.io for editing)

1. fork the repo to your own github account
2. prepend gitpod.io# to the beginning of your github url. Mine becomes: https://gitpod.io#github.com/brentley/eks-workshop
3. once gitpod has started, in the terminal, run npm install && npm run theme This will install the dependencies and clone the theme submodule.

From here, you can use the online IDE to edit /content/chapter/filename.md... If you want to preview your edits, in the terminal, run: npx hugo server. That will start the local hugo server.

A dialog box will pop up telling you "A service is listening on port 1313, but is not exposed." -- press the expose button. After that, choose "open browser" to get a new tab with your preview site. As you save edits, this tab should refresh.

When you're happy with your edits, commit, push, and open a pull request to the upstream repo's main branch. Once merged, the preview site (linked above) will be refreshed.

On a Mac:

Install Hugo: brew install hugo

On Linux:

• Download from the releases page: https://github.com/gohugoio/hugo/releases/tag/v0.64.1
• Extract and save the executable to /usr/local/bin

Clone this repo:

From wherever you checkout repos: git clone [email protected]:aws-samples/eks-workshop.git (or your fork)

Clone the theme submodule:

cd eks-workshop

git submodule init ; git submodule update

Install Node.js and npm:

You can follow instructions from npm website: https://www.npmjs.com/get-npm

Install node packages:

npm install

Run Hugo locally:

npm run server or npm run drafts to see stubbed in draft pages.

npm run build will build your content locally and output to ./public/

npm run test will test the built content for bad links

View Hugo locally:

Visit http://localhost:8080/ to see the site.

Making Edits:

As you save edits to a page, the site will live-reload to show your changes.

Auto Deploy:

Any commits to main will auto build and deploy in a couple of minutes. You can see the currently deployed hash at the bottom of the menu panel.

Any commits to a branch will auto build and deploy in a couple of minutes to a custom route named with the branch name. You can see the currently deployed hash at the bottom of the menu panel. An example is the "jenkinsworld" branch would be deployed to https://eksworkshop.com/jenkinsworld/

note: shift-reload may be necessary in your browser to reflect the latest changes.

eks-workshop's Issues

Build a deployment pipeline - slide deck

* Push the Docker image to registry (_*_ECR_*_)
* Write k8s manifest file
* Deploy the application
* Update the business logic, push changes to git, update the app using pipeline
* Sidecar Injection
    * service mesh? Let's use a super common sidecar example

Setup logging for k8s workloads

Showcase how to...

• Setup Metrics Server
• Setup HPA

Screenshots may be helpful when creating roles

https://eksworkshop.com/integration/prerequisites/iamrole/

For each step, a screenshot should accompany. If IAM is new to a user, screenshots with what should be selected / filled out will help them.

add simple example to helm chapter

it could be as simple as helm install nginx

Deploy an application

* Take an existing application, create Docker image
* how to write a dockerfile

Add detailed instructions on adding 30900 inbound SG access

https://eksworkshop.com/integration/monitoring/deploy-prometheus/

This may not be an intuitive operation for new users, so creating detailed steps will help.

Run batch workloads

• showcase how to use GPU instances
• how to use taint, toleration, affinity/anti-affinity

Deploy a Kubernetes Dashboard

Setup a deployment pipeline, push changes and watch updated application

Use Skaffold?

Deploy basic application

helm install stable/metrics-server fails

From https://eksworkshop.com/onug/logging/

When running:

helm install stable/metrics-server --name metrics-server --version 2.0.2 --namespace metrics

I get the error:

Error: release metrics-server failed: namespaces "metrics" is forbidden: User "system:serviceaccount:kube-system:default" cannot get namespaces in the namespace "metrics"

I tried to manually create the ns first and re-ran, but no dice. Any ideas whats going on? Is this dependent on another step in the tutorial that I [apparently] skipped?

Fix bugs in logging module

• add steps to create lambda_basic_execution role. repurpose log_policy folder after renaming it to iam_policy and updating references
• fluentd.yml : mention that us-west-2 should be updated if a different region is picked and update stream_name value to avoid lognameprefix issue
• Add a banner to inform that all URLs default to us-west-2 and user should select the applicable region
• Update cleanup page to remove lambda_basic_execution role

application tracing using XRay

Migrate applications across cluster

Setup and deploy applications using Helm charts

Use the same apps deployed via helm

Update and rollback the application

Create an EKS cluster (EKS ctl)

port-forward may fail if pods have not come up yet

https://eksworkshop.com/integration/monitoring/deploy-prometheus/

After

export POD_NAME=$(kubectl get pods --namespace prometheus -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")

kubectl --namespace prometheus port-forward $POD_NAME 9090

We may want to add something like:

If you get an error similar to error: unable to forward port because pod is not running. Current status=Pending the Prometheus pods may just not had a chance to come up yet. If you get this error, wait 30s, and retry the command.

Setup health checks (using probes)

Create tabular based instructions to support 3 options - Cloud9/OSX/EC2

https://eksworkshop.com/install-tracks/tabs-example/#more-tab-installation-0

replace text editing of values yaml with sed script

https://eksworkshop.com/integration/monitoring/deploy-grafana/

To improve the user experience, replace the editing of values.yaml with a sed script or similar that performs the text substitutions from the command line.

prometheus values filename is wrong

https://eksworkshop.com/integration/monitoring/deploy-prometheus/

We DL and edit it as values.yaml, but call helm install as prometheus-values.yaml.

Setting up Workshop infrastructure

Expected Outcomes

• Pre Provisioned Accounts
• Easy Start Experience
• Easy account clean-up
• Pre-deploy the setup from previous workshop, may be using Helm chart

unclear what helm ls command is achieving

In https://eksworkshop.com/integration/monitoring/prereqs/ we should articulate what exactly we're trying to achieve/prove by executing helm ls .

Prometheus cleanup steps do not remove StorageClass

We need to add that to the cleanup.

Monitoring using Prometheus & Grafana

* Monitor traffic between services (Prometheus + Grafana)

Setup cluster to autoscale based upon metrics (Cluster and HPA)

• test scaling

Introduction to k8s and basic concepts - slide deck

replace text editing of storageClass values with command-line script

https://eksworkshop.com/integration/monitoring/deploy-prometheus/

To improve the user experience, replace the editing of values.yaml with a sed script or similar that performs the text substitutions from the command line.

note helm rbac prerequisites before installing metrics server

At https://eksworkshop.com/onug/scaling/deploy_hpa/#deploy-the-metrics-server if the RBAC for helm is not setup correctly first per https://eksworkshop.com/onug/helm/deploy/ the metrics server install will fail.

helm reset then helm init is a way to recover if you installed helm before applying these correct RBAC settings.

Upgrade cluster from 1.10 to 1.11

we should be ready to have a discussion of EKS upgrade process and AMIs

Canary deployment using Istio and Spinnaker

update slack resource slide with eks channel detail

On this slide, note the EKS channel within the k8s slack workspace.

Network Policies

* Configure authn, authz and RBAC
* Configure network policies

Logging through EFK

U/I interface for Workshop

Expected Outcomes

• Create an U/I for the workshop
• Write document that shows workflow of how to update modules

getting ELB URL could benefit from a space at the end

https://eksworkshop.com/integration/monitoring/deploy-grafana/

abc123-1422578166.us-west-2.elb.amazonaws.comfoo:/tmp $ 
foo:/tmp $ 

would be nice to have whitespace between prompt and end of URL to make easier for copy/paste

service mesh using Istio

make eks user managed / aws managed chapter titles parallel

Under EKS section in intro, change chapters:

Customer Managed
Managed Control Plane
->
Customer Managed Data Plane
Amazon Managed Control Plane

Load balancing an application (Setup ingress/DNS routing to application)

Fix bugs in monitoring chapter

Prometheus and Grafana default installation uses ClusterIP to expose its metrics and dashboards. This doesn't work well with Cloud9 when you try to run port-forwarding and try to use the ports where the services are listening.

• Need to change how we expose Prometheus metrics
• Need to change how we expose Grafana dashboards

AWS networking, security, etc... - slide deck

EKSclt having issues while creating cluster

This is the error message I got while creating a cluster with the following command
eksctl create cluster --name=eksworkshop-eksctl --nodes=3 --node-ami=auto --region=${AWS_REGION}

2018-10-23T21:46:08Z [ℹ] using region us-west-2
2018-10-23T21:46:09Z [ℹ] setting availability zones to [us-west-2a us-west-2b us-west-2c]
2018-10-23T21:46:10Z [ℹ] using "ami-0a54c984b9f908c81" for nodes
2018-10-23T21:46:10Z [ℹ] creating EKS cluster "eksworkshop-eksctl" in "us-west-2" region
2018-10-23T21:46:10Z [ℹ] will create 2 separate CloudFormation stacks for cluster itself and the initial nodegroup
2018-10-23T21:46:10Z [ℹ] if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=us-west-2 --name=eksworkshop-eksctl'
2018-10-23T21:46:10Z [ℹ] creating cluster stack "eksctl-eksworkshop-eksctl-cluster"
2018-10-23T21:47:06Z [✖] unexpected status "ROLLBACK_IN_PROGRESS" while waiting for CloudFormation stack "eksctl-eksworkshop-eksctl-cluster" to reach "CREATE_COMPLETE" status
2018-10-23T21:47:06Z [ℹ] fetching stack events in attempt to troubleshoot the root cause of the failure
2018-10-23T21:47:06Z [ℹ] AWS::CloudFormation::Stack/eksctl-eksworkshop-eksctl-cluster: ROLLBACK_IN_PROGRESS – "The following resource(s) failed to create: [SubnetUSWEST2A, SubnetUSWEST2B, PublicSubnetRoute, PolicyNLB, SubnetUSWEST2C]. . Rollback requested by user."
2018-10-23T21:47:06Z [✖] AWS::EC2::Subnet/SubnetUSWEST2B: CREATE_FAILED – "Resource creation cancelled"
2018-10-23T21:47:06Z [✖] AWS::EC2::Subnet/SubnetUSWEST2C: CREATE_FAILED – "Resource creation cancelled"
2018-10-23T21:47:06Z [✖] AWS::IAM::Policy/PolicyNLB: CREATE_FAILED – "Resource creation cancelled"
2018-10-23T21:47:06Z [✖] AWS::EC2::Subnet/SubnetUSWEST2A: CREATE_FAILED – "Resource creation cancelled"
2018-10-23T21:47:06Z [✖] AWS::EC2::Route/PublicSubnetRoute: CREATE_FAILED – "route table rtb-0400aa3431520b725 and network gateway igw-0c49354aae0fc78cf belong to different networks (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterValue; Request ID: 2c781c6e-aad3-4510-959b-df0ac89ea59a)"
2018-10-23T21:47:06Z [ℹ] AWS::EC2::Route/PublicSubnetRoute: CREATE_IN_PROGRESS
2018-10-23T21:47:06Z [ℹ] AWS::EC2::SecurityGroup/ControlPlaneSecurityGroup: CREATE_COMPLETE
2018-10-23T21:47:06Z [ℹ] AWS::EC2::SecurityGroup/ControlPlaneSecurityGroup: CREATE_IN_PROGRESS – "Resource creation Initiated"
2018-10-23T21:47:06Z [ℹ] AWS::EC2::RouteTable/RouteTable: CREATE_COMPLETE
2018-10-23T21:47:06Z [ℹ] AWS::IAM::Policy/PolicyNLB: CREATE_IN_PROGRESS – "Resource creation Initiated"
2018-10-23T21:47:06Z [ℹ] AWS::EC2::Subnet/SubnetUSWEST2A: CREATE_IN_PROGRESS – "Resource creation Initiated"
2018-10-23T21:47:06Z [ℹ] AWS::EC2::Subnet/SubnetUSWEST2C: CREATE_IN_PROGRESS – "Resource creation Initiated"
2018-10-23T21:47:06Z [ℹ] AWS::EC2::RouteTable/RouteTable: CREATE_IN_PROGRESS – "Resource creation Initiated"
2018-10-23T21:47:06Z [ℹ] AWS::EC2::Subnet/SubnetUSWEST2B: CREATE_IN_PROGRESS – "Resource creation Initiated"
2018-10-23T21:47:06Z [ℹ] AWS::IAM::Policy/PolicyNLB: CREATE_IN_PROGRESS
2018-10-23T21:47:06Z [ℹ] AWS::EC2::Subnet/SubnetUSWEST2C: CREATE_IN_PROGRESS
2018-10-23T21:47:06Z [ℹ] AWS::EC2::Subnet/SubnetUSWEST2A: CREATE_IN_PROGRESS
2018-10-23T21:47:06Z [ℹ] AWS::EC2::RouteTable/RouteTable: CREATE_IN_PROGRESS
2018-10-23T21:47:06Z [ℹ] AWS::EC2::Subnet/SubnetUSWEST2B: CREATE_IN_PROGRESS
2018-10-23T21:47:06Z [ℹ] AWS::EC2::SecurityGroup/ControlPlaneSecurityGroup: CREATE_IN_PROGRESS
2018-10-23T21:47:06Z [ℹ] AWS::IAM::Role/ServiceRole: CREATE_COMPLETE
2018-10-23T21:47:06Z [ℹ] AWS::EC2::VPC/VPC: CREATE_COMPLETE
2018-10-23T21:47:06Z [ℹ] AWS::EC2::InternetGateway/InternetGateway: CREATE_COMPLETE
2018-10-23T21:47:06Z [ℹ] AWS::IAM::Role/ServiceRole: CREATE_IN_PROGRESS – "Resource creation Initiated"
2018-10-23T21:47:06Z [ℹ] AWS::EC2::InternetGateway/InternetGateway: CREATE_IN_PROGRESS – "Resource creation Initiated"
2018-10-23T21:47:06Z [ℹ] AWS::IAM::Role/ServiceRole: CREATE_IN_PROGRESS
2018-10-23T21:47:06Z [ℹ] AWS::EC2::VPC/VPC: CREATE_IN_PROGRESS – "Resource creation Initiated"
2018-10-23T21:47:06Z [ℹ] AWS::EC2::InternetGateway/InternetGateway: CREATE_IN_PROGRESS
2018-10-23T21:47:06Z [ℹ] AWS::EC2::VPC/VPC: CREATE_IN_PROGRESS
2018-10-23T21:47:06Z [ℹ] AWS::CloudFormation::Stack/eksctl-eksworkshop-eksctl-cluster: CREATE_IN_PROGRESS – "User Initiated"
2018-10-23T21:47:06Z [ℹ] 1 error(s) occurred and cluster hasn't been created properly, you may wish to check CloudFormation console
2018-10-23T21:47:06Z [ℹ] to cleanup resources, run 'eksctl delete cluster --region=us-west-2 --name=eksworkshop-eksctl'
2018-10-23T21:47:06Z [✖] waiting for CloudFormation stack "eksctl-eksworkshop-eksctl-cluster" to reach "CREATE_COMPLETE" status: ResourceNotReady: failed waiting for successful resource state
2018-10-23T21:47:06Z [✖] failed to create cluster "eksworkshop-eksctl"

The issue went away when I deleted the stack and ran the eksctl create cluster command once again

Fix bugs in Autoscaling chapter

Found some bugs and typos in Autoscaling chapter. I sent a PR that is already merged into Integration. This is ready to merge into master

missing period

missing period after:

https://eksworkshop.com/integration/monitoring/prereqs/

'if you don't have it installed'

Add configuration data/secrets to the application

Setup Pod-based roles

Deploy solution for pod based roles. will use kube2iam unless there is other feedback

few errors with eksworkshop setup (main ReadMe)

$ git submodule init git submodule update
error: pathspec 'git' did not match any file(s) known to git.
error: pathspec 'submodule' did not match any file(s) known to git.
error: pathspec 'update' did not match any file(s) known to git.

$ npm install
-bash: npm: command not found
Sorry I have brand new laptop : )
Need to include npm install steps (https://www.npmjs.com/get-npm).

$ npm run server
(truncated for abbreviation)......
ERROR 2018/09/26 11:22:31 Unable to locate template for shortcode "notice" in page "cleanup/eksctl.md"
ERROR 2018/09/26 11:22:31 Unable to locate template for shortcode "children" in page "deploy/_index.md"
ERROR 2018/09/26 11:22:31 Unable to locate template for shortcode "notice" in page "dashboard/dashboard.md"
ERROR 2018/09/26 11:22:31 Unable to locate template for shortcode "notice" in page "deploy/viewservices.md"
ERROR 2018/09/26 11:22:31 Unable to locate template for shortcode "notice" in page "eksctl/launcheks.md"
ERROR 2018/09/26 11:22:31 Unable to locate template for shortcode "children" in page "prerequisites/_index.md"
ERROR 2018/09/26 11:22:31 Unable to locate template for shortcode "notice" in page "prerequisites/account.md"
ERROR 2018/09/26 11:22:31 Unable to locate template for shortcode "notice" in page "prerequisites/k8stools.md"
ERROR 2018/09/26 11:22:31 Unable to locate template for shortcode "notice" in page "prerequisites/sshkey.md"
ERROR 2018/09/26 11:22:31 Unable to locate template for shortcode "notice" in page "prerequisites/workspace.md"
ERROR 2018/09/26 11:22:31 Unable to locate template for shortcode "notice" in page "prerequisites/workspaceiam.md"
(truncated for abbreviation)......
Error: Error building site: logged 12 error(s)

clarify that we're going to save a file and then open it and edit it.

https://eksworkshop.com/integration/monitoring/deploy-prometheus/

Change:
curl -o prometheus-values.yaml https://raw.githubusercontent.com/helm/charts/master/stable/prometheus/values.yaml

Search for # storageClass: “-”, uncomment and change the value to “prometheus”. You will do this twice, under both server & alertmanager manifests

to something like:

To install Prometheus, first we'll download the Prometheus configuration using curl, then edit the storageClass attribute.

To download the configuration as values.yaml, execute the following command:

curl -o prometheus-values.yaml https://raw.githubusercontent.com/helm/charts/master/stable/prometheus/values.yaml

Next, using a text editor, open up values.yaml, search for # storageClass: “-”, uncomment and change the value to “prometheus”. You will do this twice, under both server & alertmanager manifests

Need to change directory first

The code block on https://eksworkshop.com/codepipeline/role/ should change directory to ~/environment before running the commands. The previous commands causes you to be in ~/environment/fluentd (which is subsequently deleted if you did the clean up logs section - but Linux still has your cwd set as ~/environment/fluentd)