andrewh1978 / px-deploy

What

This will deploy one or more clusters in the cloud, with optional post-install tasks defined by template.

Supported platforms

Container

• Kubernetes
• K3s
• Docker EE
• Openshift 4 (only on AWS at this time)
• EKS (only makes sense on AWS)
• AKS (only makes sense on Azure)
• GKE (only makes sense on GCP)

Cloud

• AWS
• Azure
• GCP
• vSphere

Install & Update

1. Install and enable a container runtime such as Docker. On MacOS with Apple Silicon, it is recommended you use Docker Desktop.

2. Install the Git CLI.

3. Install bash-completion (optional), eg:

$ brew install bash-completion

You will need to restart your shell.

4. Run the install / update script:

curl https://raw.githubusercontent.com/andrewh1978/px-deploy/master/install.sh | bash

This will:

• pull the Docker image from Github
• create and populate $HOME/.px-deploy
• provide instructions to configure .bash_profile/.zshrc

It may take 5-10 minutes to complete, longer with a slow Internet connection.

Update your .bash_profile or .zshrc as directed. Source them or login again. Validate it is complete with px-deploy -h.

Review the various cloud settings in ~/.px-deploy/defaults.yml.

5. Configure Cloud specific settings

Notes for AWS

Notes for Azure

Notes for GCP

Notes for vSphere

Notes for OCP4 + AWS

Notes for Updating

• Run the same steps to update your px-deploy installation

• Please don't change the default scripts / templates as they will be overwritten during update. Feel free to create your own scripts / templates.

• Your defaults.yml will no longer be replaced. After updating please check if the new release supports newer Kubernetes / Portworx versions and update it manually.

• You'll find this information in $HOME/.px-deploy/defaults.yml.VERSION

• When updating from version < 5.0.0, please destroy all AWS / AWSTF Cloud based deployments in advance. Version 5 cannot destroy these deployments any longer.

• Update your .bashrc or .zshrc with the new px-deploy alias.

Use

This will provision a VPC and some other objects, and deploy into it from the template:

px-deploy create --name=my-deployment --template=clusterpair

Connect via SSH:

px-deploy connect --name my-deployment

Execute a command:

px-deploy connect --name my-deployment "storkctl get clusterpair"

Tear down the deployment:

px-deploy destroy --name my-deployment

NOTES

The deployments can be listed:

$ px-deploy list
DEPLOYMENT  CLOUD  REGION         PLATFORM  TEMPLATE  CLUSTERS  NODES  CREATED
foo         aws    eu-west-1      k8s       px               1      3  2020-02-11T16:14:06Z
bar         gcp    europe-north1  gcp       <none>           1      3  2020-02-04T09:50:11Z

The templates can be listed:

$ px-deploy templates
NAME                         DESCRIPTION
async-dr                     Deploys 2 clusters with Portworx, sets up and configures a cluster pairing,
                             configures an async DR schedule with a loadbalancer in front of the setup.
backup-restore               Deploys a Kubernetes cluster, Minio S3 storage, Petclinic Application and
                             Backup/Restore config
elk                          Deploys an ELK stack of 3 master nodes, 3 data nodes and 2 coordinator nodes, as per
                             https://docs.portworx.com/portworx-install-with-kubernetes/application-install-with-kubernetes/elastic-search-and-kibana/
harbor                       Deploys a single K8s cluster with Portworx and Harbor (https://goharbor.io/)
metro                        Deploys 2 K8s clusters in AWS with a stretched Portworx cluster. It configures
                             Metro, a GUI and Petclinic, ready for a manual failover demo
migration                    Deploys 2 clusters with Portworx, sets up and configures a cluster pairing, and
                             deploys a set of apps and a migration template.
px-backup                    A single Kubernetes cluster with Portworx and PX-Backup via Helm installed
px-fio-example               An example fio benchmark on a gp2 disk and a Portworx volume on a gp2 disk
px-vs-storageos-fio-example  An example fio benchmark on a gp2 disk, a Portworx volume on a gp2 disk, and a
                             StorageOS volume on a gp2 disk
px                           A single Kubernetes cluster with Portworx installed
storageos                    A single Kubernetes cluster with StorageOS installed
training                     Deploys training clusters

Generate kubeconfig files so you can run kubectl from your laptop:

$ px-deploy kubeconfig --name exampleDeployment
$ kubectl get nodes --kubeconfig $HOME/.px-deploy/kubeconfig/exampleDeployment.1
NAME                                           STATUS   ROLES    AGE   VERSION
ip-192-168-5-19.eu-west-1.compute.internal     Ready    <none>   11m   v1.21.5-eks-9017834
ip-192-168-56-14.eu-west-1.compute.internal    Ready    <none>   11m   v1.21.5-eks-9017834
ip-192-168-74-141.eu-west-1.compute.internal   Ready    <none>   11m   v1.21.5-eks-9017834

Note this is currently only tested with EKS.

The defaults.yml file sets a number of deployment variables:

• aws_ebs - a list of EBS volumes to be attached to each worker node. This is a space-separated list of type:size pairs, for example: "gp2:30 standard:20" will provision a gp2 volume of 30 GB and a standard volume of 20GB
• aws_region - AWS region
• aws_type - the AWS machine type for each node
• aws_access_key_id - your AWS API access key
• aws_secret_access_key - your AWS API secret access key
• eks_version - EKS k8s Version
• cloud - the cloud on which to deploy (aws, gcp, or vsphere)
• ssh_pub_key - ssh public key which will be added for root access on each node
• clusters - the number of clusters to deploy
• k8s_version - the version of Kubernetes to deploy
• tags - a list of tags to be applied to each node. This is a comma-separate list of name=value pairs, for example: "Owner=Bob,Purpose=Demo"
• stop_after - stop the intances after this many hours
• post_script - script to run on each master after deployment, output will go to stdout
• nodes - the number of worker nodes on each cluster
• lock - prevents deployment from being deleted by px-deploy destroy .... use px-deploy unlock -n ... to unlock
• platform - can be set to either k8s, k3s, none, dockeree, ocp4, eks, gke or nomad
• azure_type - the Azure machine type for each node
• azure_disks - similar to aws_ebs, for example: "Standard_LRS:49 Premium_LRS:50"
• azure_client_id - Azure Client ID
• azure_client_secret - Azure Access Secret
• azure_subscription_id - Azure Subscription ID
• azure_tenant_id - Azure Tenant ID
• aks_version - AKS k8s Version
• px_version - the version of Portworx to install
• gcp_project - GCP project
• gcp_disks - similar to aws_ebs, for example: "pd-standard:20 pd-ssd:30"
• gcp_region - GCP region
• gcp_type - the GCP machine type for each node
• gcp_zone - GCP zone
• gke_version - GKE k8s Version
• vsphere_host - endpoint
• vsphere_compute_resource - compute resource
• vsphere_user - user with which to provision VMs
• vsphere_password - password
• vsphere_template - full path to CentOS 7 template
• vsphere_datacenter - vcenter datacenter
• vsphere_datastore - datastore prefix
• vsphere_folder - folder for vSphere VMs
• vsphere_disks - similar to aws_ebs, for example: "20 30" (NOTE: these are not provisioned as static block devices, but they are used as clouddrives)
• vsphere_network - vSwitch or dvPortGroup for cluster ex: Team-SE-120
• vsphere_memory - RAM in GB
• vsphere_cpu - number of vCPUs
• vsphere_repo - repository to pull vsphere template
• ocp4_domain - domain that has been delegated to route53
• ocp4_version - eg 4.3.0
• ocp4_pull_secret - the pull secret '{"auths" ... }'

There are two ways to override these variables. The first is to specify a template with the --template=... parameter. For example:

$ cat templates/px-fio-example.yml
description: An example fio benchmark on a gp2 disk and a Portworx volume on a gp2 disk
scripts: ["install-px", "px-wait", "px-fio-example"]
clusters: 1
nodes: 3
cloud: aws
aws_ebs: "gp2:150 gp2:150"
post_script: cat
auto_destroy: true
env:
  px_suffix: "s=/dev/nvme1n1"
  cat: "/tmp/output"

More on scripts below.

The second way to override the defaults is to specify on the command line. See px-deploy create -h for a full list. For example, to deploy petclinic into the foo deployment:

px-deploy create --name=foo --clusters=5 --template=petclinic --nodes=6

This example is a mixture of both methods. The template is applied, then the command line parameters are applied, so not only is the template overriding the defaults, but also the parameters are overriding the template.

scripts is a list of scripts to be executed on each master node. For example:

$ cat ~/.px-deploy/scripts/petclinic
# Install petclinic on each cluster
kubectl apply -f /assets/petclinic.yml

These variables are passed to the script:

• $nodes
• $clusters
• $px_version
• $k8s_version

You can also select a different defaults file with:

$ DEFAULTS=/path/to/other/defaults.yml px-deploy create ...

All files in ~/.px-deploy/assets will be copied to /assets on the master nodes. They are then available to be used by the script, as above.

In addition to these, there are some more variables available:

• $cluster - cluster number

There is also an optional cluster object. At the moment, all it can be used for is defining cluster-specific scripts. These will be executed after the scripts above. For example:

cluster:
- id: 1
  scripts: ["script-1", "script-2"]
- id: 2
  scripts: ["script-3", "script-4"]

post_script is a script that will be run on each master node after all of the scripts have completed, and the output will go to stdout. The default is to display the external IP address of master-1, but it could be used to show benchmark outputs, for example.

Last, environment variables can be define in templates or defaults.yml, and these are also available to scripts:

$ cat templates/metro.yml
...
env:
  licenses: "XXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XXXX XXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XXXX"

Enviroment variables can also be defined on the command line:

px-deploy create -n foo -t migration -e install_apps=true,foo=bar

The install-px script looks for an environment variable called cloud_drive. If it exists, it will deploy Portworx using a clouddrive rather than looking for all attached devices. Note that this is a requirement for Openshift 4. For example:

px-deploy create -n foo -t px --cloud aws -e cloud_drive=type%3Dgp2%2Csize%3D150
px-deploy create -n foo -t px --cloud aws --platform eks -e cloud_drive=type%3Dgp2%2Csize%3D150
px-deploy create -n bar -t px --cloud aws --platform ocp4 -e cloud_drive=type%3Dgp2%2Csize%3D150
px-deploy create -n baz -t px --cloud gcp --platform gke -e cloud_drive="type%3Dpd-standard%2Csize%3D150"
px-deploy create -n foo -t px --cloud azure --platform aks -e cloud_drive="type%3DStandard_LRS%2Csize%3D150"

Notes for AWS

If you are using AWS and you have not accepted the Rocky Linux 8 terms, browse to https://aws.amazon.com/marketplace/pp/prodview-2otariyxb3mqu

Edit your defaults.yml to set aws_access_key_id and aws_secret_access_key.

Notes for Azure

1. Get access to Azure cli (either you install or use Cloud Shell on Azure UI)

2. Edit your defaults.yml to set azure_tenant_id, azure_subscription_id, azure_client_id, azure_client_secret.

$ az account list

value of id is your azure_subscription_id value

value of tenant is your azure_tenant_id value

3. create application secret

$ az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/[your azure_subscription_id]"

value of appId is your azure_client_id

value of password is your azure_client_secret

4. accept RockyLinux EULA

$ az vm image terms accept --urn "erockyenterprisesoftwarefoundationinc1653071250513:rockylinux:free:8.6.0" 

Notes for GCP

tl;dr

• setup a project

• create service account with roles compute.admin,container.admin,iam.serviceAccountUser

• download json key file for service account and move it as gcp.json into your ~/.px-deploy folder

• set gcp_project in ~/.px-deploy/defaults.yml

Detailed:

1. install CLI and ensure it is configured: https://cloud.google.com/sdk/docs/quickstarts

2. you can copy & paste the following commands. just ensure to replace all values shown in [brackets] with your specific values - without the [brackets]

3. set env variable for project / service account (example here px-deploy-proj and px-deploy-sa)

PX_DEPLOY_PROJECT="[px-deploy-proj]"

SERVICE_ACCOUNT="[px-deploy-sa]"

4. create a new project and set it as default

gcloud projects create ${PX_DEPLOY_PROJECT}

gcloud config set project ${PX_DEPLOY_PROJECT}

5. get your billing account id

gcloud beta billing accounts list

Output will look like this:

ACCOUNT_ID            NAME                OPEN  MASTER_ACCOUNT_ID

654321-ABCDEF-123456  My Billing Account  True

6. link your project to billing account id (here 654321-ABCDEF-123456)

gcloud beta billing projects link ${PX_DEPLOY_PROJECT} --billing-account [654321-ABCDEF-123456]

7. create service account

gcloud iam service-accounts create ${SERVICE_ACCOUNT}

8. assign roles to service-account

gcloud projects add-iam-policy-binding ${PX_DEPLOY_PROJECT} \
   --member="serviceAccount:${SERVICE_ACCOUNT}@${PX_DEPLOY_PROJECT}.iam.gserviceaccount.com" \
   --role="roles/compute.admin"

gcloud projects add-iam-policy-binding ${PX_DEPLOY_PROJECT} \
   --member="serviceAccount:${SERVICE_ACCOUNT}@${PX_DEPLOY_PROJECT}.iam.gserviceaccount.com" \
   --role="roles/container.admin"

gcloud projects add-iam-policy-binding ${PX_DEPLOY_PROJECT} \
   --member="serviceAccount:${SERVICE_ACCOUNT}@${PX_DEPLOY_PROJECT}.iam.gserviceaccount.com" \
   --role="roles/iam.serviceAccountUser"  

9. download service account json key file (here gcp.json)

gcloud iam service-accounts keys create gcp.json \
    --iam-account=${SERVICE_ACCOUNT}@${PX_DEPLOY_PROJECT}.iam.gserviceaccount.com

10. move gcp.json into your ~/.px-deploy/ folder

mv gcp.json ~/.px-deploy/

11. enable compute & kubernetes API

gcloud services enable container.googleapis.com

gcloud services enable compute.googleapis.com

12. edit your ~/.px-deploy/defaults.yml to add

gcp_project: "[px-deploy-proj]"

Notes for vSphere

Before you can start deploying in vSphere, a local VM template must be installed. The command

$ px-deploy vsphere-init

will read the vsphere variables from defaults.yml (please check in advance if they are correct) and provision a template at the path defined in vsphere_template.

Known Issues

Known Issues on EKS

If you see on AWS UI an error message like this: Your current IAM principal doesn’t have access to Kubernetes objects on this cluster. the IAM user being used to create the deployment isn't the same IAM user running the AWS UI.

One reason could be that access to AWS UI is authenticated using SSO.

To fix the isse find out the user/role being used by AWS UI (e.g. by running aws sts get-caller-identity on AWS UI CloudShell) and set one of the following ENV variables in ~/.px-deploy/defaults.yml (in my example its the OktaSSOAdminUser Role / developerUser1 user)

env:
    AWS_ADD_EKS_IAM_ROLE: "OktaSSOAdminUser"
    AWS_ADD_EKS_IAM_USER: "developerUser1"

More information can be found here https://docs.aws.amazon.com/eks/latest/userguide/view-kubernetes-resources.html#view-kubernetes-resources-permissions

px-deploy's Issues

Allow vSphere Network to be Variabliezed.

Currently its hard coded to use "VM Network"

Do not delete deployment if destroy fails

AutoPilot - Cockroach Compression

investigate disabled cockroachdb compression to allow for more linear growth

Improve template descriptions

Test

AWS: Support existing VPC

right now we always deploy a new VPC. It would be good to point px-deploy at an existing VPC.

Deployment name isn't added as a tag to worker nodes

cockroachdb template for ap does not include prometheus install

assign to me and i can update.

Parameterise master node type

Master node type is the same size as worker nodes.

Run the metro demo external etcd on master-2

Clean up broken deployments

Check if a deployment that px-deploy thinks exists still exists; if not, clean up the associated files

fix installer to support versions

Hello

Tidy up assets and use directories

backup-restore template is forced to use AWS and K8s Version

Will fix to work with any k8s and any cloud.

Add Progress messages during VM Provisioning

VM Provisioning takes a while, it would be nice to see progress messages, or steps being executed during the process.

Clusterpair error

Having the DisasterRecovery option in the clusterpair template causes an error on the latest Portworx version. This needs to be removed.

Hello2

AutoPilot - Optimise volume fill time

Ideal time for volume to hit 50% capacity is 4 mins, which allows AP enough time to react before the pvc hits 100%. Current timing is around 6 mins to 50%.

Add an error for missing licenses

In the case of deploying a Metro environment without setting the env var for licenses a warning should be given. If the license remains as XXXX-XXXX etc this should be caught before the deployment runs.

vSphere provisioning customizations

The current implentation of PX-Deploy doesn't allow for the following settings to be changes in vSphere:

1. Network to place VMs - deploys VMs on same network as template
2. VM Folder Placement - Deploys VMs in same folder as template
3. Requires a template already exist in the VMware environment that meets PX-Deploy requirements - Centos7-with-CloudInit

Instance types not found in different AWS region

When deploying in a different region, the default instance type is missing causing errors. Can a check be added to make sure the instance type is available with a more graceful failure if it is missing. Alternatively we could add a default instance for each region.

Add directory structure / prefixes for template type

For example:

demo/px-central
comp/storageos

Support EKS

Joe and Andrew - WIP tasks

• add kubectl patch to metro
• move storkctl install to px-install (and any other scripts that install px, eg cp4d)
• move aws stuff out of async-dr
• move aws stuff out of metro
• separate licence script
• dont apply async dr to remote clusters

Add the autopilot demo

Support spaces in paths for VM folder

https://github.com/andrewh1978/px-deploy/blob/vsphere-rp/vagrant/all-common#L63

such as My Awesome Users/Myself

Support GKE

Add Docker EE platform

Port to AWS/GCP SDKs

px-deploy uses the AWS and GCP CLI to manage infrastructure; we should move to the SDKs

Add Oracle Cloud support

Numerous customers are interested in Oracle on Portworx. It would be good to be able to deploy in Oracle Cloud to allow for multi-cloud demos and labs.

It's currently used by the MOD, various other UK public sector and Edu oppos.

AutoPilot - Sort Event List

Sort AutoPilot events by date

Add local as a cloud option for VirtualBox

Add an option to provision an environment locally on a laptop / home server using VirtualBox

Test2

Add a reaper or deployment timeout setting to reduce costs

Add the PXC CLI extension for kubectl

Retry failed node provisions

Somes a node fails to provision for transient reasons; we should retry, automatically or otherwise

Add NFS server to master-1

In order to demo proxy volumes we need an NFS server running inside a cluster. I believe it should be included as standard in all Portworx deployments so it should probably be added to the k8s-master provisioning script.

PWX CSI: support install of PX with CSI

right now I think we dont do this by default which is fine, but i dont think there is a knob?

Unify Central and Backup templates

We currently have three ways to deploy central and backup - backup.yml, px-central.yml, px-backup.yml. We believe there should be two templates.

There is a use case for standalone backup and central with backup (which you may not necessarily demo). Our plan is the following:

1. Wait for the release of Central 1.0.4 which should include helm install with an option to include backup
2. Update the backup.yml template and scripts to use this new method, replicating the existing environment with latest versions of the software.
3. Deprecate current px-central.yaml (this only deploys central) and replace with the refactored backup.yml which deploys both central and backup already. Call this px-central.yml
4. Automate creation of demo clusters and the inclusion of them in the relevant dashboard in both px-backup.yml and the new px-central.yml
5. Automate adding an S3 backup location in both templates (environment var already exists and can be set in defaults.yaml).

Support OCPv4

No error if SSH isn't ready

If you run a new deployment then quickly try to SSH to the master using pxctl connect the command exists without any error output.

px-deploy create stuck in Provisioning VMs.. better handling possible?

I was setting up async-dr template and since I forgot to update my license in the ~/.px-deploy/defaults.yaml and not sure if there's some improvement in handling, as the create process got stuck (no timeout?) - when checking the first master's /var/log/px-deploy/ directory, i see licenses file was continually filling with:

Unable to use a TTY - input is not a terminal or the right kind of file
ERROR: Could not activate license: Error processing license: Please reduce number of nodes from 3 to 0, or add a new license
command terminated with exit code 1

I am wondering if there could be some better handling of detecting this is failing? There were no errors passed through to the user, and so after half an hour of no progress I started to wonder if something was wrong.. Asked in the #px-deploy slack channel and got a quick answer, but this could probably be handled better from a tool standpoint?

Clean up failed deployments

If a deployment fails, clean up the artifacts and delete associated files

AutoPilot - add storage pool expand

extend demo to include storage pool expand

Add central demo

AutoPilot - integrate GUI

Integrate the GUI so it auto loads the cockroachdb UI

issues creating async-metro-backup-demo.yml

templates/async-metro-backup-demo.yml does not seem to be usable?

It does not seem to install PX. https://github.com/andrewh1978/px-deploy/blob/master/templates/async-metro-backup-demo.yml#L3

I wonder if this is just misssing install-px and licenses scripts

[root@master-2 ~]# cat /var/log/px-deploy/clusterpair-dr 
No resources found in kube-system namespace.
waiting for portworx
No resources found in kube-system namespace.
waiting for portworx

Add k3s platform

test