This repository provides some Terraform scripts to force a heavy load on a TFE environment by creating 9 workspaces and running a big terraform simultaneously on these workspaces.
By utilizing so much load on your TFE environment you simulate multiple scenario's described here
Make sure you have a TFE running. Use the following repository for example
Running this load on a TFE environment with some incorrect settings on your TFE environment you should see the system swapping in the memory performance metrics.
- On your TFE environment make sure you change the swapping levels to 95
sysctl vm.swappiness=95
- On your TFE environment make sure that worker memory is 256MB
# see the current value
replicatedctl app-config export --template "{{.capacity_memory.Value}}"
# change it to 256
replicatedctl app-config set capacity_memory --value 256
# restart the application to apply the config changes
replicatedctl app apply-config
- Clone the repository to a machine that you want the load to run
git clone https://github.com/munnep/tfe_workload.git
- Go to the directory tfe_workload/terraform_workspaces
cd tfe_workload/terraform_workspaces
- add a file called
variables.auto.tfvars
with the following content
organization = "test"
tfe_token = "your_token"
tfe_hostname = "patrick-tfe6.bg.hashicorp-success.com"
- initialize terraform
terraform init
- apply the changes to create the workspaces
terraform apply
- login to your terraform cloud environment
terraform login <tfe-url>
- change the below settings to match your environment in the file base/main.tf hostname organization
terraform {
backend "remote" {
hostname = "patrick-tfe6.bg.hashicorp-success.com"
organization = "test"
workspaces {
prefix = "test-"
}
}
}
- execute the script that will make the directories and execute the terraform code
bash run.sh
- Within TFE you should see all workspaces have jobs running now
- In your performance charts of your TFE instance you should see that swap will be getting used now.
Example screenshot of Netdata that was installed on the TFE server
- destroy the workspaces by going into the directory tfe_workload/terraform_workspaces
terraform destroy
Running this load on a TFE environment with some incorrect settings on your TFE environment you should get an oom-kill on the terraform run.
- On your TFE environment turn swapping off
sudo swapoff -a
- On your TFE environment make sure that worker memory is 128MB
# see the current value
replicatedctl app-config export --template "{{.capacity_memory.Value}}"
# change it to 128
replicatedctl app-config set capacity_memory --value 128
# restart the application to apply the config changes
replicatedctl app apply-config
- Clone the repository to a machine that you want the load to run
git clone https://github.com/munnep/tfe_workload.git
- Go to the directory tfe_workload/terraform_workspaces
cd tfe_workload/terraform_workspaces
- add a file called
variables.auto.tfvars
with the following content
organization = "test"
tfe_token = "your_token"
tfe_hostname = "patrick-tfe6.bg.hashicorp-success.com"
- initialize terraform
terraform init
- apply the changes to create the workspaces
terraform apply
- login to your terraform cloud environment
terraform login <tfe-url>
- change the below settings to match your environment in the file base/main.tf hostname organization
terraform {
backend "remote" {
hostname = "patrick-tfe6.bg.hashicorp-success.com"
organization = "test"
workspaces {
prefix = "test-"
}
}
}
- execute the script that will make the directories and execute the terraform code
bash run.sh
- Within TFE you should see all workspaces have jobs running now
- you should see the tasks getting killed at some point like the following
- Download a support bundle from your dashboard
- If you look in the bundle and search for oom-kill you will see you will get a find like the following
[17612.605083] br-c70e2168ae73: port 2(vethe99ed86) entered forwarding state
[17744.519687] terraform invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[17744.519696] CPU: 2 PID: 323609 Comm: terraform Not tainted 5.13.0-1022-aws #24~20.04.1-Ubuntu
- destroy the workspaces by going into the directory tfe_workload/terraform_workspaces
terraform destroy