kazu0716 / spanner-stress-test-demo

Spanner Load Test App

This repository contains sample game API code, load test sample code and arroud middleware for distributed load testing:

• Game API server
• Locust load test code
• Setup make file via terraform and gcloud

Requirements

Highly recommend: you to use it on Cloud Shell, because of easing to prepare settings for this demo

Cloud Environment

We expected to set organization policy default

• Google Cloud Project
• Google Account
  • It needs roles/owner permission on your project

Of course, you can deploy user account, but we recommend you to use Service Account.

• Service Account
  • It needs roles/owner permission on your project
• Service Account Key file

# Example to create account and keyfile
$ export SA_NAME=terraformer
$ export PROJECT_ID=test
# Create Service Account
$ gcloud iam service-accounts create $SA_NAME \
    --description="Operation service account for spanner stress demo" \
    --display-name=$SA_NAME
$ gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" \
    --role="roles/owner"
# Download a key file
$ export KEY_FILE=terraformer.json
$ gcloud iam service-accounts keys create $KEY_FILE \
    --iam-account=${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com
# We recommend key file type is JSON

Deploy Tools

Cloud Shell has already installed followings:

• gcloud
• terraform (auther version: Terraform v1.2.9 on linux_amd64)
• make command
• envsubst

Development Tools

If you need to develop this repository, you should install followings:

• python (auther version: 3.9.2)
  • pip (auther version: 22.2)
  • pipenv (auther version: 2022.7.24)
• docker (auther version: 20.10.17, API version: 1.41)
• docker-compose (auther version: version 1.24.0)
• tbls (Optional)
• terraform-docs (Optional)
• act (Optional)

How to use this demo

Setup env settings

You should set own environment by followings:

$ git clone https://github.com/kazu0716/spanner-stress-test-demo.git
$ cd spanner-stress-test-demo

# You should change .env_sample to .env for gcloud environment
$ cp .env_sample .env
$ vim .env
# You should edit .env file

Spesifications of .env file

Key name	Description	Example value
GOOGLE_CLOUD_PROJECT	Google Cloud Project ID	test
ACCOUNTS	Account list for access to bastion	["serviceAccount:[email protected]", "serviceAccount:[email protected]"]
KEY_PATH	Credential key file path	~/workspace/terraformer.json
REGION	Google Cloud Region you hope	asia-northeast1
SERVICE_NAME	Cloud Run's service name	sample-game
INSTANCE_NAME	Cloud Spanner's instance name	spanner-demo
DATABASE_NAME	Cloud Spanner's database name for testing	sample-game
ENV	Env id, but we expected to use "production" when you deploy on Google Cloud.	production
LOG_LEVEL	Loglevel of app and Locust	INFO
POD_NUM	The number of Pod for load test containers	10
USERS	Peak number of concurrent Locust users	10
RUN_TIME	Stop after the specified amount of time	1h

Deploy on Google Cloud

# Create environment via terraform and gcloud
$ make create.cloud.environment

# Run load testing
$ make deploy.locust

Watch the result

You can check the result in real time by followings:

• Google Cloud's operations suite

  • Cloud Monitoring
  • Cloud Logging
  • Cloud Trace

• Locust plugin dashboard

How to connect to grafana dashboard

# SSH port-forwording to grafana
$ make open.grafana
Grafana dashboard: https://3443-xxxxyyyzzzz

# Cloud Shell envronment(You can access via proxy)
$ open https://3443-xxxxyyyzzzz

# Other environment
$ open http://localhost:3443

Change Cloud Spanner cluster size

Note: If your spanner mark high CPU usage, you should change Cloud Spanner size manually

• How to change conpute capacity of Cloud Spanner

Delete all components after tests

$ make delete.cloud.environment

Features

Asis

• Rest API for dummy game
  • opperate Cloud Spanner(database schema)
• API docs of Swagger by FastAPI
• Distributed Stressor by Locust
• IaC by terraform and gcloud

Tobe

• Auto Scalling for Cloud Spanner, instead of manual scalling

Reference

Internal modules reference

• Sample game app documents
• Test senario documents by locust
• Terraform documents

3rd party reference

• FastAPI
• Locust
• Locust Plugins
• OpenTelemetry FastAPI Instrunentation

Contribution

Note: Understand how to check your PR before you send PR

• CONTRIBUTING docs

License

This is Apache 2.0 License

Note

This is not an officially supported Google product