cloudspannerecosystem / spanner-gaming-sample Goto Github PK

After the expiration time, if an order is not purchased, handle expiring the trade order:

• mark order as not active
• make item visible to the listing player

With the current setup, game_items table will be empty. This prevents game-workload and tradepost-workload from operating correctly.

There needs to be a way to seed the game_items table. A simple ./scripts/seed_game_items.sh could be created to run after the schema migration. But this would need to leverage Locust locally (or a dockerized version of the item workload ).

Or we just keep a small .sql file that gets imported.

Either way, the seeding step happens right after the very first schema migration, but before the service deployment.

It was reported that doing terraform destroy will destroy APIs that are enabled for the project, and thus projects with other Spanner instances will be messed up.

The TF scripts should be updated to not manage APIs. APIs should be enabled via a helper script that leverages the gcloud CLI and docs updated accordingly.

Use the official module to set up spanner instance. Do not apply DDL, as that's handled outside of Terraform.

PRs should initially run unit and integration test packages.

Releases should build binaries and create a .tar.gz with generators, binaries and readmes on 'vX.X.X' tags creation.

Update the current behavior to leverage cloud build/deploy to run the services and workloads.

• Update terraform to add pipelines
• Create a services cloud-build.yaml and skaffold.yaml
• Create a workload cloud-build.yaml and skaffold.yaml
• remove custom gcloud scripts

The output from the automated release workflow for generating release notes is:

Run git log $(git describe HEAD~ --tags --abbrev=0)..HEAD --pretty='format:* %h %s' --no-merges >> ".github/CHANGELOG.md"
fatal: Not a valid object name HEAD~

So the release notes are not automatically generated. This should be fixed

When running item workload at the same time as tradepost workload, there is a noticeable contention issue.

Investigate this and determine best way to alleviate it.

Currently the player balance update method in item-service is tied to a player_ledger. It seems this should actually be tied to a given player.

Now that workloads run on GKE, the docs should be updated to indicate how to run the Locust WebUI for the workloads.

The query to get a random open game before closing the game does a full table scan. This can be improved by adding the following index:

CREATE INDEX FinishedGames ON games(finished, created DESC);

Requires:

• adding terraform support for a GKE environment
• creating service pods

Currently, the generators/match_server.py file can only close games open during the same workload run.

This was done to prevent getting a big list of open games, which was an endpoint added specifically for the workload generator and not very performant.

Instead, we can implement a 'GET /games/open' endpoint with a filter on oldest to retrieve the oldest open game that can then be closed.

Due to the way matchmaking currently works, there is a bottleneck on scaling matchmaking and player signups:

• If we have more than 1 createGame attempt at a time, we run into deadlocks as matchmaking tries to assign the same players to different games.
• Due to locking for these deadlocks on the 'PlayerGame' index, actually inserting new players (CreatePlayer) suffers higher write latency.

To mitigate this, instead of assigning players to a game from ALL players ever, introduce a queue that simulates players 'getting ready to play'. This should reduce/remove lock contention on the PlayerGame index. We may still get deadlocks for concurrent CreateGame attempts, but it should be much smaller chance.

Design:

• modify schema to allow for queue
• modify matchmaking to assign players longest in the queue
• keep track of average wait time in queue
• game cannot be created if not enough players are in the queue.
• game should not be created if players wait < average wait time. If there's not an average wait time, set a floor to prevent a flood of game creations.

The initial frontend UI will have the following elements:

• Ability to browse items listed on trading post
• Leaderboard for players with games won vs games played

Each time a service's integration test is run, a new docker network is created. This can lead to issues with integration tests not succeeding.

To avoid this, the network should be created with the checkDuplicates: true option, as seen in the testcontainers-go docs

Leverage wrench for schema migration through cloud build. This is a separate step from application deployments, where schema migrations should be handled first and be backwards compatible.

This is a supplement to the current manual schema migration, which will be improved in #61

The idea that each service would be created in a different language is no longer a goal. Future services can be written in different languages to be useful as examples.

So the existing services should be moved to a backend_services directory to represent a full project.

Currently Github Actions runs lint, unit and integration tests for all backend services when any backend service changes. This causes long check times when PRs are opened, as well as dependency issues if one service is breaking unrelated to the one being modified.

The backend_pr.yaml workflow should be split up to only trigger the respective backend service's tests.

This depends on functionality from #12

We want to track player login behavior over time. There are two ways to do this:

• in Spanner itself, introduce a new 'ledger' table for player events such as 'login', 'logout', etc
• Use BigQuery as an example for using Change Streams. This option has added architectural complexity.

To prevent unauthorized access to service endpoints, services should authenticate clients that are calling the endpoints.

In order to get the lowest latencies from the backend services, the load generator should run as close to the services as possible. This requires running on GKE once the services are located there.

Dependencies

Requires a GKE environment #2

To allow scaling load beyond the limits of a single Locust process, the generators need to run in distributed fashion.

Dependencies

#3 should be completed before working on this.

There's a Python library, Faker, that can be used to generate fake player names. This should be done in the profile workload for new players.

Names will have to be backfilled for existing players.

If the gke cluster is not setup, the kubernetes provider will fail to work.

Providers do not have a depends_on option currently. But Data blocks do. So setting the data.google_container_cluster.gke_provider to depend on the gke cluster resource being available should solve this.

Game workloads get a random player for acquiring items and money (each call). The 'GetPlayer' endpoint is expensive and shouldn't be called very often.

Explore caching a list of players that are actively playing games for this workload.

The risk is a cached player could end their game before they're used to acquire items/money.

Backend services shouldn't be externally accessible. They should probably leverage ClusterIP rather than LoadBalancer.

To improve the security stance of these samples, use workload identity instead of service accounts.

Containerd is referred by testcontainers-go.

Update testcontainers-go to latest version and verify these issues are resolved.

Determine quality list for items and implement item quality.

Initial custom metrics that are interesting for creating a business-centric dashboard for the sample game might be:

• players playing games
• current open games
• num. items listed on tradepost

These should be posted every 10 seconds for now.

Using transaction tags can help diagnose performance issues.

https://cloud.google.com/spanner/docs/introspection/troubleshooting-with-tags#transaction_tags

Ensure each golang file has package and function comments.

Items should have categories.

Come up with generic category list, and implement.

This action should only be done by the lister.

• Mark order as not active
• make item visible to listing player again.

It seems the Spanner Emulator now supports the 'DEFAULT' syntax in schemas. So this needs to be updated, and then fix the hacks that removed those defaults from the integration tests of the services.

Issue to track various bugs when installing on a clean project.

• Terraform apply has some bugs based on missing services and incorrect ordering of resources based on dependencies.

The query to get a random open game to be closed looks something like this:

query := fmt.Sprintf("SELECT gameUUID FROM (SELECT gameUUID FROM games WHERE finished IS NULL ORDER BY created DESC LIMIT 10) TABLESAMPLE RESERVOIR (%d ROWS)", 1)

These random queries are too expensive in terms of CPU, especially as the games table grows. Additionally, this is called very frequently.

It would be better to improve the generator logic to get a list of 100-1000 open games to store in a queue. When the queue is empty, another call to get more games to close is issued to the backend.

Currently the schema/players.sql and schema/trading.sql files are copied into the backend service's folder to run integration tests.

It would be nice to use wrench as part of integration tests to validate schema migrations are working.

There are some issues with this though:

• applying migrations against the emulator will eventually take a while. This can be solved by loading the 'current' schema into a file. Question would be where to get that 'current' schema?
• The emulator can lag on feature parity with Cloud Spanner. This means that integration tests can fail if the emulator doesn't support a particular SQL feature we're taking advantage of in this sample gaming app.

So will need to figure out how to handle those two points when improving integration tests.

Use golangci-lint for PR workflow

This is to prevent users from having to manually type in locust commands.

The script should ideally allow running in headless mode or in webui by forking the process.

Currently the services and workloads use a shared workload identity: k8s-service-account. Each should have their own identity.

Add endpoints for players to login and logout

New endpoints for authentication server and players service:

• PUT /players/login: takes username and password as arguments. Returns playerUUID on success
• PUT /players/logout: takes playerUUID as argument, returns 200 success, but no content (some concerns about authenticating access to this endpoint, but really no endpoint should be able to be hit without proper credentials)

This will update a 'logged_in' timestamp field in the players table, which will be set to null on logout.

Track login events in a separate table player_login_entries

Use Pylint

Specifically to highlight GKE architecture