thundergolfer / example-bazel-monorepo

🌿💚 Example Bazel-ified monorepo, supporting Golang, Java, Python, Scala, and Typescript

👋 Hi there, thanks for stopping by

I'm here a lot, building data-intensive products from the ground up. I'm bullish on, and a big booster of:

• modal.com! Truly cloud-native application development. The fastest way to run code in the cloud.
• 🌿💚 @bazelbuild and generally using functional principles to keep things sane. (I've open-sourced a few tools for the Bazel ecosystem, see below)
• Good old-fashioned books 📖. Two recent favourites are The Plot Against America and The Netanyahus.
• Effective Altruism. Helping, not harming. Doing, not having. Building, not consuming. (Mosquito nets, not the SBF/FTX-style embarassment)

Active Projects

• Writing about software. I semi-regularly publish a post at thundergolfer.com/blog. A few are...
  • Computing Cultural Literacy
  • From Data Engineer to SysAdmin: Put down the K8s cluster, your pipelines can run without it
  • The first 99,999 writes are free: Or, why lazy StringIO.write() may sprint into a memmove wall
  • What's in a name? The Data Scientist vs. Machine Learning Engineer title bore.
• ddiaflashcards.com - Building a study deck for a much loved software textbook. A small digital product, available for 🥜 $$.

Past Projects

• Growing Python in the Bazel ecosystem. See various projects in my pins. I previously did maintenance duty for bazelbuild/rules_python. Unfortunately, I no longer have time to dedicate to the ecosystem.
• • whitelist.sh - A goal-oriented, calendar aware content feed that I would have wanted to use during my undergrad. 👷🏻‍♂️

---

Computing is pop culture. [...] Pop culture holds a disdain for history. Pop culture is all about identity and feeling like you're participating, It has nothing to do with cooperation, the past or the future—it's living in the present. I think the same is true of most people who write code for money. They have no idea where [their culture came from]. - Alan Kay

example-bazel-monorepo's Issues

Consider incorporating parts of https://github.com/bazelbuild/examples

Buildkite Pipeline - Add other relevant pipeline steps

Description

Running bazel test //... is great and all, but there are other CI steps that any respectable Bazel monorepo should run. Right now I can think of:

1. BUILD file and other Bazel file linting. Important in context of project
2. Code linting (this could be costly if this project involves 4-5 different languages)
3. Some sort of WORKSPACE file validation

Failure after clone

Description of the problem / feature request:

I cloned the repositoy locally. Then executed bazel build //.... However, the build fails.

🐛 Bug Report: Expected Behavior

Successful build.

🐛 Bug Report: Actual Behavior

INFO: Invocation ID: 7dbc255e-7eeb-4176-a0c1-d5aff91a5ad1
INFO: Repository bazel_gazelle_go_repository_config instantiated at:
  /Users/brettw/Documents/dev/example-bazel-monorepo/WORKSPACE.bazel:40:21: in <toplevel>
  /private/var/tmp/_bazel_brettw/8b843efc2057a180abb205fca8381752/external/bazel_gazelle/deps.bzl:73:25: in gazelle_dependencies
Repository rule go_repository_config defined at:
  /private/var/tmp/_bazel_brettw/8b843efc2057a180abb205fca8381752/external/bazel_gazelle/internal/go_repository_config.bzl:57:39: in <toplevel>
ERROR: An error occurred during the fetch of repository 'bazel_gazelle_go_repository_config':
   Traceback (most recent call last):
	File "/private/var/tmp/_bazel_brettw/8b843efc2057a180abb205fca8381752/external/bazel_gazelle/internal/go_repository_config.bzl", line 25, column 31, in _go_repository_config_impl
		config_path = ctx.path(ctx.attr.config)
Error in path: Not a regular file: /Users/brettw/Documents/dev/example-bazel-monorepo/WORKSPACE
ERROR: Error fetching repository: Traceback (most recent call last):
	File "/private/var/tmp/_bazel_brettw/8b843efc2057a180abb205fca8381752/external/bazel_gazelle/internal/go_repository_config.bzl", line 25, column 31, in _go_repository_config_impl
		config_path = ctx.path(ctx.attr.config)
Error in path: Not a regular file: /Users/brettw/Documents/dev/example-bazel-monorepo/WORKSPACE
ERROR: error loading package '': Encountered error while reading extension file 'buildifier/def.bzl': no such package '@com_github_bazelbuild_buildtools//buildifier': no such package '@bazel_gazelle_go_repository_config//': Not a regular file: /Users/brettw/Documents/dev/example-bazel-monorepo/WORKSPACE
INFO: Elapsed time: 0.128s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (1 packages loaded)
    currently loading: cli ... (6 packages)
    Fetching @py_deps; Restarting.
    Fetching @com_github_bazelbuild_buildtools; Restarting.
    Fetching @graknlabs_bazel_distribution; fetching

🐛 Bug Report: Steps to Reproduce the Problem

1. Clone the repo
2. Execute bazel build //...

🐛 Bug Report: Specifications

• Platform: MacOS 11.6.6 (Intel)

Add tagging system to Bookstore

Refs:

• https://charlesleifer.com/blog/a-tour-of-tagging-schemas-many-to-many-bitmaps-and-more/
• https://softwareengineering.stackexchange.com/questions/113292/database-design-for-tagging-system
• http://howto.philippkeller.com/2005/04/24/Tags-Database-schemas/

What is the point of bootstrap_agent.sh

Description of the problem / feature request:

It looks like bootstrap_agent.sh does just about nothing. What is the point of this script, and why do you have it in a s3 bucket?

🐛 Bug Report: Expected Behavior

🐛 Bug Report: Actual Behavior

🐛 Bug Report: Steps to Reproduce the Problem

🐛 Bug Report: Specifications

• Version:
• Platform:

CI broken

Auto-Configuration Error: Cannot find gcc or CC; either correct your path or set the CC environment variable 😖

Update Bazel to latest as at Mar 2020 (2.2.0)

Description

Requires rebuilding the AMI, and doing brew upgrade bazelbuild/tap/bazel locally.

buildkite pre-command returns Error: No command has been provided

Description of the problem / feature request:

I setup a buildkite agent using the steps for aws in the builtkite UI. Now when the project builds I get this error. Not sure if this is a bug or I did something wrong...

🐛 Bug Report: Expected Behavior

🐛 Bug Report: Actual Behavior

🐛 Bug Report: Steps to Reproduce the Problem

1. Setup a buildkite agent using the steps for aws in the builtkite UI
2. use the ami created by prepare.sh

🐛 Bug Report: Specifications

• Version:
• Platform:

Properly document how Buildkite + Bazel was setup

Quick sketch of what I remember doing:

• Create a new pipeline with a single step that just runs bazel test //...
• Go the Buildkite (BK) agents page and select the AWS Quickstart guide. Click the button that auto-creates a CloudFormation stack in your account.
• After that CloudFormation (CF) stack has spun up, it's likely that you'll be able to see some agents appear in your agents list in the BK UI, but they won't be able to run the step yet.
• Follow the instructions here on how to create an SSH keypair that BK and Github will share so that your private code can be pulled
• Create a bootstrap script like the boostrap_agent.sh script found in this repo that is able to install Bazel and its requirements (e.g Java SDK) on the BK-default 'Amazon Linux2' AMIs.
• Upload that script into a new S3 bucket next to the S3 buckets buildkite created (like the one for private SSH keys).
• Go into IAM in the AWS console and copy the policy that grants the buildkite-agent role access to the SSH keys bucket and modify it so that it grants that role access to the bucket that contains your bootstrap script.
• If this is working correctly, then when you re-run BK pipeline tasks you should see Bazel correctly starting.
• Go into the CloudFormation stack and click the "Update" button to update some parameters:
• Update MaxInstances to 1 (it's cheaper)
• Update MinInstances to 0 (it's cheaper)
• Set BuildkiteTerminateInstanceAfterJobDecreaseDesiredCapacity and
  BuildkiteTerminateInstanceAfterJobtotrue`. This will scale down the ASG instances when BK jobs are finished, which is cheaper
• Reduce the EBS volume size to 100GB, which should be plenty when starting out
• Up the instance from t2-nano to at least t2-small, because otherwise you'll get out of memory errors.

Add a README section linking to info about why monorepos are good

Spin up a Bazel remote cache

Description

We want:

• one for CI, which runs Linux
• one for local, where I'm running MacOSX. Should look at https://docs.bazel.build/versions/master/remote-caching.html#disk-cache

Buildkite Pipeline - AWS infra related improvements

My Buildkite setup is now at a stage where I can run bazel test //... and have things pass 😄, but things are far from perfect.

Improvements to be made:

1. Currently cannot access like buildkite-agent instances. The Buildkite CloudFormation template has something about ssh keyPair that can supposedly address this
2. EC2 Instance setup currently happens in the bootstrap_agent.sh script. I think it would be much better to bake this setup into an AMI and then use that as the agent AMI instead of the base buildkite-stack AMIs. Having 1 from this list will help achieve this.
3. Lower Priority: I had to go up from t2-nano to t2-small to avoid a JVM out-of-memory error. See if I can do some tuning or find a better instance type for these BK agents.

Upgrade to Bazel 1.0.0

Add hermetic toolchain for Python

Description of the problem / feature request:

This is long overdue.

Why BUILD.bazel over BUILD?

https://github.com/thundergolfer/the-one-true-bazel-monorepo/blob/7a11ce0e890efc997b384414430a94c2384072f0/fruit_sorting/BUILD.bazel

I think using BUILD is the best as it is consistent with the official doc. I understand if it may have a syntax highlighting issue depending on what you use but that should be the syntax highlighting plug-in issue that it should recognize BUILD as a Bazel file.

Node and Ruby currently aren't working in C.I

#20 has links to the NodeJS related problems.

The Ruby problems were around some symlink already existing. Was able to reproduce if I spun up an instance and cloned down the repo onto it.

Setup Flake8 for Python code

Description

black (which this project is already using) is only a code formatter and doesn't catch a bunch of issues that crop up in Python code (eg. unused vars).

I'd like to continue using https://github.com/thundergolfer/bazel-linting-system but it might require modifications to enable support for more than 1 'linter' to be registered for 1 language.

lint.sh script is now really slow

I think it has something to do with the addition of Rust into the repo.

Incorporate rustfmt into project linting

Description

Thanks to bazelbuild/rules_rust@fe50d3b#diff-d5825ae12e45363c451d288f385bfda1L147 rustfmt is now easily available in the WORKSPACE. See:

bazel run @rust_darwin_x86_64//:rustfmt -- --help

Will have to use a select to handle the fact that the external repo will be different on CI (Linux).

CI can stuck in an infinite loop

Description

https://buildkite.com/thundergolfer-inc/example-bazel-monorepo/builds/268#eaf8ebeb-fd64-4cec-92be-db2557d79e11

Add Issue + PR Templates

React app isn't working?

Description of the problem / feature request:

I may be missing something but when I run the frontend code it looks like only the html is rendered. The "React" code doesn't seem to be loading...

🐛 Bug Report: Expected Behavior

🐛 Bug Report: Actual Behavior

🐛 Bug Report: Steps to Reproduce the Problem

🐛 Bug Report: Specifications

• Version:
• Platform:

disable build buddy by default

Description of the problem / feature request:

please disable build buddy by default [ and remove https://app.buildbuddy.io/invocation/9a58fab0-e62e-4034-875d-b382e5f9a42b ]

Learn from GoogleCloudPlatform/gke-bazel-demo

https://github.com/GoogleCloudPlatform/gke-bazel-demo

'python_register_toolchains' not found

Description of the problem / feature request:

I have this in my WORKSPACE:

######################
# PYTHON SUPPORT
######################
rules_python_version = "0.7.0"

http_archive(
    name = "rules_python",
    sha256 = "15f84594af9da06750ceb878abbf129241421e3abbd6e36893041188db67f2fb",
    strip_prefix = "rules_python-0.7.0",
    url = "https://github.com/bazelbuild/rules_python/archive/refs/tags/0.7.0.tar.gz",
)

load("@rules_python//python:repositories.bzl", "python_register_toolchains")

python_register_toolchains(
    name = "python39",
    # Available versions are listed in @rules_python//python:versions.bzl.
    python_version = "3.9",
)

load("@python39_resolved_interpreter//:defs.bzl", python_interpreter = "interpreter")
load("@rules_python//python:pip.bzl", "pip_install")

🐛 Bug Report: Actual Behavior

When I run a build (of any target) I get this error:

ERROR: Traceback (most recent call last):
	File "/Users/brettw/Documents/dev/jnb-ping/WORKSPACE", line 94, column 49, in <toplevel>
		load("@rules_python//python:repositories.bzl", "python_register_toolchains")
Error: file '@rules_python//python:repositories.bzl' does not contain symbol 'python_register_toolchains'
ERROR: error loading package '': Encountered error while reading extension file 'defs.bzl': no such package '@python39_resolved_interpreter//': error loading package 'external': Could not load //external package
INFO: Elapsed time: 0.128s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)

🐛 Bug Report: Specifications

• Platform: MacOS 11.6.6

My bazel version output:

Bazelisk version: development
Build label: 5.2.0
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Jun 7 16:11:03 2022 (1654618263)
Build timestamp: 1654618263
Build timestamp as int: 1654618263