giltayar / bilt

A build tool for NPM monorepos

Bilt

A build tool for NPM monorepos

What is Bilt

Bilt is a CLI that builds, tests, and publishes NPM packages in your monorepos, and does that in the proper order, according to the packages dependency graph, while guaranteeing that only those packages that were not built (and their dependents) get built.

Why Bilt

(To go directly to the meat of Bilt, read Structure of a Bilt monorepo and then go to the Getting Started chapter.)

Monorepos are a wonderful concept! They enable us to structure our code as a large set of loosely-coupled packages, instead of one big monolithic codebase, which is usually tightly-coupled.

But the current build tools we have for monorepos are lacking. They are either difficult to work with (Bazel), not powerful enough for big monorepos (Lerna), or can work only with specific codebases (Nx). For more, see the Alternatives section.

Bilt is designed for small and large repos, and is simple to use, assuming your monorepo is built as a series of NPM packages linked together by an NPM dependency graph.

Table of Contents

1. Monolithic codebases vs monorepos
2. Alternatives to Bilt
3. Bilt concepts
4. Structure of a Bilt monorepo
5. Getting started
6. How Bilt works
7. Reference
8. Build configurations

Contributing

If you want to help contribute, to built, go here to learn about the Bilt code, how to build it, and how to contribute.

bilt's Issues

Support biltrc configuration in a `.bilt` directory

When loading, Bilt looks for its configuration file (.biltrc) using cosmiconfig, which is library that goes up the directory tree and searches for configuration files named .biltrc, .biltrc.json, .biltrc.js, etc.

This is great, and allows much flexibility, but it also means that the configuration file MUST be in the root of the monorepo, which isn't very nice. I propose having Bilt look there, but also look for the same files in a .bilt directory of the monorepo.

This is pretty easy to implement using Cosmiconfig's "search places", which allows specific places to search the configuration files.

Host our site at bilt.js.org

Just follow the directions here: https://js.org/ and in 24 hours we will have our "site" at bilt.js.org!

When first showing which packages will be built, show them in the order they will be built

In command-build.js in the package cli, the first thing we should the user is the list of packages we are going to build. That list is now not ordered in the build order, and it sucks. But note that we already have code that finds the list of packages in their correct order (showPackagesForDryRun). Merge the two so that it will be the same code.

If `--message` is not specified, prompt for it when it is needed

So that --dry-run (and others) can be executed without --message.

The idea is this: in command-build.js, if message is undefined, then wait for the execution of the during phase, and just before it, ask for the message (using the inquirer package from NPM) and update the options.

We're doing it before the "during" phase and not the after phase, because some jobs need the option during the during phase (for example, Roundforest's deploy job.

npm script 'develop-in-branch' not working on windows

When trying to run develop-in-branch the spawn fails with ENOENT.
based on this answer from stack overflow I changed the spawn command and it works on windows

When checking which packages changed, ignore commits with "[skip ci]"

There is a convention in CI where if you write "[skip ci]" in the commit, then the CI shouldn't run because it's a commit that doesn't really need a build. Unfortunately, that won't work with Bilt, because the next build will look at this commit and say that some packages need change.

So Bilt should ignore commits with "[skip ci]" in their commit. Obviously, this string needs to be configurable via .biltrc.

Implement jest-like output

I would recommend using https://github.com/cenk1cenk2/listr2. Looks cool

Add new command `biltn <package-name>` that returns full name of package

TIf there is a package named @bilt/npm-next-version, then biltn next-ver will return @bilt/npm-next-version.

If not unique name, error.

This is useful in situations where you want to do things like npm install `(biltn next-ver)`

filterByBilditIgnore is badly implemented

It searches and reads for bilditignore files for every file. It should cache and not read them if it can.

Usage URL broken

Your Usage URL is broken, the one in the README file.

Gracefully manage conflicts when pulling

When doing a git pull (in the end) and there is a merge conflict in a package, gracefully manage that by somehow not pushing the change for that package, but pushing the change for the other packages.

Need to think about it, because I'm not sure it's even possible.

`bilt graph` to show the dependency graph of a monorepo

Nice feature that generates a PNG or SVG (or HTML) of the dependnecy graph of the monorepo.

Something like bilt graph for a dependency graph of all packages or bilt graph . for a subgraph?

Python support

Hi @giltayar,
Is it feasible to add python support? If so I'd be happy to take a shot at it. Would be great to get some basic pointers.
Thanks

Adding the "build in branch" functionality directly into bilt

The "dev" directory has scripts that can install a local registry that aids in local development and in branches. We can incorporate this into Bilt (probably using a separate CLI?)

Something like this:

bilt-local start @bilt

...to start the local npm registry and have the @bilt scope publish to it. And this:

bilt-local list

...to list of all scopes running under a local npm registry. And finally, this:

bilt-local stop @bilt

...to remove the local registry and have the @bilt scope point to the previous registry.

Enable shortcuts to package names

So instead of bilt @bilt/npm-next-version, one could do bilt next-ver.

It is an error if name is not unique.

Howto: figuring out which packages there are in the monorepo happens in extractPackageInfos in comand-build.js in the cli package. That function uses convertUserPackagesToPackages to convert package names to package directories. This is where you should change the code so that if it finds a package name, it searche the list of packages to find it using includes and not ===.

When checking whether a package was built, take branch into consideration

To check that a package is built, Bilt looks for commits with [bilt-with-bilt] in their message. This works nicely with trunk-based development, but not so much in the OSS workflow of PR branches, where the PR branches also run bilt, but that bilt doesn't create real artifacts.

The solution? Add the branch name to [bilt-with-bilt], i.e. [bilt-with-bilt-main] and when looking for this, use the current branch name. That way, the main branch can ignore commits merged from the PR branches

Add option to build --from

A --from (as opposed to --upto) could be useful for scenarios where you change and underlying package and want to just build all it's dependents.

Create a CLI that generates a nice dependency graph

Use https://github.com/dagrejs/dagre/wiki, which also uses graphlib: the same librray we use to manipulate our dependency graph

The idea is this:

bilt graph . --upto ... will generate some json output, which can be used by show-graph which is a new CLI package that generates an HTML page (PNG? SVG?) that shows the dependency graph

Improve look of site

Because it looks horrible these days

Initial set of packages should not be built if they are already built

When doing bilt <some-packages...>, and some of the packages are already built, don't rebuild them

Add `web-snake` to sample-bilt-monorepo

The repository https://github.com/giltayar/sample-bilt-monorepo is a sample monorepo that shows how a Bilt repo works.

It implements the snake game (a very simple one) in the console. The idea is to do the same, but a web version that is also dependent on the snake package

Enable pre* and post* phases for `before`, `during`, and `after`

To enable postduring that commits and pushes, but only if --envelope and --after. I.e. a pre* and post* will operate only if the two phases it is between are on.

Add duration to progress report when building

Whenever outputting the end of a phase, output how long it took for that phase to get done, as part of the output of the end of that phase. This should all happen in command-build.js.

Make a dependency depend on semver

So that if version A depends on version B@^1.0.0 and B is version 2.0.0, then there should be no dependency between the packages. This should be a change in the function createDependencyGraph in the package packages-to-build, so that it will not take all the dependencies of a package, but only those that are semver-compatible with the version of the package.

So, as an example, if I have package A depending on package B. Let's say package B has version 2.1.5 and package A depends on it thus:

  {
    "dependencies": {
      "B": "^2.0.7"
    }
  }

Then there should be a dependency from A to B, because ^2.0.7 is semver-compatible with B's version 2.1.5. But if A depends on B thus:

  {
    "dependencies": {
      "B": "^1.0.3"
    }
  }

Then there should be a dependency from A to B, because ^1.0.3 is NOT semver-compatible with B's version 2.1.5.

To figure out whether a version is semver-compatible with a semver range, you can (and should) use the semver package in npm: https://www.npmjs.com/package/semver

Build a way to utilize the parallelization of jobs in Github actions to run built in parallel

The idea is this:

• A pre-commit gets the dependency graph and builds a Github Actions workflow file
• The first job is a special job that figures out what packages need to be built (using bilt libraries) and creates outputs per-package using ::set-output
• There will be one job per-package
• Each job will use jobs.job.needs to define the dependency it has on other jobs/packages
• Each job will use jobs.job.if to check the output of the special job to see if it needs to run

And that should work!

Needs product definition

Use npm/yarn workspaces to enable working locally on a subset of the packages

The idea is to create ad-hoc workspaces for the subset of the packages you are working on, and thus obviate the need to publish packages while working, and defer the "real" bilt process till the end (or to CI).

SOmething like a new command bilt-workspace which looks at which packages were changed and create a workspace for just those packages (and npm install-s them). Obviously, just like the bilt CLI does., the command should also accept the packages to develop on, and not just let Bilt infer them from the changed packages.

Make the default when no package is specified be the current package

When doing a regular bilt with no directories specified, it should build the package in the current directory. To support a "build all", add support for bilt --all

Make `console-snake` nicer in the sample-bilt-monorepo

The repository https://github.com/giltayar/sample-bilt-monorepo is a sample monorepo that shows how a Bilt repo works.

It implements the snake game (a very simple one) in the console. But it's really badly implemented:

1. It refreshes badly (with lots of refreshes)
2. It doesn't end the game when the snake collides with itself
3. The snake should wrap when it gets to the borders

bilt --version shows "undefined" instead of the version

And I have no idea why... Or why it should show anything at all.

Remove the `artifact-finder` package

The artifact-finder package is a very old and unmaintained package that implements a feature that nobody uses: the * auto-find functionality that automatically finds packages in a repo. I'm not sure it's documented.

So:

1. Delete the package directory
2. Remove any use of it from the cli package
3. Remove any reference about it from the documentation

Show "# of #" in progress and header reports

When showing progress or headers like these:

**** [packages/metacritic-product-catalog-deploy] building

then show how many packages there are, and which build it is. Somethign like

**** [packages/metacritic-product-catalog-deploy] building (2 of 4)

How to:

This should happen in command-build.js in the package cli. The outputting happens in makePackageBuild which is a function that returns a function, so it's not trivial, but it could go like this:

• Pass the length of the array of packages to the makePackageBuild function so that it can output how many packages there are, thus taking care of "building (? of <number-of-packages)".
• The index of the package is more difficult, because the build of the package happens in the anonymous function returned by makePackageBuild, and the only thing passed to it is the package info. And that anonymous function is called by the build package, which we don't want to change.
  To deal with that, we'll need buildPackages to pass it the index of the current package being built by modifying its loop to also increment an index, and passing that index to the buildPackage func.