bottlerocket-os / twoliter Goto Github PK

Description

Create a subcommand in Twoliter named make which passes through arguments to cargo make.
This will be used to execute Makefile.toml targets during the Bottlerocket monorepo transition to Twoliter.

For example, if a user wants to run cargo make -e FOO=bar variant, then the following command will run (as an alias in the Bottlerocket monorepo):

twoliter make -e FOO=bar variant

The arguments can be taken exactly as the come and be passed directly to cargo make.

Definition of Done

When twoliter make spins up its bootstraping container, and executes cargo make inside it with the same arguments it received.

It is not important for this issue that proper directories be mounted, and the cargo make target will certainly fail. This is fine.

Depends on:

• bottlerocket-os/bottlerocket-sdk#108

id: texec_command
epic: twoliter_exec

Related to bottlerocket-os/bottlerocket#2669

Description

There may be cases where it is convenient not to specify which Bottlerocket SDK is preferred.
Additionally there may be overlapping cases where this cannot be inferred from dependence on the bottlerocket-core-kit.

Furthermore, there may be cases where Twoliter needs to use the SDK before it knows what SDK the relevant kits were built with. For example, when it is doing dependency resolution of the specified kit dependencies, it will not yet have

We need to figure out:

• How the user should specify the desired SDK when the required one cannot be determined
• How twoliter should do dependency resolution, which needs to occur before the correct SDK is known
• How twoliter should know which target arch SDK to pull if it is doing something where target arch doesn't matter

In general the goal is to avoid pulling the wrong SDK just so that we can find out what the right SDK is.

Definition of Done

When

• the design document is refined to explain how we can avoid pulling the wrong SDK before we know what SDK we need
• the described mechanism has been implemented

Depends on:

• #10

id: texec_default_sdk
epic: twoliter_exec

Related to bottlerocket-os/bottlerocket#2669

Description

bottlerocket-os/bottlerocket#3372 (comment)

Running cargo make -e PACKAGE=libfoo build-package results in running cargo build --manifest-path variants/Cargo.toml --package libfoo. This reproduces the error a little faster.

The key to understanding the error lies in the new Cargo feature resolver v2 separating activated features by both package and where/when it is going to be used. When checking for activated features of a package, Cargo takes both into account. It bails when it unexpectedly cannot find a combination of {package, where to use it}.

The following is my current understanding of the matter, but it is to be taken with a grain proper amount of salt:
We are using dependencies and build-dependencies in the Cargo manifests for specifying runtime and buildtime dependencies both for the target architecture which is a bit different from what Cargo uses these for. Specifying a dependency in the build-dependencies section of a manifest, I believe, activates features in this dependency package as HostDep. When:

a variant is built, the roots of all (RPM) package trees to pull into the final image are likewise defined as build-dependencies (e.g. in variants/metal-dev/Cargo.toml). When resolving the feature set for dependencies upon building these top-level packages (already known as HostDeps), Cargo looks for and finds their (and their transitive) dependencies also as HostDep.
when just building a single package it is looking for activated features in a NormalOrDev context for the single target to be built (e.g. libfoo)--while still knowing libfoo and all of its dependencies as HostDep only.
I'm not sure what would a good way to resolve this would look like. Perhaps reverting to the feature resolver v1 and accepting warnings is the lesser evil for now, so we can avoid the problem by having the unified feature namespace.

Definition of Done

When Bottlerocket and any other Twoliter project can use build package with workspace resolvers set to v2.

This tracking issue represents an internal security review of the Twoliter app before we declare it generally available.

In the design document, #3, the section entitled Twoliter Update needs refinement.

This may require prototyping and implementation to figure out.
Let's not forget to update the design doc when we have a better handle on this.

The section in its original for read as follows:

Twoliter Update

TODO - This section is a bit hand-wavy and will have its design further refined.

twoliter update is another inspiration from cargo, this time inspired by cargo update.
This command makes it easy for maintainers to update the packages (kits actually) they are using to the latest versions.

In the original Bottlerocket build system, we used Cargo crates to model RPM dependencies.
In Twoliter, we will again lean on Cargo to manage dependencies for us.
We will create a local Cargo registry to represent kit and SDK versions, then use cargo update to resolve the latest dependencies.

Before describing this procedure, we should consider that no two versions of the same kit can be in the dependency tree.
Since they are Yum repositories, this would mess things up.
Cargo allows multiple major versions of a package to exist in a build, but this is behavior we do not want.
We will manipulate version numbers to so that Cargo gives us the desired behavior.

First Twoliter will read the kit dependencies from each variant and add any external kits to its graph.
Next Twoliter will read the kit definitions in the kits directory, and add all of their external dependencies to its graph.

Once it has this list, Twoliter will begin a traversal.
For each kit it will pull all the -metadata images for each external kit JSON files locally.
Each time it finds a new external kit (i.e. an external kit that is depended on by an external kit), it will add this new dependency to the graph.

Once it has all the dependency and version information, Twoliter will build a local cargo registry.
Each kit will be represented by an empty Cargo lib.rs project with the same name as the kit.
The Cargo package version will be given by deterministically adding the kit's version to 1.x (TODO - hand-waved).

The SDK will be a special Cargo package.
Two SDKs may have the same version, but come from different container registries.
To ensure a single SDK is used, it will be given a version in Cargo like this (TODO - or some hash of the necessary unique information):

[dependencies]
sdk = "=0.50.0-public.ecr.aws_bottlerocket_bottlerocket-sdk-x86_64"

At this point Twoliter will build a package that represents the maintainers top level dependencies.

[dependencies]
bottlerocket-core-kit = "1.1.15" # TODO - best way to normalize this into 1.x
bottlerocket-aws-kit = "1.1.15"  # in this example the patch version from 1.15.1 has been lost

Twoliter will run cargo check --locked to make sure the current state of the project is good.
If that succeeds then cargo update will change the Cargo.toml file to the highest versions of all kits possible.
The results of this will be parsed and propagated back to the project.
(TODO - how)

Definition of Done

A PR updating the design doc should answer the following questions and eliminate TODOs in the doc that relate to kit and SDK dependency resolution and updates:

• Where will the maintainer specify external kit dependencies?
• How will twoliter discover the complete list of external depenencies?
• How will local kit dependencies be defined?
• How will a maintainer lock a certain kit dependency to a certain version?
• How will Twoliter record the results of its dependency resolution?
• Does Twoliter need to communicate the resolved depenencies to buildsys?

Depends on

• #1

Description

Twoliter will need to "bootstrap" an environment from which it will call cargo make on the Bottlerocket Makefile.toml.
Twoliter will build this container on-the-fly using bottlerocket-sdk as its base.

In this initial implementation, it is not important which SDK version is used or what Twoliter adds to it.
For this issue we just need Twoliter to execute a docker build command and successfully create an SDK-based container image.

Definition of Done

When the twoliter build variant command first creates an SDK-based container.

Depends on:

• #6

id: texec_container
epic: twoliter_exec

Related to bottlerocket-os/bottlerocket#2669

What I'd like:
I want cargo make validate-repo to return different exit codes for the different errors it encounters.

In order to build proper automation around repository validation, I want to be able to differentiate actual service problems from local system environment problems.

Tracking issue for subissues related to writing documentation for Twoliter.

The design doc in #3 is not very specific about how kit containers (and their RPMs) will be signed. Refine this design and update the design doc.

Originally it said this:

Kit Signing

In order to satisfy self-signing requirements, Kit RPMs will need to be GPG-signed.
The kit will be distributed with the GPG public key, which means we need the container to also be signed.
Existing container-signing mechanisms will be made available.

Use this as a tracking issue for the milestone of eliminating conditional compilation in Bottlerocket's build of both Rust code and packages. Even though the sub-issues will probably be in Bottlerocket, we are tracking those issues here because it is a part of the larger Twoliter out-of-tree-builds-project.

This tracking issue represents work that we need to do internally in order to publish Bottlerocket kits publicly.

Description

Make a command, twoliter build kit my-awsome-kit, which will build the desired kit from the maintainer's Twoliter project packages and kits directories.

Definition of Done

When the twoliter build kit command creates a docker image containing rpm packages and yum repo metadata.

Depends on:

• #16

id: buildkit
epic: kits

Related to bottlerocket-os/bottlerocket#2669

Description

We need to use the SDK as a base and build a twoliter "bootstrapping" container on the fly.
Update the design document so that it comports with this plan.

We also intend to move the Bottlerocket monorepo over to a twoliter wrapper as soon as possible.
Describe how this will be done using a twoliter exec command in place of cargo make in the monorepo.

Definition of Done

Done when a PR is merged that described twoliter exec and how this is the first milestone of the project.

id: texec_design
epic: twoliter_exec

Related to bottlerocket-os/bottlerocket#2669

Description

Implement the algorithm by which Twoliter finds Twoliter.toml.
Implement a simple schema which includes:

• The twoliter schema version (required)
• The SDK to use (it is optional for the user to specify this)
• Optional: the version of the Twoliter binary to lock to (it is optional for the user to specify this)

Definition of Done

Done when Twoliter commands demand that Twoliter.toml exist, can find it, and deserialize it.

Depends on:

• #6

id: texec_basic_projtoml
epic: twoliter_exec

Related to bottlerocket-os/bottlerocket#2669

Description

Currently we default to a hard-coded x86_64. It might be preferable to default to the host machine architecture.

Originally posted by @cbgbt in #23 (comment)

Definition of Done

When Twoliter no longer assumes we are building x86_64.

This does what it is supposed when a user, i.e. the Bottlerocket project, calls it. But it used to check rpm2img and other scripts in the tools dir, but now it doesn't because it's not being called in Twoliter's CI. Add these checks to Twoliter's CI.

What I'd like:
bottlerocket-os/bottlerocket#2486 covers the modify-image flow where modifying a pre-built image is faster or more advantageous than building from source. There are really two distinct pieces here, the modification of images vs downloading built images from the TUF repo and placing them in the working directory for modification. This issue covers importing those images.

Ideally this would be a make target such as cargo make import-images that takes a variant and version and downloads from a configured TUF repo. There is likely work to also have the kmod-kit and migrations also downloaded but I think we could scope that to yet a different issue if needed since modifying the images is really the only thing strictly needed.

I wrote a bash version of this target but ideally, this should go in pubsys to keep all the logic around where these images should be in one place.

Any alternatives you've considered:
Building from source, which sort of defeats the point of bottlerocket-os/bottlerocket#2486

Description

When building the Twoliter bootstrapping container, copy the necessary build tools into place.
In __ we defined a container using the SDK as its base.
In this issue we will add:

• buildsys, pubsys, etc.
• rpm2img and all other sdk-environment scripts.
• Bottlerocket's Makefile.toml

The above items should have their git history preserved.

Definition of Done

When Twoliter builds its container on the fly and copies in the items listed above.

Depends on:

• #8
• #10

id: texec_tools_cpy
epic: twoliter_exec

Related to bottlerocket-os/bottlerocket#2669

Description

Having update-metadata duplicated in Bottlerocket and Twoliter is highly problematic because this is fundamental to how Bottlerocket operates. We probably need to publish this on crates.io.

sources/updater/update_metadata

Duplicating this makes me more than a little sad but I suppose in the short term it's unlikely to change given how difficult it us to evolve the format. It probably ought to be an independent crate, like bottlerocket-os/bottlerocket-update-metadata.

Originally posted by @bcressey in #37 (comment)

Definition of Done

When update-metadata is no longer duplicated in both Bottlerocket and Twoliter.

Description

          `SchemaVersion` could be in a different module. Currently it is `project.rs`

Originally posted by @bcressey in #23 (comment)

Definition of Done

When SchemaVersion has been moved to a module that makes more sense than the project.rs module.

I'm trying to install twoliter from source with a small change to buildsys. I noticed that installing v0.0.2 from source works, but installing from v0.0.3 from source fails:

[fedora@ip-10-0-162-41 bottlerocket]$ cargo install --quiet --force --git file:///home/fedora/src/twoliter --locked --tag v0.0.2 --bin twoliter twoliter 
[fedora@ip-10-0-162-41 bottlerocket]$ echo $?
0
[fedora@ip-10-0-162-41 bottlerocket]$ cargo install --quiet --force --git file:///home/fedora/src/twoliter --locked --tag v0.0.3 --bin twoliter twoliter 
error: could not find `twoliter` in file:///home/fedora/src/twoliter?tag=v0.0.3 with version `*`

A tracking issue for making Twoliter feature-complete enough to declare it generally available.

Tracking issue for issue related to making it possible to publish Bottlerocket's packages as part of kits. These will not be made publicly available at this stage, but it should be possible to build these kits in the Bottlerocket project. Some of the issues will be in the Bottlerocket project.

• #56
• #57

Create a design document in this repository that describes what twoliter is, what its requirements are, and how we think it can be built.

We need to use cargo-deny to make sure we don't introduce objectionably licensed software to twoliter or the tools.

If we don't integrate cargo deny yet then I'd at least like a backlogged issue for it.

Originally posted by @bcressey in #37 (review)

Description

Consider publishing this as a separate tool and/or lib.

sources/bottlerocket-variant

I expect this will go away as we eliminate variant-specific builds of anything. If that matches your understanding, I'm cool with this being duplicated as well. Otherwise, potentially we could turn it into a CLI tool that build scripts can invoke.

Originally posted by @bcressey in #37 (comment)

Definition of Done

When bottlerocket-variant has been published to crates.io and Bottlerocket and Twoliter are both using it, OR, when we have decided not to do so.

Description

With #14, twoliter make will be able to build bottlerocket so long as cargo make is available on the host. Eliminate this requirement.

Possibilities

• Use cargo make as a library.
• Embed cargo make as a binary in the tarball and install it
• Port all Makefile.toml functionality to native Rust functions (this would be ideal, but may take some time)

Definition of Done

When using Twoliter to build a Bottlerocket image no longer requires cargo make on the host.

Prerequisite

• #14

Create a workflow by which binary releases are attached to GitHub releases. Hint: cargo dist.

Use this as a tracking issue for all issues needed such that Twoliter can build a variant using local kits. In other words, change the Bottlerocket build so that kits are being created and used on-the-fly during the Bottlerocket build.

• #56
• #19
• #73
• #71

Pretty soon testsys will move to this repo, and when it does we need to deal with this in toml 0.7 or later. It is deprecated in 0.6 and removed on 0.7.

error[E0425]: cannot find function `tables_last` in module `toml::ser`
  --> testsys-config/src/lib.rs:24:39
   |
24 |     #[serde(flatten, serialize_with = "toml::ser::tables_last")]
   |                                       ^^^^^^^^^^^^^^^^^^^^^^^^ not found in `toml::ser`

          Sure, here's the log. Not sure where the error comes from, AFAICS everything should be built in docker and hence machine-agnostic? Not sure though where the local rust part comes in...

  bottlerocket-os/bottlerocket#21 236.6 error: could not compile `sundog`
  bottlerocket-os/bottlerocket#21 236.6 
  bottlerocket-os/bottlerocket#21 236.6 Caused by:
  bottlerocket-os/bottlerocket#21 236.6   process didn't exit successfully: `/usr/bin/rustc --crate-name sundog --edition=2018 api/sundog/src/main.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C opt-level=3 -C embed-bitcode=no -C debuginfo=2 -C metadata=f1dda1c0b15e0c90 -C extra-filename=-f1dda1c0b15e0c90 --out-dir /home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps --target x86_64-bottlerocket-linux-gnu -C linker=/usr/bin/x86_64-bottlerocket-linux-gnu-gcc -L dependency=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps -L dependency=/home/builder/.cache/release/deps --extern apiclient=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libapiclient-cee067d451d7b166.rlib --extern constants=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libconstants-37ede6ace73bb42f.rlib --extern datastore=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libdatastore-dbb12debb9ff125d.rlib --extern http=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libhttp-68066c2bc6086886.rlib --extern log=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/liblog-56fb0fccdab81be0.rlib --extern model=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libmodel-0c570bc5f68d5a14.rlib --extern serde=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libserde-0fe5639f2e2024f0.rlib --extern serde_json=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libserde_json-43d9398d57cbcc08.rlib --extern simplelog=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libsimplelog-192c7c07a4f82e7a.rlib --extern snafu=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libsnafu-8194cd9e9fb48158.rlib --extern tokio=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libtokio-0c8ac197f1b09636.rlib -Cprefer-dynamic -Copt-level=3 -Cdebuginfo=2 -Ccodegen-units=1 -Clink-arg=-Wl,-z,relro,-z,now -L native=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/build/ring-e73751118ec72435/out` (signal: 9, SIGKILL: kill)
  bottlerocket-os/bottlerocket#21 236.6 warning: build failed, waiting for other jobs to finish...
  bottlerocket-os/bottlerocket#21 372.4 error: Bad exit status from /var/tmp/rpm-tmp.t16z5j (%build)
  bottlerocket-os/bottlerocket#21 372.4 
  bottlerocket-os/bottlerocket#21 372.4 RPM build errors:
  bottlerocket-os/bottlerocket#21 372.4     Bad exit status from /var/tmp/rpm-tmp.t16z5j (%build)
  bottlerocket-os/bottlerocket#21 ERROR: executor failed running [/bin/sh -c rpmbuild -ba --clean       --undefine _auto_set_build_flags       rpmbuild/SPECS/${PACKAGE}.spec]: exit code: 1
  ------
   > [rpmbuild 7/7] RUN --mount=source=.cargo,target=/home/builder/.cargo     --mount=type=cache,target=/home/builder/.cache,from=cache,source=/cache     --mount=type=cache,target=/home/builder/rpmbuild/BUILD/sources/models/src/variant,from=variantcache,source=/variantcache     --mount=type=cache,target=/home/builder/rpmbuild/BUILD/sources/logdog/conf/current,from=variantcache,source=/variantcache     --mount=source=sources,target=/home/builder/rpmbuild/BUILD/sources     rpmbuild -ba --clean       --undefine _auto_set_build_flags       rpmbuild/SPECS/os.spec:
  ------
  executor failed running [/bin/sh -c rpmbuild -ba --clean       --undefine _auto_set_build_flags       rpmbuild/SPECS/${PACKAGE}.spec]: exit code: 1


  --- stderr
  BuildAttempt: Failed to execute command: 'docker build . --network none --target package --tag buildsys-pkg-os-x86_64-d4ae984f528b --build-arg PACKAGE=os --build-arg ARCH=x86_64 --build-arg GOARCH=amd64 --build-arg VARIANT=aws-k8s-1.24 --build-arg VARIANT_PLATFORM=aws --build-arg VARIANT_RUNTIME=k8s --build-arg VARIANT_FAMILY=aws-k8s --build-arg VARIANT_FLAVOR= --build-arg REPO=default --build-arg SDK=public.ecr.aws/bottlerocket/bottlerocket-sdk-x86_64:v0.29.0 --build-arg TOOLCHAIN=public.ecr.aws/bottlerocket/bottlerocket-toolchain-x86_64:v0.29.0 --build-arg NOCACHE=3534030891 --build-arg TOKEN=d4ae984f528b'
[cargo-make] ERROR - Error while executing command, exit code: 101
[cargo-make] WARN - Build Failed.

Originally posted by @pat-s in bottlerocket-os/bottlerocket#2685 (comment)

Description

          Reading this I found myself wondering if Rust had TCO.

Presumably we would not encounter a directory structure in the wild that could blow the call stack, but I still learned something reading about it.

Originally posted by @cbgbt in #23 (comment)

And Ben said

Potentially we could stop as soon as we reach a directory that has a .git subdirectory, since beyond that point it's unlikely we're going to find the Twoliter.toml that the user expects.

#23 (comment)

Description of Done

When Twoliter has a mechanism to stop its recursive search for the Twoliter.toml file such that a stack overflow cannot occur.

Description

When a kit is built, also create the metadata container that provides kit-metdata.json (or whatever we name it). The kit metadata file says what kits the current kit depends on. See design doc for more info.

TODO - prior to #4 it is unclear whether the responsibility for this will be in twoliter or buildsys. Move this issue or create a Bottlerocket issue if it becomes the responsibility of buildsys.

Definition of Done

When building a kit also creates a metadata container with a JSON file in it describing the kit's dependencies.

Depends on:

• #4
• bottlerocket-os/bottlerocket#3217

id: kitmeta
epic: kits

Related to bottlerocket-os/bottlerocket#2669

Description

Design an implement a way for Twoliter to re-use its bootstrapping container from one invocation to the next.
It needs to tag the built container in a consistent way based on the version of Twoliter that is running.
That way, it can refrain from re-building the container if the expected version is available.

Definition of Done

When Twoliter no longer needs to build its bootstrapping container every time it is invoked.
Also, when Twoliter builds the container because the version of the existing container does not match what it needs.

Depends on:

• #10

id: texec_container_ver
epic: twoliter_exec

Related to bottlerocket-os/bottlerocket#2669

Description

Original Description

It has always made me sad that we install tuftool so that pubsys can shell out to it. Why not make a lib.rs in tuftool that pubsys can use?

Anyway if tuftool is a requirement of pubsys then it should probably be added to twoliters bindeps.

Addendum

We should use tuftool as a library instead of calling out to a binary.

Definition of Done

When the tuftool binary is no longer embedded and part of Twoliter's tools install.

This change didn't make it over. Port this as a patch or just edit it. Basically get this change added to the Makefile.toml

https://github.com/bottlerocket-os/bottlerocket/pull/2757/files#diff-0f47e53927f2a76fca4bf29143c158748d4969cceac86d63c48c9f360a5a8656

Tracking issue for all sub-issues that are required for somewhat of a minimum viable product of out-of-tree builds. This will require the use of external kits from the Bottlerocket project. At this stage these will not be published externally, but provided one can build the Bottlerocket kits and place them in a container registry, then Twoliter should be able to use them and build an out-of-tree variant with some custom package definitions.

What I'd like:
I'd like to be able to use pubsys to push AMIs into multiple accounts within the same region in a single execution.

One possible way that this could be implemented is to optionally allow a list of credentials to be provided for aws.region.$REGION in Infra.toml, as opposed to a single element, e.g. allow both of these forms:

[aws.region.us-west-2]
# Account in us-west-2...

[[aws.region.us-east-1]]
# Account 1 in us-east-1...

[[aws.region.us-east-2]]
# Account 2 in us-east-1

Any alternatives you've considered:
This can currently be done by using multiple Infra.toml files as a workaround.

Description

Right now the Bottlerocket Makefile.toml accepts TWOLITER_LOG_LEVEL which turns on logging for the Twoliter binary. However, if a user needs to turn on a verbose level of logging for pubsys, pubsys-setup, or testsys, there is no way to do so.

Add to Twoliter's Makefile.toml variables for doing this and pass them to invocations of those tools.

Definition of Done

When builders of Bottlerocket can see verbose logging from pubsys, pubsys-setup, and testsys.

Use this as a tracking issue for all issues that need to be completed for kit support.

• #16
• #72
• #70
• #17

A tracking issue for subissues leading to Twoliter's ability to update kit and SDK dependencies with the command twoliter update.

Description

Get the twoliter make command working for all Makefile.toml targets.
This will require mounting the source repo into the Twoliter execution environment.

Definition of Done

When all of the cargo make targets in the Bottlerocket Makefile.toml can be executed through twoliter make.

Depends on:

• #13
• #7
• #9
• #32
• bottlerocket-os/bottlerocket#3294

id: texec_command_works
epic: twoliter_exec

Related to bottlerocket-os/bottlerocket#2669

Description

Before we can build a variant, we need to aggregate all of the necessary kits so that dnf install can see all the packages.
Implement this phase of the build such that either buildsys or twoliter (depending on what #4 decided) creates this aggregation image.

Definition of Done

When cargo make variant and/or twoliter build variant creates an aggregation of all the yum repos needed for the variant build.

Depends on

• #18

id: aggregatekits
epic: kits

Related to bottlerocket-os/bottlerocket#2669

Description

Twoliter will embed buildsys, pubsys, testsys and any other binaries that are built from the tools workspace.
These need to be built as part of the Twoliter build.
We will use bindeps for this: https://rust-lang.github.io/rfcs/3028-cargo-binary-dependencies.html

Installing Twoliter would look like this:

RUSTC_BOOTSTRAP=1 cargo -Z bindeps install twoliter

Some creativity will be needed.
What we want to happen during the above invocation is:

• buildsys, pubsys, etc are built and installed.
• The build.rs script for twoliter can find these binaries and create a tarball from them.
• The tarball is embedded into the twoliter binary.

This will allow the twoliter binary to add these binaries to its bootstrapping container.

Definition of Done

When RUSTC_BOOTSTRAP=1 cargo -Z bindeps install twoliter produces a binary that has buildsys and all the other tools embedded in it as a tools.tar.gz byte array constant.

Depends on:

• #6

id: texec_embed_tools
epic: twoliter_exec

Related to bottlerocket-os/bottlerocket#2669

What I expected to happen:
cargo make validate-repo to successfully validate the repo.

What actually happened:

Failed to validate repository: Failed to read target
'<>' from repo:
System time stepped backward: system time '2023-03-20 15:40:20.025217420 UTC',
last known time '2023-03-20 15:40:20.025739556 UTC' 1679326959302

See awslabs/tough#590 for more information.

How to reproduce the problem:

Remove generate-readme from the Twoliter project. Ensure that the README information for any package that happens to be currently using generate-readme is not duplicated. We should choose either main.rs/lib.rs *or8 README.md as the home for our documentation.

I've never been a fan of the generated README's anyway and Ben said:

sources/generate-readme

Should we just remove this dependency if we're not going to generate README's for everything? We could just patch that dependency out of bottlerocket-variant and parse-datetime build scripts after the import.

Originally posted by @bcressey in #37 (comment)

Description

Implement the kit dependency resolution algorithm described by #4. Write the resolved dependencies to Twoliter.lock.

Definition of Done

When Twoliter can resolve all external and local kit dependencies and write this dependency graph to Twoliter.lock.

Depends on:

• #4
• #7
• #17
• #16

id: kitdepsimpl
epic: kits

Related to bottlerocket-os/bottlerocket#2669

A tracking issue for moving Bottlerocket's buildsystem to Twoliter and building Bottlerocket with twoliter.

Description

Currently, the location of buildsys and other binaries is fixed in the repo at tools/bin.
This will not work with out-of-tree builds where we need to use pre-built build tools.
Make it possible to install and call the build tools binaries from arbitrary locations.

Definition of Done

A PR is merged that allows the user to decide where buildsys and other build tools will be installed and called from.

id: br_toolsdir
Related to bottlerocket-os/bottlerocket#2669