Integrating attestation verification into `brew install` (and `brew verify`) about brew

[carlocab]

How difficult would it be to support verifying attestations natively in Ruby?

I'd prefer to avoid needed gh to verify attestations, since that presents challenges to verifying the gh attestation.

[MikeMcQuaid]

Thanks @woodruffw!

[ ] Add a BREW_VERIFY_ATTESTATIONS environment config, which will default to false during the beta period.

Should probably default to an empty/unset variable being false rather than handling false, just FYI.

• When BREW_VERIFY_ATTESTATIONS=true, brew install will additionally perform verification of each bottle's attestation as it gets downloaded (or reused from cache).

Should probably be =1 or really set to any, non-blank value.

• Regardless of BREW_VERIFY_ATTESTATIONS, the separate brew verify [formulae ...] command will allow users to explicitly verify bottle(s) by their formulae names.

👍🏻

• How long should the beta last?

I think it's less about the time and more about just gradual ramp-up based on bugs/issues/performance regressions seen.

Generally these things go through:

• require ENV variable
• enabled by default for HOMEBREW_DEVELOPERs
• enabled by default in CI
• optional: enabled by default for HOMEBREW_DEVCMD_RUN
• enabled on master, not yet in a stable tag
• in a stable tag

• Attestation verification currently requires the gh CLI, meaning that BREW_VERIFY_ATTESTATIONS=true will require a bootstrapping phase with brew install gh or similar to start. Should this be hash-pinned or similar?

I think this is fine while the ENV variable is opt-in or for the brew verify command. I agree that making this a hard requirement for all of brew install invocations is something we probably need to resolve before this can go out to everyone as default behaviour (and probably even go to homebrew developers/CI).

Once enabled by default, this will enable build provenance (modulo the backfill) verification for every bottle in homebrew-core, changing Homebrew's bottle integrity guarantee from "it matches a known digest" to "it matches a known digest and it was produced on Homebrew's own CI/CD."

Sounds good, assuming the performance overhead here is minimal.

[segiddins]

https://github.com/segiddins/sigstore-cosign-verify should almost be usable!

[Bo98]

The closest implementation is probably the one that the RubyGems folks have been working on (they have similar native-Ruby-only requirements), but IIUC they're still in active development as well and probably aren't ready for downstream dependencies yet.

This might be an interesting watch. We don't necessarily have to move quickly here. If this route looks dead later in the year we can re-evaluate options but it seems worthwhile watching that space as a potential way forward removing the env-gate.

If RubyGems needs/likes any input/contributions from me then I'm happy to do that too.

[MikeMcQuaid]

Makes sense. To align this and @carlocab's comment: is the thought that users would have to explicitly brew install gh first, only after which BREW_VERIFY_ATTESTATIONS=1 brew install ... would be effective?

I think it'd be OK for it to install gh itself (we have a one-liner for such things), it's more just that it can't really be default until it's entirely in Ruby.

[Bo98]

even if only as a dev-cmd

I'd be ok as a dev-cmd if we make it clear in the command description too that it's under development. We can promote it to cmd when we're ready to remove the env-gate.

[woodruffw]

seems to work well, but verifying attestations seems a bit slow:

Yeah, I think there's some low hanging fruit that I can optimize here: gh attestation verify is probably pulling more things down the pipe than it strictly needs to, and there may be optimizations that it can further do internally. I'll do some profiling.

[woodruffw]

See also #16543 for the prior phase of this 🙂

[Bo98]

I'm OK relying on gh as an opt-in env for brew install (and keeping brew verify in a separate tap) initially.

I'm less OK with doing so unconditionally as we've generally been moving away from arbitrary external tools (beyond guaranteed system-provided tools on macOS). See past efforts of: moving from patchelf to patchelf.rb, readelf to rbelftools, the current PR for ldd and the existence of Portable Ruby. The chicken egg problem of verifying gh itself makes this more awkward and having to issue a new brew release every time gh is updated sounds annoying. A mini bundled gh (like Portable Ruby) used exclusively for this may however be possible if necessary.

But a lot of this can be "questions for later". I was going to point out gh-attestation being closed source but I think that got fixed in yesterday's gh release? So seems like the picture is already changing. How it actually works is now clearer and I'll take a look over the code to give myself a more informed idea of what's actually involved here rather than guessing what the black box does.

[woodruffw]

How difficult would it be to support verifying attestations natively in Ruby?

I'd prefer to avoid needed gh to verify attestations, since that presents challenges to verifying the gh attestation.

I strongly agree, although unfortunately the answer to "how difficult" is currently "very difficult" -- verifying an attestation here requires a full Sigstore verifier implementation, and there currently isn't a suitable native Ruby one that's ready for use here.

The closest implementation is probably the one that the RubyGems folks have been working on (they have similar native-Ruby-only requirements), but IIUC they're still in active development as well and probably aren't ready for downstream dependencies yet.

---

Should probably default to an empty/unset variable being false rather than handling false, just FYI.

Should probably be =1 or really set to any, non-blank value.

Makes sense! I was using shorthand here; I figure this would go in Homebrew::EnvConfig, which would abstract all the 1, bool, etc. variants 🙂

---

I'm less OK with doing so unconditionally as we've generally been moving away from arbitrary external tools (beyond guaranteed system-provided tools on macOS). See past efforts of: moving from patchelf to patchelf.rb, readelf to rbelftools, the current PR for ldd and the existence of Portable Ruby. The chicken egg problem of verifying gh itself makes this more awkward and having to issue a new brew release every time gh is updated sounds annoying. A mini bundled gh (like Portable Ruby) used exclusively for this may however be possible if necessary.

Makes sense. To align this and @carlocab's comment: is the thought that users would have to explicitly brew install gh first, only after which BREW_VERIFY_ATTESTATIONS=1 brew install ... would be effective?

If so, that makes sense to me (in the sense that it does an end-run around the bootstrapping problem) and IMO is an acceptable solution until we can vendor-in a pure Ruby attestation verifier.

My proposal with that:

• BREW_VERIFY_ATTESTATIONS=0 brew install: current, default behavior
• BREW_VERIFY_ATTESTATIONS=1 brew install with no gh installed: failure
• BREW_VERIFY_ATTESTATIONS=1 brew install with gh installed: performs attestation verification

[woodruffw]

Also: re: brew verify: keeping it in an external tap is OK by me if that's the consensus, but I'd prefer to upstream it (even if only as a dev-cmd) if possible: my reasoning is that brew verify will share almost exactly the same underlying implementation as the feature-flagged verification under brew install, so IMO it makes sense to de-dupe the code 🙂

[woodruffw]

If RubyGems needs/likes any input/contributions from me then I'm happy to do that too.

Paging @segiddins 🙂

---

I'd be ok as a dev-cmd if we make it clear in the command description too that it's under development. We can promote it to cmd when we're ready to remove the env-gate.

Sounds good! We'll make that explicit.

[woodruffw]

I've opened #17049 with the first bits of this: that PR adds HOMEBREW_VERIFY_ATTESTATIONS and directs brew update to bootstrap gh if HOMEBREW_VERIFY_ATTESTATIONS is set. Once set, brew install verifies the attestations for any bottles being fetched from homebrew-core.

[carlocab]

#17049 seems to work well, but verifying attestations seems a bit slow:

❯ brew uninstall hello
Uninstalling /opt/homebrew/Cellar/hello/2.12.1... (11 files, 177KB)
❯ time (HOMEBREW_VERIFY_ATTESTATIONS=1 brew install hello &>/dev/null)
( HOMEBREW_VERIFY_ATTESTATIONS=1 brew install hello &> /dev/null; )  0.85s user 0.27s system 37% cpu 2.982 total
❯ brew uninstall hello
Uninstalling /opt/homebrew/Cellar/hello/2.12.1... (11 files, 177KB)
❯ time (HOMEBREW_VERIFY_ATTESTATIONS= brew install hello &>/dev/null)
( HOMEBREW_VERIFY_ATTESTATIONS= brew install hello &> /dev/null; )  0.62s user 0.19s system 79% cpu 1.018 total

Note: bottle pre-fetched so download time is not included