It should be possible to override protoc-bin-vendored about buck2

[thoughtpolice]

@thoughtpolice are you installing Buck2 by writing a Nix derivation? If we included that Nix derivation in the root of the repo, would that make it easy in Nix world?

Yes, this Nix derivation is here. There are one or two ugly hacks inside of it, but otherwise it is able to be built like (almost!) any other Rust program in Nixpkgs (which is the big library of Nix code that contains the set of things that Nix, the package manager, can install for you): https://github.com/thoughtpolice/buck2-nix/blob/main/buck/nix/buck2/default.nix

You might notice I also rename the binary from buck2 to buck. Slightly heretical I admit, but I'm not a user of the original Buck so I can get away with it :)

It would make it somewhat easier, but I hesitate to just add it, because:

• A normal Nix expression isn't enough in this case; we would likely want Buck to have its own "Nix Flake". Flakes are sort of an abstraction on top of Nix, but they allow some fun features.
  • It's not much more code in any case, so I wouldn't be scared.
  • It would let you do some cool things, for example any Nix user anywhere could simply run nix run facebook/buck2 -- --help (assuming buck2 eventually graduates to @facebook proper) and get a working copy of buck2 to play with! Yes, it's (really!) that easy.
• It would need to be CI'd in order to stop regressions
  • But that means it would be very annoying for non-Nix users to fix failures
  • Nix is not really an easy tool to debug if you haven't used it
  • e.g. the protoc-vendored patch would have been completely held up on it
• But most of all, we will need a working Cargo.lock file in order for Nix to build buck2.
  • This is effectively a must-have requirement.
  • Right now, I have a shell script that updates the buck2 Nix expression, and every time it does this, it re-generates Cargo.lock and includes it.
    • Enjoy and please excuse the 1-letter Haskell variable names: https://github.com/thoughtpolice/buck2-nix/blob/0ec7878847a7c4cc811ef302f5ca31ac45c47938/buck/nix/update.sh#L80-L124
  • My understanding is that Meta does not use lockfiles for its Rust projects, precisely because there is already a globally consistent view of all Rust dependencies inside your main repository (thanks to vendoring things and letting Buck itself drive the build). Therefore there is no need, because when you write needed-foo-dep = "0.8", there is no ambiguity about what it refers to (e.g. 0.8.1, 0.8.2, etc) — it refers exactly to the version of needed-foo-dep inside the repository itself.
• Inevitably, one day, a buck2 package will also appear upstream in Nixpkgs. This is perfectly OK and can live alongside any Nix expressions located in this repository (the Nix code here is used for development, the Nix code in Nixpkgs used to distribute things to users.) Not this day, but one day...

At the moment, I'd be happy to contribute one, but at the pace buck2 is moving before its stabilization I'm not sure if y'all want all these burdens. And the lockfile one is absolutely mandatory, but also seems to affect other projects like Sapling, where Nix developers have had to generate our own lockfiles and include them. So I don't know what the policy should be here.

That said — I find Nix to be wonderful to adopt in some sense, for projects like this, because I find it helps encourage codebases to follow best open source practices; when in Rome, do as the Romans. So even if it slows you down sometimes, it may help "keep you honest" e.g. you can't pull hacks like "oh I'm going to download something over the internet during the build", you have to explicitly declare all your needed deps, stuff like that. Stuff I think you probably certainly agree with!

I'd be happy to think about writing and helping maintain an upstream Nix derivation, but maybe it would be easier to hash out over a video call or something. Unfortunately I'm incredibly busy at least until Saturday, at this rate, so it would probably have to wait for Monday or Tuesday. Maybe something to think about!

[ndmitchell]

I think having all Nix users forced to carry a patch would be very sad, so definitely don't want to do that.

In a previous incantation, if PROTOC or PROTOC_INCLUDE were already set, we left them alone and didn't change them. That would be one thing to do, Thoughts @stepancheg ?

I also wonder if on Nix the protoc_bin_vendored should return an Err? Or maybe that's too hard to detect reliably.

[thoughtpolice]

Detecting when Nix is being used is probably not easy or reliable. Well, you could just ask if /usr exists, or you could try to just execute protoc --version and see if it fails, which is a pretty good, but...

It might also be worth just making the usage of protoc-bin-vendored a feature in the Cargo file, that can be disabled or enabled, I think? That way you have to opt into it pretty explicitly without just setting some magic environment variables that are ambient and not easily discoverable.

Please note that I have not tried to self-bootstrap buck2 with buck2 (I don't know if that will be the "recommended" way to build production binaries in the long run, once remote execution and caching are in place, for example) so I don't know if this might be impacted, but from reading previous commit messages, I think y'all have that already handled regardless.

[stepancheg]

PROTOC or PROTOC_INCLUDE were already set, we left them alone and didn't change them. That would be one thing to do

Should be OK, if we rename them to BUCK2_PROTOC and BUCK2_PROTOC_INCLUDE to avoid accidentally reading environment variables we are not supposed to.

on Nix the protoc_bin_vendored should return an Err? Or maybe that's too hard to detect reliably.

It is not the job of the crate to do it. It just returns the binary packed in the crate, that's it.

patchelf

Patchelf seems to be too hard to do reliably (because if has to do it inside the dependency crate).

Some people in the internet suggest using nix-ld. Can't it be used? I cannot understand it from the page description.

[ndmitchell]

My view is that working out the box is a good thing, so that means having protoc-bin-vendored be default is good, as it will reduce a lot of friction for Windows/Ubuntu users. Having that overridable by environment variables also sounds good (not too fussed on their names). Detecting Nix, running patchelf or a custom nix-ld all seem to go against the Nix philosophy, so seem less desirable.

@thoughtpolice are you installing Buck2 by writing a Nix derivation? If we included that Nix derivation in the root of the repo, would that make it easy in Nix world?

[thoughtpolice]

Some people in the internet suggest using nix-ld. Can't it be used? I cannot understand it from the page description.

nix-ld is clever but inappropriate. The assumption behind it is that you are running NixOS, and thus a path like /usr/lib does not exist. It then mounts a fake FUSE filesystem onto these paths, and intercepts calls for .so libraries. So, for one, it requires a FUSE daemon to exist, so that the filesystem can translate paths like /lib64/ld-linux-x86-64.so.2 into a different path in the /nix/store. And, this assumes you're running NixOS, but to be clear I am not; I am using normal Ubuntu Linux, with Nix (the package manager) installed on top of it. I do actually have a real /lib64/ld-linux-x86-64.so.2 on my system. So why is this all a problem?

This is because when you tell Nix to build something, it creates a hermetically sealed filesystem namespace under a different (random) Unix user, then runs the build process (e.g. cargo build) inside of it. It does all builds under these sealed namespaces, and they have a highly restricted view of the filesystem, in fact it isn't even possible to see any other /nix/store paths than ones you've declared as dependencies! This sealed namespace, where the build takes place, does NOT allow any other filesystems, network access, or anything else to be mounted inside the build.

If your build process says "Run this random pre-built binary we have provided, to complete the build", that binary has to work inside this environment, where it is not possible to do tricks like mounting things.

To be fair I should have been more clear about this second requirement, where the build takes place, but this same basic problem applies to NixOS too, not just the "Nix Build Environment." Any reliance on ambient paths like /usr/lib is inevitably destined to fail and need a workaround for these cases.

It is very good for normal users though, especially on cases like macOS or Windows (anything to help reduce reliance on third party things), so I think the overall idea is a good one, even if it can't help Nix people.

[stepancheg]

I'll restore env vars.

[stepancheg]

Committed. Closing the issue. Please reopen if this does not work as expected.