raitobezarius / buildxyz

Build your Nix package automatically

buildxyz

Build your (Nix) package automatically.

Introduction

buildxyz is a Rust program running your build system invocation and injecting into many commonly used environment variables extra search directories to trap any filesystem access that cannot be provided by your existing environment.

By doing so, buildxyz can know what your build system needs and will provide it dynamically using the Nix store and a nixpkgs index database.

Finally, once your build system built, buildxyz remembers what it actually had to provide and can rematerialize the set of dependencies provided for any usage, e.g. providing this data to nix-init to automatically write a Nix derivation, fix implicit dependencies, etc.

Design

buildxyz relies on the FUSE technology to instantiate an instrumented filesystem which does not require any higher privilege.

Here is a sequence example of buildxyz operating:

sequenceDiagram
    Build system->>Operating System: open(/usr/bin/pip)
    Operating System-)Build system: OK
    Build system->>Operating System: open(/usr/lib/cuda/...)
    Operating System->>Build system: ENOENT
    Build system->>Operating System: open(/tmp/buildxyz/lib/cuda/...)
    Operating System->>BuildXYZ: lookup(/tmp/buildxyz/lib/cuda/...)
    BuildXYZ->>Database: search this path
    Database-)BuildXYZ: /nix/store/eeeeeeeeee-cuda/
    BuildXYZ->>User: Do you want to use this dependency or provide your own?
    User-)BuildXYZ: Use the preferred candidate
    BuildXYZ->>Build system: This is a symlink to /nix/store/eeeeeeeeee-cuda/
    Build system->>Operating System: readlink(/nix/store/eeee..-cuda)
    Operating System->>Build system: OK

Actually implemented

BuildXYZ can already provide dependencies to your build system based on a precise revision of nixpkgs, pinned in the default.nix.

It is known to work on Python's packages (through pip install --no-binary :all:) and sometimes on certain autotools project depending on their complexity.

Resolutions

When BuildXYZ receives a new filesystem access, it means that the existing environment failed to provide it.

If the filesystem access has a match in the nixpkgs index database, two options are possible:

• provide it
• do not provide it

We call resolution the information composed of a filesystem access identified by a canonical path and a decision: provide it or not.

Not all filesystem accesses should be provided even if we have matches for them, that's why we enable custom resolutions which can be managed through policies: user interaction, language-specific resolutions, etc.

The resolution data for a project is very interesting as it is exactly the "implicit dependencies" data that is required to build a project, which is often described through instructions.

Goals & TODO

Current objective: get Nix to compile without any manually provided dependency using BuildXYZ.

• Proper restart & program lifecycle (Ctrl-C, SIGTERM)
• Proper discovery of existing resolutions databases
• Proper flags to record new resolutions or merge them in an existing file
• Human/machine-readable format for resolutions
• Extend graphs of dependencies with implicit dependencies
• nix-init integration

Usage

Run the project:

nix run github:RaitoBezarius/buildxyz

Build the project:

nix build

Run all tests:

nix flake check -L

Run formatters:

nix fmt

buildxyz's Issues

Build system support

Base targets

• pip
• CMake
• autotools
• meson

No node2nix with current revision of buildxyz ?

Using a clone of https://github.com/mathieudutour/medium-to-own-blog I was able (./result-2/bin/medium-to-own-blog --help) to nix-build after nix run github:RaitoBezarius/buildxyz/8a40f63c8283a04127a8e2bf1f418b415ea10dbb -- "node2nix --lock ./package-lock.json"
which doesn't seem to work that way anymore (ed54bde).

Is that use case in the scope of buildxyz and if so can it be documented how to achieve this ?

Error handling

Channels error management

Each .expect or unwrap on send/recv should be checked wrt to the semantics of channels.

Propagating correctly errors

TODO.

Integration with nix-init

This is more of an open-ended discussion, if you are interested.

I am the maintainer of nix-init, which in short, generates nix expressions for packages interactively, and has some dependency inference based on language specific dependencies (e.g. Cargo.lock). It was suggested to me that an automated build-fail-adjust loop would be nice, which sounds similar to what buildxyz does under the hood, maybe we can work something out?

nixpkgs issues in data

• https://github.com/NixOS/nixpkgs/blob/nixos-22.11/pkgs/applications/science/logic/saw-tools/default.nix#L37-L38 : expose libraries as a top-level
• sourcehut.python: toplevel → should be hidden in passthru
• TRACE - extracting pop for /nix/store/9nawav9bn28w8k38gvr5nfcnm7d2ska4-xorg-sys-opengl-3: xorg_sys_opengl for libexpat.so.1: https://github.com/NixOS/nixpkgs/blob/nixos-22.11/pkgs/os-specific/linux/opengl/xorg-sys/default.nix

Project does not build

╭─[/tmp/tmp.mZBos6dwNd/buildxyz]─[main]
╰─ % git checkout record-resolutions
branch 'record-resolutions' set up to track 'origin/record-resolutions' by rebasing.
Switched to a new branch 'record-resolutions'

╭─[/tmp/tmp.mZBos6dwNd/buildxyz]─[record-resolutions]
╰─ % nix build
error: build of '/nix/store/v24j070axpv9nmgrrk51xv7inrni25rn-buildxyz-0.0.1.drv' on 'ssh-ng://[email protected]' failed: builder for '/nix/store/v24j070axpv9nmgrrk51xv7inrni25rn-buildxyz-0.0.1.drv' failed with exit code 101;
       last 25 log lines:
       >
       > error[E0308]: mismatched types
       >    --> src/fs.rs:348:49
       >     |
       > 348 |             let nix_path = data.store_path.join(&data.file_entry_name.into()).into_owned().as_str().as_bytes().to_vec();
       >     |                                            ---- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected enum `Cow`, found reference
       >     |                                            |
       >     |                                            arguments to this function are incorrect
       >     |
       >     = note:   expected enum `Cow<'_, str>`
       >             found reference `&_`
       > note: associated function defined here
       >    --> src/cache/package.rs:211:12
       >     |
       > 211 |     pub fn join(&self, entry: Cow<str>) -> Cow<str> {
       >     |            ^^^^        ---------------
       > help: consider removing the borrow
       >     |
       > 348 -             let nix_path = data.store_path.join(&data.file_entry_name.into()).into_owned().as_str().as_bytes().to_vec();
       > 348 +             let nix_path = data.store_path.join(data.file_entry_name.into()).into_owned().as_str().as_bytes().to_vec();
       >     |
       >
       > For more information about this error, try `rustc --explain E0308`.
       > warning: `buildxyz` (bin "buildxyz") generated 3 warnings
       > error: could not compile `buildxyz` due to previous error; 3 warnings emitted
       For full logs, run 'nix log /nix/store/v24j070axpv9nmgrrk51xv7inrni25rn-buildxyz-0.0.1.drv'.
error: builder for '/nix/store/v24j070axpv9nmgrrk51xv7inrni25rn-buildxyz-0.0.1.drv' failed with exit code 1

╭─[/tmp/tmp.mZBos6dwNd/buildxyz]─[record-resolutions]─[1m 1s]