deepfire / nh

Manage Nix Haskell override sets

nh: manage Nix Haskell override sets

What

Manage a set of Nix Haskell package overrides, somewhat automatically.

Put another way, it aids definition and validation of properties of every override in a set:

• validity (whether it breaks things)
• necessity (whether it’s needed to keep things working, along with a proof)
• actuality (whether the override status is up to date with the upstream, Hackage and Nixpkgs)

Definitions are managed somewhat conveniently, like:

nh hackage    foo 1.2.14
nh hackage    foo                               # Pull the latest one
nh upstream   bar
nh upstream   bar ghc-8.4
nh unmerged   bar concerned-citizen PR-number

Brutal, no-nonsense introduction

1. Init a package DB:

   nh init
       

   ..which will ask you to create a config file first – follow the instuctions.

2. Add overrides:

   nh hackage lens                               # easy: there's a Hackage release, get the latest
       

   nh import       microlens-th                  # import metadata for microlens-th from Hackage cabal file
   nh upstream     microlens-th master           # Fetch the tip of upstream's master (any refspec goes)
   nh chdir        microlens-th microlens-th     # -- in case the Cabal metadata was wrong
   nh set-issue    microlens-th 222              # Specify the relevant upstream issue ID
       

   nh set-upstream hedgehog hedgehogqa
   nh unmerged     hedgehog gwils 134 [GIT-REV]  # the third argument to "unmerged" is the Github PR #
       

3. Observe the changes:

   nh show lens
   nh show microlens-th
   nh show hedgehog
       

   ..and you always can generate overrides.nix from the very latest package database – although it happens transparently most of the time:

   nh apply
   nh override  lens                             # ..to see the individual override
       

   The --disable-ghc-configuration switch will position your override set as a replacement to the global, version-specific GHC configuration.

   If you set TARGET_NIXPKGS in the .nh configuration file to a non-empty value, nh will emit non-local overrides directly into the Nixpkgs GHC configuration for the selected GHC version.

4. Validate if the overridden packages build now:

   nh acme                                       # Build all defined overrides
   nh build NAME..                               # ..as an alternative.
       

   ..and..

   nh progress                                   # ..to watch progress in another terminal
       

5. See what the audit tool thinks about the situation:

   nh audit --skip-acme [NAME..]                 # Defaults to everything, once more
       

   It should complain, at the very least because the trim part was not run, and so there’s no proof that any of the overrides are necessary.

   Note that the nt audit subcommand provides suggestions, that can be auto-executed when the --auto-fix option is supplied.

6. Let’s try trimming the set:

   nh trim [NAME..]
       

   That would take a while, and also it won’t do anything to the recorded overrides – it merely collected information, as an intermediate step.

   Let’s execute on that information, then:

   nh execute-trims
       

7. Audit again:

   nh audit
       

   ..and this is probably going to fail, because the trim is often over-eager. So, some of the overrides need to be reintroduced, and the figuring-out-what-and-why part is manual (albeit assisted). Thankfully, we can record the manually-collected information in the database (which would prevent trimming of this override in the future):

   nh set-explanation broken-attribute doCheck "....."
       

   Rinse, repeat – and remember that individual attributes can be rebuilt using nh baseline or nh build.

   Also, explanations (aka proofs of override) can be shown per-override with:

   nh explain happy src
       

Finally: look at suite.sh for inspiration.

Overview: lifecycle of fixes

First, we describe the general flow of fixes, to establish a terminology.

1. Fixes are generally born on a third-party Github repo, and they are expected to be submitted upstream via pull request.
2. The PR gets merged upstream.
3. Upstream cuts a release, bumping the package version in the cabal file.
4. Upstream performs a Hackage upload.
5. Nixpkgs imports Hackage, adding a versioned package-attribute_1_2_1_0.
6. Nixpkgs promotest the versioned package-attribute_1_2_1_0 to package-attribute, which completes the cycle.
7. Nixpkgs also supports non-source tweaks (jailbreaking out of restrictive version bounds, test and Haddock generation disables).

Overview: above lifecycle, seen by nh

nh maps the above into a status, per attribute:

unmerged

phases #1

upstreamed

phases #2 and #3

hackaged

phase #4

shadowed

phase #5 (after shadow attributes – those shadowing non-versioned ones)

config

not-really-phase #7

Key points

1. nh tracks the aforementioned attribute status and content of the attribute overrides in a package database (aka PKGDB). This is just a file-system directory – but it’s better to version it in Git, to be able to recover, when nh goes off-rails and breaks overrides.
2. The result is delivered in the form of a Nix file defining a GHC package set override (customarily called overrides.nix).

   This trivially-structured, generated file is then supposed to be imported into another, static Nix file called packages.nix, which then forms a proper GHC package set. That one can be passed to nix-build.

3. The major package DB operations that nh provides are:

   acme

   Build every attribute in the override set using a proxy, that depends on everything overridden (really, acme).

   trim

   Try to remove overrides, one by one, and record the results of those attempts in the package DB – trying to deducing whether these overrides are necessary. It is a heuristic.

   execute-trims

   Modify the package DB in accordance with the trim step. This effectively removes any overrides that weren’t found necessary. This is also error-prone (more things are sometimes removed than is feasible).

   audit

   Verify every attribute against a set of status-dependent invariants, that ensure:

   • the override necessity (along with the existence of proof), and
   • the override being up-to-date.

4. nh keeps as much build information as possible, and that includes store derivation links, store source links, override expressions and build logs for every build attempt that takes place. In particular every attribute build attempt happens in three phases, handled separately:
   • attribute instantiation
   • dependency pre-build
   • build of the attribute itself

Appendix: Example workflow of importing existing overrides

$ nh x hackage funcmp 1.9
downloading ‘http://hackage.haskell.org/package/funcmp-1.9.tar.gz’... [0/0 KiB, 0.0 KiB/s]
path is ‘/nix/store/akhnn03wfi3jlx2rqgwjdz07qpz983iz-funcmp-1.9.tar.gz’
  - 1d5appkjhajb9ndv2gwnfz8lw2w53v8baajzmrhg26ihzj1bkch8
  - https://hackage.haskell.org/package/funcmp-1.9

$ nh set-explanation funcmp src
funcmp.def/meta.src.explanation: Needed for (<>) in prelude

$ nh jailbreak deepseq-generics

$ nh set-explanation deepseq-generics jailbreak
deepseq-generics.def/meta.jailbreak.explanation: https://github.com/haskell-hvr/deepseq-generics/pull/4

$ nh import securemem  # this fetches metadata like repoName, upstream, chdir etc.

$ nh unmerged securemem shlevy 12 6168d90b00bfc6a559d3b9160732343644ef60fb
- 06dhx1z44j5gshpdlsb4aryr3g4was3x4c2sgv1px8j57zrvlypx
- https://github.com/vincenthz/hs-securemem/commit/6168d90b00bfc6a559d3b9160732343644ef60fb

Appendix: Structure of the package database

def

definitions

meta

non-override metadata

over

overrides

hackage, github

src-specific information, per-attribute-override

cache

override cache, per-attribute

build

build output information: logs, expressions, derivations

Appendix: help

Usage:  nh [--cls] [--nixpkgs] [--trace] [--debug] [--quiet] SUBCMD [SUBARGS..]

NOTE:  if --nixpkgs is passed, non-local overrides instead serve as definition
       for /home/deepfire/nixpkgs/pkgs/development/haskell-modules/configuration-ghc-8.4.x.nix


  PKGDB:

   forall-defined-edit TYPE FIELD
                             Interactively edit all FIELD definitions of TYPE

  Metadata (non-override):

   ls-meta ATTR              List attribute's metadata (as opposed to overrides
   meta ATTR META            Print a single metadata entry of an attribute
   set-meta ATTR META VAL    Set a single metadata entry of an attribute
   edit-meta ATTR META       Edit the current attribute's meta value using readline
   disable ATTR[.OVER]       Disable all/single overrides for an attribute
   enable ATTR[.OVER]        Re-enable previously disabled overrides
   with-disabled-attrs ATTR..
                             Disable all listed attribute overrides and pause;  Re-enable on exit or newline in stdin
   ls-disabled               List all disabled attributes
   set-explanation ATTR OVER VAL
                             Manually supply explanation for an override's existence
   set-erdeps ATTR 'ATTR..'  Set attribute's essential rev-deps that must keep working
   chdir ATTR SUBDIR         Change directory before build;  "" removes the override
   local ATTR                Mark ATTR as local: not subject for Nixpkgs GHC configuration
   nonlocal ATTR             Remove marking of ATTR as local

  Override manipulation (low level):

   remove ATTR[.OVER]        Remove specified overrides
   ls-over ATTR              List attribute's overrides
   ls-input-overs ATTR       List attribute's input overrides
   get ATTR OVER             Get an attribute's override value
   set ATTR OVER VAL         Set an attribute's override value;  "" removes the override
   edit ATTR OVER            Edit the current attribute's value using readline
   set-input-over ATTR INPUT VAL
                             Set ATTR's override for INPUT
   edit ATTR OVER            Edit the current attribute's value using readline
   check ATTR                Disable an existing dontCheck override
   dontCheck ATTR            Disable tests
   haddock ATTR              Disable an existing dontHaddock override
   dontHaddock ATTR          Disable Haddock generation
   jailbreak ATTR            Turn on jailbreaking
   dontJailbreak ATTR        Disable an existing jailbreak override
   {library,executable,test}Haskell ATTR [ATTR..]
                             Specify extra *HaskellDepends;  "" removes the override
   add-patch ATTR SHA256 URL Add a patch to ATTR

  Status:

   status ATTR               Print status of a single attribute
   ls-shadowed               List all attributes with status 'shadowed'
   ls-hackaged                                            ...'hackaged'
   ls-upstreamed                                          ...'upstreamed'
   ls-unmerged                                            ...'unmerged'
   ls-config                                              ...'config'

  Nix-level inferences:

   drv ATTR                  Store derivation for a single override
   pprint-drv ATTR           Pretty-print ATTR's derivation (requires nix-derivation-pretty)
   src ATTR                  Store source derivation for a single override
   src-drv ATTR              Store source derivation of ATTR
   src-url ATTR              Source URL of ATTR
   inputs ATTR               ATTR's store inputs
   deps | refs | references ATTR
                             ATTR's store drv dependencies
   rdeps | referrers ATTR    ATTR's store reverse drv dependencies
   realise-drv ATTR          Realise ATTR's derivation
   drv-pprint STORE-DRV      Pretty-print a Nix-stored .drv file
   src-drv ATTR              Store source derivation of ATTR
   src-drv-url STORE-DRV     Source URL of a Nix-stored source-.drv file
   drv-inputs STORE-DRV      Store inputs for a Nix-stored .drv file
   drv-refs | drv-references STORE-DRV
                             Store .drv references for a Nix-stored .drv file
   deriver-of STORE-PATH     Store .drv for a Nix store path.  Will fail if built non-locally

  PKGDB emission to Nix overrides:

   over | override | show-override ATTR
                             Print the attribute's override defined by PKGDB
   apply [--reuse-cache]     Apply all overrides via /home/deepfire/overrides.nix
   cache [--require-descs]   Regenerate override cache
   show-cache ATTR           Print the cached text of attribute's override (DEBUG)

  General:

   ls [REGEX]                List all overridden attributes
   info ATTR                 Overview of an attribute's PKGDB
   overview [ATTR..]         List overridden attributes, grouped by status + relevant info

  Hackage:

   import ATTR               Scrape ATTR's Cabal file from Hackage for some properties
   cabal ATTR                Print the latest released cabal file for ATTR
   hackage ATTR [RELEASE=upstream-latest]
                             Override to a Hackage release

  Github:

   github ATTR [REF]         Override ATTR to its latest upstream Github commit
   unmerged ATTR USER PR# [REV=HEAD]
                             Override to a 3rd-party Github commit
   upstream ATTR [REV=HEAD]  Override to an upstream Github commit
   set-upstream ATTR GITHUB-USER
                             Specify an attribute's Github upstream username
   edit-upstream ATTR        Edit an attribute's Github upstream username
   set-pr ATTR PR#           Set the PR# of an attribute's Github override
   set-issue ATTR ISSUE#     Set the Issue# of an attribute's Github override
   set-repoName ATTR REPO    Set an attribute's Github repository name
   edit-repoName ATTR        Edit an attribute's Github repository name

  Build & results:

   instantiate [--reuse-cache] [ATTR..]
                             Instantiate overridden attrs (or specified subset)
   acme [--reuse-cache]      Build everything at once, collecting all failures
   build [COMMON-OPTS] ATTR  Build a single attribute with current overrides
   log ATTR [OVER=baseline]  Obtain trim build logs for a single override
   failure ATTR [OVER=baseline]
                             Obtain trim failure kind of an override
   failure-log ATTR [OVER=baseline]
                             Obtain trim failure log of an override
   failure-type ATTR [OVER=baseline]
                             Obtain trim failure type of an override
   proof ATTR [OVER]         Print an override's proof of necessity. When OVER is empty, print context.

  Override database maintenance:

   trim [--reuse-cache] [ATTR..]
                             Suggest a reduction to the override set (or specified subset)
   trim-override ATTR OVER   Attempt trimming a specific override of a given attribute
   show-trims                Show the trim suggestion
   execute-trims             Execute the suggestion
   audit [--autofix] [--autoonly] [--skip-acme] [--reuse-{overrides,cache}] [ATTR..]
                             Sanity check the overridden attrs (or specified subset).  --autofix applies suggestions
   extra-validation-attributes
                             Edit the set of attributes validated regardless of being overridden
   edit-fixed-content        Edit the static part of the GHC configuration

  Nix shell:

   shell                     Nix shell with up-to-date overrides (shell.nix required)
   shell-for ATTR            Nix shell for building ATTR
   cabal-shell               Nix shell from a cabal file (nothing else required)
   clone-upstream-fixer-shell
                             Nix shell from a cabal file (nothing else required)
   try-fix ATTR              Push the current commit and try the fix
   find-module NAME          Convenience alias for 'ghc-pkg find-module NAME'
   list-packages ...         Convenience alias for 'ghc-pkg list ...
   describe-package ATTR     Convenience alias for 'ghc-pkg describe ATTR
   package-modules ATTR      List ATTR's exposed modules
   phases ATTR               Print ATTR's build phases

  Miscellanea:

   eval BASH-EXPR            Passthrough, to execute anything defined.
   loop-hunter               Detect attribute loops:  nix-shell 2>&1 | nh loop-hunter
   ls-builds                 List active builds
   progress [LOG]            Live summary of new, complete and failing builds
   watch                     Observe the current build, as it hits the logs..
   ghc                       Shell with current GHC
   prefetch-ghc GITREV       Prefetch a GHC revision
   less-ghc-config [NEEDLE]  Run less on the Nixpkgs GHC configuration
   git OPTIONS.. ARGS..      Run git inside controlled Nixpkgs
   nixpkgs-diff [(base-head|base-master|head-master] [REF]
                             Diff of current GHC configuration

nh's Issues

Another case of munged newline

1. funcmp was overridden to 1.9 and jailbroken in the PKGDB.
2. The trimmer for jailbreak was building 1.8: https://github.com/deepfire/configuration-ghc-8.4.x/blob/master/build/log/funcmp.jailbreak
3. This happened due to the version shadowing having been commented due to a missing newline: https://github.com/deepfire/configuration-ghc-8.4.x/blob/master/build/expr/funcmp.jailbreak#L25

Rewrite in Haskell

Once the prototyping phase is complete and #2 is resolved, this needs to be rewritten in Haskell.

Gang `chdir` overrides with 'src' when trimming

chdir only happen because we override src from Hackage to a git checkout.

Gang it with src, since disabling them separately guarantees breakage.

Better yet, demote it from an override to a def/github property and conditionalise emission.

Trim of hspec-core fails to capture evidence

deepfire@andromedae:~/holotype$ nh audit --skip-baseline
Generating override cache: 67/67
Assembling overrides..................................................................... done.
                ChasingBottoms: no upstream specified
              hspec-core.doCheck: no proof:  nh x emit_explanation doCheck hspec-core
              hspec-core.doCheck: retrim:    nh trim --skip-baseline hspec-core
               vector-algorithms: no upstream specified
deepfire@andromedae:~/holotype$ nh trim --skip-baseline hspec-core
Generating override cache: 67/67
-( trimming in progress
TRIM hspec-core>
TRIM hspec-core -all>  Assembling overrides..................................................................... done.
FATAL: failed to build 'hspec-core', without overriding 'default'
FATAL: logs:  nh get build/log default hspec-core
HASKELL-MISSING-INSTANCE
    • No instance for (Semigroup Summary)
        arising from the superclasses of an instance declaration
    • In the instance declaration for ‘Monoid Summary’
FAILED
TRIM hspec-core -doCheck>    ## Needs bump to a versioned attribute
  ##
  ##     • No instance for (Semigroup Summary)
  ##         arising from the superclasses of an instance declaration
  ##     • In the instance declaration for ‘Monoid Summary’
  hspec-core = super.hspec-core_2_4_7;

Assembling overrides..................................................................... done.
FATAL:  failed to instantiate hspec-core, the repro is:
FATAL:  nix-instantiate packages.nix -A hspec-core --argstr compiler ghc841 --show-trace
error: while evaluating the attribute ‘buildInputs’ of the derivation ‘hspec-core-2.4.7’ at /home/deepfire/src/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:148:11:
while evaluating the attribute ‘buildInputs’ of the derivation ‘silently-1.2.5’ at /home/deepfire/src/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:148:11:
while evaluating the attribute ‘buildInputs’ of the derivation ‘temporary-1.2.1.1’ at /home/deepfire/src/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:148:11:
while evaluating the attribute ‘buildInputs’ of the derivation ‘base-compat-0.9.3’ at /home/deepfire/src/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:148:11:
while evaluating the attribute ‘propagatedBuildInputs’ of the derivation ‘hspec-2.4.7’ at /home/deepfire/src/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:148:11:
infinite recursion encountered, at undefined position
FATAL: failed to instantiate 'hspec-core', without overriding 'doCheck'
FATAL: logs:  nh get build/log doCheck hspec-core.instantiate
FAILED
TRIM hspec-core -src>    hspec-core = overrideCabal super.hspec-core (drv: {
    doCheck         = false;
  });

Assembling overrides..................................................................... done.
FATAL: failed to build 'hspec-core', without overriding 'src'
FATAL: logs:  nh get build/log src hspec-core
HASKELL-MISSING-INSTANCE
    • No instance for (Semigroup Summary)
        arising from the superclasses of an instance declaration
    • In the instance declaration for ‘Monoid Summary’
FAILED
deepfire@andromedae:~/holotype$ nh audit --skip-baseline
Generating override cache: 67/67
Assembling overrides..................................................................... done.
                  ChasingBottoms: no upstream specified
              hspec-core.doCheck: no proof:  nh x emit_explanation doCheck hspec-core
              hspec-core.doCheck: retrim:    nh trim --skip-baseline hspec-core
               vector-algorithms: no upstream specified

Batch `revision` and `editedCabalFile` together

Those go together well, and not so much apart.. batch them! : -)

Better Haskell-as-bash experience

Obtain a Haskell library that would accomodate bash-style programming better than https://github.com/psibi/shell-conduit / https://github.com/yesodweb/Shelly.hs / https://github.com/Gabriel439/Haskell-Turtle-Library.

src-overriden attributes confusingly keep the original version

It can be confusing when overriding a non-version-suffixed attribute whose Nixpkgs definition is lagging behind upstream Github / hackage -- the override's effective version will not match.

We should override the version as well.

Extend validation beyond what is overridden

Sometimes you care about a specific package which is not in need of an override -- and yet it can be broken, directly or indirectly, by override set changes.

We'd need to facilitate this testing -- at least by adding these target packages to the baseline set.

`baseline` workflow is too ping-pongy: failure batching needed

The workflow of nh baseline suffers from a lack of batching -- it could benefit from a two-pass processing:

1. Collect names of failed derivations using nix-build --keep-failing -A X, where X is some kind of a synthetic derivation depending on all our overrides.
2. Rebuild failed derivations individually, to collect precise failures.

Seamless management of `pkgs/development/haskell-modules/configuration-ghc-8.4.x.nix`

Currently, to manage the GHC-version-specific override sets, a number of manual steps are required:

1. Interning of the new overrides from Nixpkgs that aren't present in the PKGDB
2. Decision of what from the above should remain unmanaged
3. Removal of all managed overrides from the GHC configuration file
4. Actual management
5. Re-insertion into the Nixpkgs's GHC configuration file
6. Double-checking that all is still fine (nh baseline)

This is mostly avoidable if we do:

1. Eliminate the necessity for unmanaged overrides (mostly there)
2. Automate the override set generation/movement

A way to store a subset of override metadata directly in `configuration-ghc-*.nix`?

We're likely to need the stateful database for many things (TODO: list them), but at least some of the override metadata can be stored directly in the override configuration file itself.

Some of the metadata can be even lifted from the override definitions themselves -- by parsing the Nix file with hnix, for example.

Reuse `/nix/var/log/nix/drvs/`

/nix/var/log/nix/drvs/ carries logs for all derivation builds, including failed ones.

This could partially replace logging for one of the four build steps in override_builder().