ryantm / nixpkgs-update

Updating nixpkgs packages since 2018

nixpkgs-update

Please read the documentation.

nixpkgs-update's Issues

Compute rebuilds to determine staging/master base

Integrate other package updaters

It would be really cool if we could integrate a few other package updaters into nix-update. We have a couple of different manual package updaters being run regularly. Usually people are good at doing this regularly but it definitely increases Nixpkgs' bus factor. Automating this process through nix-update could be very helpful. Here's my initial list along with the maintainers:

• MELPA and ELPA updaters /cc @mdorman @ttuegel
• Hackage updaters /cc @peti
• NPM updaters /cc @svanderburg

Some other packages also have "update.nix" that might be possible to use. This was initially done by @garbas "update script" but I'm not sure if it's still actively being maintained.

Every maintainer now has a Github handle!

With NixOS/nixpkgs#36119 closed, the maintainers.nix now contains a complete set of github handles for (nearly) each maintainer. They are checked to be correct (see PR). Attention, some maintainers don’t have a github field.

Enhancement: tag auto-generated pull requests

It would be great to have the tool label the generated pull requests as "auto" or similar (in a nice color of course :-) Helps browse and filter the growing list of pull requests.

PR lifecycle tools

If nixpkgs-update was more aware of the PRs it generates it could have a lot of nice features:

• Delete branches of all closed or merged PRs
• Maintain PR collection PRs: For example, there could be a PR that suggested "merge all the currently open staging PRs to staging." It could have "closes #blah" in the PR text, so it auto closed all the ones it was currently based on.
• Move all PRs with tag 501+ linux to staging
• Maybe something like #29

Incorrect derivation file sometimes

Our new way of figuring out the derivation file is sometimes wrong. For example,

libgit2 picked the specific version instead of the non-specific one: "libgit2_0_25: 0.25.1 -> 0.26.3"
NixOS/nixpkgs#37474

Same for isl "isl_0_15: 0.15 -> 0.19"
NixOS/nixpkgs#37453

cc @matthewbauer

Update criteria too restrictive

1. It wants to see the old version repology says nixpkgs-unstable is at to do an update, so it won't update in the case where someone else has updated the expression on master to a version between the OLD and NEW versions.
2. It wants no branch to exist for updating the package, so it won't do an update if there is a pending PR updating to a version between the OLD and NEW versions.

Try ignoring trailing ".0" in version string

Here's an example where Debian/Ubuntu used "19.0" for the version even though upstream says the version is "19":
NixOS/nixpkgs#43357

If the version from Repology doesn't work and contains trailing ".0"s we should try removing them one by one to see if that works.

detect expressions that take arguments to produce multiple derivations

For example in NixOS/nixpkgs#39860 (comment) , the same file is used for the qt4 and qt5 qscintilla derivations.

If we can detect expressions used by multiple derivations, we can avoid building them, and eventually figure out how to build/test both.

Put link to Repology meta-package in PR

This might be helpful when maintainers aren't 100% sure if this is truly the "latest" version.

URL is:

https://repology.org/metapackage/$PACKAGE_NAME/versions

Cannot handle base16 hashes

Some hashes in nixpkgs are base16 instead of base32. The way to fix this is to stop downloading the old versions to get the hash, and just get it directly from the src attribute.

Aliased attr-path got used instead of the primary one

NixOS/nixpkgs#38330 (comment)

Move nixpkgs-update parts of repology-api here

Choose branch based on rebuilds

I like what you wrote here for updating packages, although it does create a lot of work for the people that need to merge it.

One improvement I would like to see is if you would check before opening such a PR how much needs to be rebuild, and based on that choose a branch. @GrahamcOfBorg does already a bit of this so you might want to coordinate with @grahamc how to solve this. Maybe @GrahamcOfBorg can be improved to modify the branch, in which case there is no need to make a change here.

Add warning about broken=true

Sometimes packages are set to broken=true, my bot ignores that, but we should warn in the PR text that someone should remove the broken=true part before merging.

Example: NixOS/nixpkgs#37654

Group related packages in a single PR

For example, the libdvdread and libdvdnav packages are always updated as a unit, so they should go into a single PR.

Don't update packages pinned to specific version

In NixOS/nixpkgs#37474, you tried to update libgit2_0_25 to version 0.26.3, which doesn't make sense.

[Ongoing] Latest nixpkgs-update/r-ryantm log

This issue will be updated on an ongoing basis.
2017-08-17 log of nixpkgs-update/r-ryantm run
2017-08-08 log of nixpkgs-update/r-ryantm run
2017-07-22 log of nixpkgs-update/r-ryantm run
2018-07-13 log of nixpkgs-update/r-ryantm run
2018-09-06 log of nixpkgs-update/r-ryantm run
2018-10-03 log of nixpkgs-update/r-ryantm run
2018-10-23 log of nixpkgs-update/r-ryantm run
2018-10-24 log of nixpkgs-update/r-ryantm python-only run
2018-11-08 log of nixpkgs-update/r-ryantm run

check version by running binary with no arguments

NixOS/nixpkgs#39761 (comment)

Don’t auto-update golang (npm, rust, …) packages

golang and npm use the concept of lockfiles, carrying all their files with them. Just changing the version string doesn’t cut it in these cases, the dependencies have to be updated as well (if the lockfile changes), for example NixOS/nixpkgs@413e41d#diff-78d6ebbd9a005053af6388888c66decf (in this case the lockfile luckily didn’t change).

The languages I know of that use lockfiles are:

• node (yarn.lock for yarn and package-lock.json for npm)
• rust (cargo.lock)
• golang (sometimes master, sometimes vendor.conf sometimes something different)

It could be automated, but requires some work. Until then I suggest we are conservative.

Setup automatic shellcheck linting

11:30 < foobar__> ryantm: nix-env -iA nixos.shellcheck && shellcheck *.sh # to be executed in your repository

Don't do work if origin has branch already

Only CC maintainers in PR

This requires teaching the scripts how to directly submit pull requests.

Rework binary checking

Trigger: NixOS/nixpkgs#40516

Let's rework binary checking. Instead of saying when things pass, let's warn them when they don't:

• Warn if there is a binary that does not have at least one of, non-zero exit code, responds to help, has the version in it's output
• Print how many binaries there are and how many have passed the above test
• Maybe make a table of binaries and which tests they passed?

Some better system for not submitting the same PR twice.

Right now there is a problem where I'm submitting the same PR twice. I could blacklist the packages that are being updated in a different way, but I'd prefer some way of ignoring a particular version, instead of blacklist them outright.

NixOS/nixpkgs#37195 (comment)

Pass "reviewers" to hub command

Hub should support passing reviewers in the latest version:

mislav/hub#1453

This would be useful to move the "maintainers" stuff here.

blacklist qscintilla

NixOS/nixpkgs#39860

False positive in checking version output by binary

NixOS/nixpkgs#37979 (comment)

Probably just requiring some whitespace around the version would fix this issue.

Make new github user for nix-update

rename scripts to more clear name

ups.sh -> nixpkgs-update.sh (like discussed in #13)
up.sh -> ???
to-staging.sh -> ?-to-staging.sh

blacklist checkbashisms

NixOS/nixpkgs#39552

Handle derivations using fetchgit

Try https url while updating

Don't try to update derivation files that contain multiple fetchers

Recently almost all the xorg.XXXX updates failed because the derivations are all in the same file and some have the same version number. For example: https://github.com/NixOS/nixpkgs/pull/37582/files

Let's skip updating files with multiple fetchers until we figure out how to do it better.

Use master and staging common ancestor as base

NixOS/nixpkgs#36721 (comment)

xcircuit needs repology rule

NixOS/nixpkgs#42418 (comment)

Make sure src URL contains a reference to version attribute

To avoid problems like https://github.com/NixOS/nixpkgs/pull/36702/files

Show the directory tree diff

It would be nice to see added and removed files in the directory tree.

Fully automate nix-update

Here's my master plan for nix-update:

• Integrate other package updaters #12
• Move "nix-update" to NixOS org
• Create a GitHub user that will author the automatic commits and pull requests (like with @GrahamcOfBorg : maybe @RyanTMOfBorg ?)
• Setup a cron job for nix-update that could run on some shared NixOS infrastructure (maybe the same machines that OfBorg runs on?). I don't want to overload the infrastructure but "updating" is usually less intensive than "building". /cc @grahamc

This is kind of a long term goal for Nixpkgs in general. This will hopefully increase the "bus factor" in Nixpkgs.

build results not found on cachix

When I try to use cachix (great feature!) as per instructions in r-ryantm's PRs, build results are not found on cachix. Accessing them directly via curl fails too.

I tried a few of r-ryantm's PRs and could not find any results on cachix.
Maybe the build results aren't pushed, maybe I'm doing something wrong here...

Example (nixpkgs PR #43028):

store path is /nix/store/hk60glvzvq8psbzxj585dy6q6dhyl0ig-eom-1.21.0 but

$ curl -i https://r-ryantm.cachix.org/hk60glvzvq8psbzxj585dy6q6dhyl0ig.narinfo
HTTP/1.1 404 Not Found

Expected result

similar to this (for cachix itself):

$ curl https://cachix.cachix.org/fgqfmzv1vzb60041sqwb1sprnw456v7r.narinfo
StorePath: /nix/store/fgqfmzv1vzb60041sqwb1sprnw456v7r-cachix-0.1.0.1
URL: nar/bb4f7c671e49346b1bbc12e4ffbc9415c56b8aae99cec6ca4e26176c2b2f3e46.nar.xz
Compression: xz
FileHash: sha256:bb4f7c671e49346b1bbc12e4ffbc9415c56b8aae99cec6ca4e26176c2b2f3e46
FileSize: 1862008
NarHash: sha256:0qd3ij4b0f8r1gdzghkk2q4imr4pxbz60dagqi1jb5vsp07y6ms4
NarSize: 14738784
References: 14nlrnm6bv7z25c9ismlhc2mrxsriyq2-gmp-6.1.2 6r6hk0qq5jd8n0sbfmpri0wd7icszi16-gcc-7.3.0-lib d0az5vfrk5cm65wcvxmfj2hjk613k9ni-xz-5.2.4 fr9a6l744pzn2nc68ayksl1v4400r27w-ncurses-6.1 gf00m2nz8079di7ihc6fj75v5jbh8p8v-zlib-1.2.11 pr73kx0cdszbv9iw49g8dzi0nqxfjbx2-glibc-2.27
Deriver: w8qqbm6mdnlka134x3iffnw3rg62l1l0-cachix-0.1.0.1.drv
Sig: cachix.cachix.org-1:KbOwKbc+zE5/MG+kTjNHjA789Rg5N0EZmACfaiX7kOLlfzhRpTjqNVVpq8fvPwOrvYUWsghLff8m8dSC9St4Ag==

Log of latest run (what parts we still need to work on getting better)

This might be helpful for people thinking about how to improve how much nix-update can do. Here is the log of the latest run and some statistics of errors encountered:

https://gist.github.com/ryantm/ee637b8e7a5d2c8c186d0aa58d587ad7

495 failed "could not prefetch new version source"
919 failed "nix-env -q failed to find package name with old version" means it couldn't find the derivation file to update
255 failed "nix build failed"

Handle other result output types, currently just handles result

Some nixpkgs pkgs not in repology?

NixOS/nixpkgs#38976

"@ryantm For unknown reason repology does not see the presence of this package in nixpkgs; this results in your bot skipping it. There could be more packages like this one"

add meta.description contents to Github PR comment

Gives context to us mergers. Sometimes the upgrade looks good, so I merge, but I have no idea what I'm merging...

Only make automated PRs when GrahamCOfBorg is unloaded

19:04 < gchristensen> https://events.nix.ci/stats.php monitor that evaluator.consumers.messages.in_progress <= 2 perhaps
19:04 < gchristensen> s/in_progress/waiting

Include output directory sizes in checks

NixOS/nixpkgs#40322 (comment)

Source URL got fixed incorrectly

https://github.com/NixOS/nixpkgs/pull/38061/files

Do work while waiting for 15 minutes

Include changelogs in the PR body

• For GitHub projects, we could use API
• For other projects we could diff the NEWS (CHANGELOG.md) files in the tarballs before and after the update, keeping the added lines.

Build in sandbox

NixOS/nixpkgs#39774 (comment)