nixpkgs-update
Please read the documentation.
Updating nixpkgs packages since 2018
License: Creative Commons Zero v1.0 Universal
Please read the documentation.
It would be really cool if we could integrate a few other package updaters into nix-update
. We have a couple of different manual package updaters being run regularly. Usually people are good at doing this regularly but it definitely increases Nixpkgs' bus factor. Automating this process through nix-update could be very helpful. Here's my initial list along with the maintainers:
Some other packages also have "update.nix" that might be possible to use. This was initially done by @garbas "update script" but I'm not sure if it's still actively being maintained.
With NixOS/nixpkgs#36119 closed, the maintainers.nix
now contains a complete set of github handles for (nearly) each maintainer. They are checked to be correct (see PR). Attention, some maintainers don’t have a github field.
It would be great to have the tool label the generated pull requests as "auto" or similar (in a nice color of course :-) Helps browse and filter the growing list of pull requests.
If nixpkgs-update was more aware of the PRs it generates it could have a lot of nice features:
Our new way of figuring out the derivation file is sometimes wrong. For example,
libgit2
picked the specific version instead of the non-specific one: "libgit2_0_25: 0.25.1 -> 0.26.3"
NixOS/nixpkgs#37474
Same for isl
"isl_0_15: 0.15 -> 0.19"
NixOS/nixpkgs#37453
OLD
and NEW
versions.OLD
and NEW
versions.Here's an example where Debian/Ubuntu used "19.0" for the version even though upstream says the version is "19":
NixOS/nixpkgs#43357
If the version from Repology doesn't work and contains trailing ".0"s we should try removing them one by one to see if that works.
For example in NixOS/nixpkgs#39860 (comment) , the same file is used for the qt4 and qt5 qscintilla derivations.
If we can detect expressions used by multiple derivations, we can avoid building them, and eventually figure out how to build/test both.
This might be helpful when maintainers aren't 100% sure if this is truly the "latest" version.
URL is:
Some hashes in nixpkgs are base16 instead of base32. The way to fix this is to stop downloading the old versions to get the hash, and just get it directly from the src attribute.
I like what you wrote here for updating packages, although it does create a lot of work for the people that need to merge it.
One improvement I would like to see is if you would check before opening such a PR how much needs to be rebuild, and based on that choose a branch. @GrahamcOfBorg does already a bit of this so you might want to coordinate with @grahamc how to solve this. Maybe @GrahamcOfBorg can be improved to modify the branch, in which case there is no need to make a change here.
Sometimes packages are set to broken=true, my bot ignores that, but we should warn in the PR text that someone should remove the broken=true part before merging.
Example: NixOS/nixpkgs#37654
For example, the libdvdread and libdvdnav packages are always updated as a unit, so they should go into a single PR.
In NixOS/nixpkgs#37474, you tried to update libgit2_0_25
to version 0.26.3, which doesn't make sense.
This issue will be updated on an ongoing basis.
2017-08-17 log of nixpkgs-update/r-ryantm run
2017-08-08 log of nixpkgs-update/r-ryantm run
2017-07-22 log of nixpkgs-update/r-ryantm run
2018-07-13 log of nixpkgs-update/r-ryantm run
2018-09-06 log of nixpkgs-update/r-ryantm run
2018-10-03 log of nixpkgs-update/r-ryantm run
2018-10-23 log of nixpkgs-update/r-ryantm run
2018-10-24 log of nixpkgs-update/r-ryantm python-only run
2018-11-08 log of nixpkgs-update/r-ryantm run
golang and npm use the concept of lockfiles, carrying all their files with them. Just changing the version string doesn’t cut it in these cases, the dependencies have to be updated as well (if the lockfile changes), for example NixOS/nixpkgs@413e41d#diff-78d6ebbd9a005053af6388888c66decf (in this case the lockfile luckily didn’t change).
The languages I know of that use lockfiles are:
yarn.lock
for yarn and package-lock.json
for npm)cargo.lock
)vendor.conf
sometimes something different)It could be automated, but requires some work. Until then I suggest we are conservative.
11:30 < foobar__> ryantm: nix-env -iA nixos.shellcheck && shellcheck *.sh # to be executed in your repository
This requires teaching the scripts how to directly submit pull requests.
Trigger: NixOS/nixpkgs#40516
Let's rework binary checking. Instead of saying when things pass, let's warn them when they don't:
Right now there is a problem where I'm submitting the same PR twice. I could blacklist the packages that are being updated in a different way, but I'd prefer some way of ignoring a particular version, instead of blacklist them outright.
Hub should support passing reviewers in the latest version:
This would be useful to move the "maintainers" stuff here.
Probably just requiring some whitespace around the version would fix this issue.
ups.sh -> nixpkgs-update.sh (like discussed in #13)
up.sh -> ???
to-staging.sh -> ?-to-staging.sh
Recently almost all the xorg.XXXX updates failed because the derivations are all in the same file and some have the same version number. For example: https://github.com/NixOS/nixpkgs/pull/37582/files
Let's skip updating files with multiple fetchers until we figure out how to do it better.
To avoid problems like https://github.com/NixOS/nixpkgs/pull/36702/files
It would be nice to see added and removed files in the directory tree.
Here's my master plan for nix-update:
This is kind of a long term goal for Nixpkgs in general. This will hopefully increase the "bus factor" in Nixpkgs.
When I try to use cachix (great feature!) as per instructions in r-ryantm's PRs, build results are not found on cachix. Accessing them directly via curl fails too.
I tried a few of r-ryantm's PRs and could not find any results on cachix.
Maybe the build results aren't pushed, maybe I'm doing something wrong here...
store path is /nix/store/hk60glvzvq8psbzxj585dy6q6dhyl0ig-eom-1.21.0
but
$ curl -i https://r-ryantm.cachix.org/hk60glvzvq8psbzxj585dy6q6dhyl0ig.narinfo
HTTP/1.1 404 Not Found
similar to this (for cachix itself):
$ curl https://cachix.cachix.org/fgqfmzv1vzb60041sqwb1sprnw456v7r.narinfo
StorePath: /nix/store/fgqfmzv1vzb60041sqwb1sprnw456v7r-cachix-0.1.0.1
URL: nar/bb4f7c671e49346b1bbc12e4ffbc9415c56b8aae99cec6ca4e26176c2b2f3e46.nar.xz
Compression: xz
FileHash: sha256:bb4f7c671e49346b1bbc12e4ffbc9415c56b8aae99cec6ca4e26176c2b2f3e46
FileSize: 1862008
NarHash: sha256:0qd3ij4b0f8r1gdzghkk2q4imr4pxbz60dagqi1jb5vsp07y6ms4
NarSize: 14738784
References: 14nlrnm6bv7z25c9ismlhc2mrxsriyq2-gmp-6.1.2 6r6hk0qq5jd8n0sbfmpri0wd7icszi16-gcc-7.3.0-lib d0az5vfrk5cm65wcvxmfj2hjk613k9ni-xz-5.2.4 fr9a6l744pzn2nc68ayksl1v4400r27w-ncurses-6.1 gf00m2nz8079di7ihc6fj75v5jbh8p8v-zlib-1.2.11 pr73kx0cdszbv9iw49g8dzi0nqxfjbx2-glibc-2.27
Deriver: w8qqbm6mdnlka134x3iffnw3rg62l1l0-cachix-0.1.0.1.drv
Sig: cachix.cachix.org-1:KbOwKbc+zE5/MG+kTjNHjA789Rg5N0EZmACfaiX7kOLlfzhRpTjqNVVpq8fvPwOrvYUWsghLff8m8dSC9St4Ag==
This might be helpful for people thinking about how to improve how much nix-update can do. Here is the log of the latest run and some statistics of errors encountered:
https://gist.github.com/ryantm/ee637b8e7a5d2c8c186d0aa58d587ad7
495 failed "could not prefetch new version source"
919 failed "nix-env -q failed to find package name with old version" means it couldn't find the derivation file to update
255 failed "nix build failed"
"@ryantm For unknown reason repology does not see the presence of this package in nixpkgs; this results in your bot skipping it. There could be more packages like this one"
Gives context to us mergers. Sometimes the upgrade looks good, so I merge, but I have no idea what I'm merging...
19:04 < gchristensen> https://events.nix.ci/stats.php monitor that evaluator.consumers.messages.in_progress <= 2 perhaps
19:04 < gchristensen> s/in_progress/waiting
NEWS
(CHANGELOG.md
) files in the tarballs before and after the update, keeping the added lines.A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.