debian / ratt

Rebuild All The Things!

README

Debian packages, maintains and distributes many projects developed using GitHub. This account was created to facilitate push/pull interactions with the upstream developers of such projects. If you maintain a package whose upstream developers use GitHub, please feel free to join this group and mirror such project here.

This account is not intended to serve as the canonical (specified with Vcs-* fields of debian/control) location for corresponding Debian source packages. Most often such repositories should be made available on the Debian project's public forge Salsa to guarantee autonomy.

How to join

Open a new issue with a signed statement asking to be added to the organization. The signature needs to be made with your PGP key currently in the Debian keyring. All active Debian Developers will be approved.

Tips

You might find following tools available from Debian useful for your interaction with GitHub

github-backup backs up everything GitHub knows about a repository, to the repository

Acknowledgements

Many thanks to the GitHub admins for their prompt action to release the previous (unused) "Debian" account.

Disclaimers

This GitHub organization is not an endorsement of GitHub by Debian. Debian does not maintain or distribute the GitHub engine codebase because it is not available under free and open-source license (see Wikipedia for a list of available free and open-source alternatives). Moreover, this GitHub organization is not an official part of the Debian project. It is maintained by individual Debian developers (signed below) with the sole purpose of being useful.

-- Charles Plessy [email protected] Thu, 14 Jun 2012 09:11:55 +0900

-- Yaroslav Halchenko [email protected] Thu, 14 Jun 2012 13:22:03 -0400

ratt's Issues

Support rebuilding against multiple new packages

The user case is, some packages should be bumped up version together.
For example, I want to bump up golang-github-azure-azure-sdk-for-go, but it requires bump up golang-github-azure-go-autorest too.
If I just test golang-github-azure-go-autorest first, some packages fail, because of the incompatible of old golang-github-azure-azure-sdk-for-go and new golang-github-azure-go-autorest.That causes a lot noise.
So I hope ratt can support rebuilding multiple packages.

diff build package against archive version

What about optionally diffing the generated packages against the version in the archive, so we can detect needed rebuilds as well? Useful tools would be:

• diffoscope
• abipkgdiff
• Pkg-ABIdiff (not yet in Debian)

I could work in this, it it's wanted.

go get fails

Trying to get ratt going on Debian/jessie. I tried:

% go get -u github.com/Debian/ratt
package github.com/Debian/ratt
    imports pault.ag/go/debian/control: unrecognized import path "pault.ag/go/debian/control"
package github.com/Debian/ratt
    imports pault.ag/go/debian/version: unrecognized import path "pault.ag/go/debian/version"

At looks like https://pault.ag/go/debian is returning 502 Bad Gateway errors.

Error: EOF

Took me a while to realize this was an error.

Seems to occur while parsing /var/lib/apt/lists/ftp.au.debian.org_debian_dists_experimental_InRelease

2015/10/22 14:12:38 Loading changes file "django-filter_0.11.0-1_amd64.changes"
2015/10/22 14:12:38  - 5 binary packages: python-django-filters python3-django-filters python-django-filters-doc django-filter python-django-filter
2015/10/22 14:12:38  - corresponding .debs (will be injected when building):
2015/10/22 14:12:38     django-filter_0.11.0-1_all.deb
2015/10/22 14:12:38     python-django-filter_0.11.0-1_all.deb
2015/10/22 14:12:38     python-django-filters-doc_0.11.0-1_all.deb
2015/10/22 14:12:38     python-django-filters_0.11.0-1_all.deb
2015/10/22 14:12:38     python3-django-filters_0.11.0-1_all.deb
2015/10/22 14:12:38 EOF

This is apt 1.0.9.8.1 from Jessie.

'go get' for installing executables is deprectated

following the instructions in the README on debian sid with go1.18.4, go get -u github.com/Debian/ratt returns:

go: go.mod file not found in current directory or any parent directory.
        'go get' is no longer supported outside a module.
        To build and install a command, use 'go install' with a version,
        like 'go install example.com/cmd@latest'
        For more information, see https://golang.org/doc/go-get-install-deprecation
        or run 'go help get' or 'go help install'.

Support rebuilding packages in other distributions than sid

Sometimes when I want to push a new upstream bugfix version to stable, I want to be able to prove to release managers that it doesn't break any other package and showing a successful rebuild is a good start. But then I need to be able to rebuild in a distribution which is not sid...

Support Different Source Paths

Right now, there's a dependency on /var/lib/apt/lists/ which means a dependency on the system apt. When I run apt-get on the system, I have to pull down a lot of sources that I probably don't need at the time... unless I'm running "sudo apt-get update" just so I can run ratt without builds failing because sources can't be found.

I can understand why the design decision was made in the first place, but it'd nice if we would extend ratt to read a custom sources.list and maintain (and optionally update) it. If nothing else, an option top point at a separate directory for these files so that I can write a script to maintain them separately.. that also makes sense. :)

use chdist grep-dctrl-sources command instead of dose-ceve

for example for doxygen:

time chdist grep-dctrl-sources unstable -F Build-Depends-Indep -F Build-Depends -r doxygen -n -s Package | sort | uniq
    735     735    7288

real    0m0,194s
user    0m0,179s
sys     0m0,017s

for comparison, dose.ceve finds 738 in 4 minutes (!); whereas for texinfo source:

time (chdist grep-dctrl-sources unstable -F Build-Depends-Indep -F Build-Depends -r texinfo -n -s Package && chdist grep-dctrl-sources unstable -F Build-Depends-Indep -F Build-Depends -r info -n -s Package && chdist grep-dctrl-sources unstable -F Build-Depends-Indep -F Build-Depends -r install-info-dbgsym -n -s Package && chdist grep-dctrl-sources unstable -F Build-Depends-Indep -F Build-Depends -r install-info -n -s Package && chdist grep-dctrl-sources unstable -F Build-Depends-Indep -F Build-Depends -r info-dbgsym -n -s Package && chdist grep-dctrl-sources unstable -F Build-Depends-Indep -F Build-Depends -r texinfo-dbgsym -n -s Package) | sort -u | wc
   1582    1582   23026

real    0m0,619s
user    0m0,581s
sys     0m0,045s

for this dose-ceve returns 30676 packages (!) in 6 minutes 30 seconds

also see: https://salsa.debian.org/ruby-team/meta/-/blob/master/build-and-upload#L150

partial rebuilds: filtering packages with regular expressions

in some cases there can be a lot of build deps (i.e. doxygen: 700+); after the first ratt run, it is sometimes useful to focus on a few packages; so it would be nice to be able to invoke ratt as in:

ratt -recheck -include '^(hwloc|fltk1.3|wcslib|ccfits|qevercloud|libstxxl|caffe|frobby|starpu)$' ../doxygen_1.8.15-1_amd64.changes

report wallclock time required to build each package

this would be useful to create a record track of previous ratt runs for a package that has quite a few build dependencies

JSON input and output

It would be handy to cache the output of dose-ceve (see the TODO "Cache this output ..." at line 148).
Also, getting the ratt run results in JSON format opens the way for a few interesting use cases, including ratt-as-a-service.

The output JSON format would be such that with some basic filtering it can be used to feed ratt with the list of packages to build.
For example when passed the -jsonout ratt.json option, ratt would output the following JSON file:

{
  "start": 2147483647,
  "end": 2147483647,
  "changes": "../texinfo_6.7.0.dfsg.2-5_amd64.changes",
  "debs": ["../info-dbgsym_6.7.0.dfsg.2-5_amd64.deb",
    "../info_6.7.0.dfsg.2-5_amd64.deb",
    "../install-info-dbgsym_6.7.0.dfsg.2-5_amd64.deb",
    "../install-info_6.7.0.dfsg.2-5_amd64.deb",
    "../texinfo-dbgsym_6.7.0.dfsg.2-5_amd64.deb",
    "../texinfo_6.7.0.dfsg.2-5_amd64.deb"
  ],
  "built": [{
      "package": "kdbusaddons",
      "status": "passed",
      "time": 1234
    },
    {
      "package": "libltcsmpte",
      "status": "failed",
      "time": 567
    },
    {
      "package": "libevhtp",
      "status": "maybe",
      "time": 890
    },
    {
      "package": "libtheora",
      "status": "passed",
      "time": 123
    }
  ]
}

This file can be easily filtered with jq, as in:

cat ratt.json | jq '[.built | .[] | select(.time < 600) | .package]'
cat ratt.json | jq '[.built | .[] | select(.status == "failed") | .package]'

obtaining for example:

[
  "libltcsmpte",
  "libtheora"
]

This format would be suitable for input, i.e. when ratt is started with -jsonin ratt1.json passing the above file, it would skip invoking dose-ceve and start rebuilding the libltcsmpte and libtheora packages straightaway.

Verify failures by rebuilding without the extra package

This is the output of a recent test run:

2017/07/19 19:45:38 PASSED: golang-github-rsc-letsencrypt_0.0~git20160929.0.76104d2-2
2017/07/19 19:45:38 PASSED: cadvisor_0.25.0+dfsg-1
2017/07/19 19:45:38 PASSED: golang-github-go-chef-chef_0.0.1+git20161023.60.deb8c38-1
2017/07/19 19:45:38 PASSED: swarmkit_1.12.0+git20170201.779.1c7f003d-1
2017/07/19 19:45:38 PASSED: rkt_1.21.0+dfsg-1
2017/07/19 19:45:38 PASSED: fuji_1.0.2-1
2017/07/19 19:45:38 PASSED: docker-registry_2.6.0~rc.1+git20161216.38.28602af3-1
2017/07/19 19:45:38 PASSED: kubernetes-addon-heapster_1.2.0+dfsg-1
2017/07/19 19:45:38 PASSED: golang-github-aanand-compose-file_0.0~git20161122.0.a3e5876-1
2017/07/19 19:45:38 PASSED: golang-github-appc-docker2aci_0.14.0+dfsg-2
2017/07/19 19:45:38 PASSED: golang-github-aws-aws-sdk-go_1.1.14+dfsg-2
2017/07/19 19:45:38 PASSED: acbuild_0.4.0+dfsg-1
2017/07/19 19:45:38 PASSED: golang-github-hashicorp-go-getter_0.0~git20160316.0.575ec4e-1
2017/07/19 19:45:38 PASSED: gitlab-ci-multi-runner_1.11.4+dfsg-1
2017/07/19 19:45:38 PASSED: goiardi_0.11.5-1
2017/07/19 19:45:38 PASSED: golang-github-docker-engine-api_0.4.0-1
2017/07/19 19:45:38 FAILED: notary_0.1~ds1-1 (see buildlogs/notary_0.1~ds1-1)
2017/07/19 19:45:38 FAILED: kubernetes_1.5.5+dfsg-2 (see buildlogs/kubernetes_1.5.5+dfsg-2)
2017/07/19 19:45:38 FAILED: golang-github-xenolf-lego_0.3.1-4 (see buildlogs/golang-github-xenolf-lego_0.3.1-4)
2017/07/19 19:45:38 FAILED: nomad_0.4.0+dfsg-1 (see buildlogs/nomad_0.4.0+dfsg-1)
2017/07/19 19:45:38 FAILED: docker.io_1.13.1~ds1-2 (see buildlogs/docker.io_1.13.1~ds1-2)
2017/07/19 19:45:38 FAILED: grafana_2.6.0+dfsg-3 (see buildlogs/grafana_2.6.0+dfsg-3)
2017/07/19 19:45:38 FAILED: prometheus_1.6.3+ds-1 (see buildlogs/prometheus_1.6.3+ds-1)
2017/07/19 19:45:38 FAILED: rclone_1.36-1 (see buildlogs/rclone_1.36-1)
2017/07/19 19:45:38 FAILED: docker-swarm_1.2.5+dfsg-2 (see buildlogs/docker-swarm_1.2.5+dfsg-2)
2017/07/19 19:45:38 FAILED: packer_0.10.2+dfsg-6 (see buildlogs/packer_0.10.2+dfsg-6)

Notably, all of the failures are unrelated to the new package, but this isn’t conveyed to the user.

As a first step, we should verify failures by running sbuild without --extra-package and marking a package as already-failing if so.

Another idea is to track these failures somewhere and have ratt query packages for known build failures.

find reverse dependencies for distributions other then unstable

Sometimes I would like to be able to find all reverse dependencies, say from wheezy, and rebuild them.

AFAIK Currently you can rebuild packages on wheezy, but this will cause errors if unstable has more packages then what was available on wheezy.

Support Parallel Builds

It'd be nice if ratt supported parallel builds... and even nicer if it also supported limiting number of processes available to sbuild. It seems like something with sync.WaitGroup would work well to support building up to X sbuilds.

I have a new server and an ambition to do efficient regression testing against a lot of packages and being able to efficiently abuse it would be helpful. :)

Support pbuilder

It doesn't look like it should be a very large change to support pbuilider/cowbuilder. To be honest, I'd never once touched sbuilder until I was trying to use ratt. For now, I'm going to learn a little about sbuilder, but in the long-term, I want to make ratt also support pbuilder.