lamby / buildinfo.debian.net

Proof-of-concept .buildinfo server

bidb / buildinfo.debian.net

Local database setup

• Create PostgreSQL user with id matching your UNIX username:

  $ sudo -u postgres createuser $(whoami) -SDR

• Create a database owned by this user:

  $ sudo -u postgres createdb -E UTF-8 -O $(whoami) bidb

• Run any initial migrations:

  $ ./manage.py migrate

---

Hardware sponsored by:

buildinfo.debian.net's Issues

Internal Server Error on submission

https://jenkins.debian.net/job/reproducible_builder_amd64_1/54380/console

Get GPG signatures on all Jenkins nodes

API to fetch specific .buildinfo files for a certain package/arch/builder

h01ger: lamby: say i want the .buildinfo file for zsh 5.4.2-1 on amd64, how can i get that from buildinfo.debian.net?
lamby: h01ger: https://buildinfo.debian.net/sources/zsh/5.4.2-1 ?
lamby: Note that there isn't "the" .buildinfo - there are (potentially) multiple ones.
lamby: So there isn't a canonical URL for a (src, version, buildarch) tuple that leads to a single buildinfo, it's more (src, ver, arch) → [.buildinfo]
h01ger: | lamby: i think we need such an API, or do you expect consumers (=tools) to scrape those webpages?
| h01ger: Obviously avoiding scraping. Please file a wishlist bug if you want an API endpoint for that :)

502 on binaries pages

I'm geetting a "502 - Bad Gateway" when I click on a build-dependency on a buildinfo page.
For example this page returns a 502: https://buildinfo.debian.net/binaries/autoconf

Rejecting submission: Could not determine GPG uid

I've been manually submitting .buildinfo files for packages I've uploaded to Debian. But after recent updates to my key, it appears to no longer accept the signatures:

$ cat ../u-boot_2018.09+dfsg-1_amd64.buildinfo | curl -X PUT --max-time 30 --data-binary @-
https://buildinfo.debian.net/api/submit
Rejecting submission: Could not determine GPG uid

My guess is either it has an outdated keyring, or does not support ed25519 signatures, or potentially both.

Thanks for maintaining buildinfo.debian.net!

Attached is the submitted .buildinfo (compressed, to make github happy).
u-boot_2018.09+dfsg-1_amd64.buildinfo.0.gz

live well,
vagrant

Add a (source, version) page showing reproduciblity status

Queue incoming .buildinfo to avoid accidental DDoS?

Add support for .buildinfo 1.0 ("Only Format: 0.2 is supported")

I wanted to submit a buildinfo file from my Debian unstable system, where the buildinfo format version has been bumped to 1.0, but the server was rejecting it:

Rejecting submission: Only Format: 0.2 is supported

Strategy for large number of queries

https://jenkins.debian.net/job/reproducible_compare_Debian_sha1sums/ now downloads all the .buildinfo files for all packages in sid (with the versions in sid), each.

It would be really cool if there was a way to download them all as one big .tgz or whatever.

Move to salsa.debian.org

Hi,

once this is less of a PoC I think this should life together with our other repositories which currently can be found at https://anonscm.debian.org/git/reproducible

Thanks a lot for working on this prototype!

Duplicate submissions are accepted

.Buildinfo submissions that have the exact same hash are accepted (and thus pollute the database with duplicate entries). Rejecting duplicate submissions (probably by making the hash of the buildinfo file by a primary key) may be in order.

Parses all building files as having a source version 1.0 regardless of actual version

Presumably due to the update of the .buildinfo file format, buildinfo.debian.net is somehow parsing the "Format" field as the "Version" field, and all .buildinfo files are registering as version "1.0". Just grabbing the first three examples from https://buildinfo.debian.net at the moment:

stress-ng_1.0_arm64.buildinfo from 0xAAA3AE5F86A5D0BA
stress-ng_1.0_arm64.buildinfo from 0x1D3EA4D86F2FB555
libcatalyst-model-adaptor-perl_1.0_all.buildinfo from 0xFAC0B94FFF2617A2 cbxi4b (Automatically generated key for signing .buildinfo files)
libcatalyst-model-adaptor-perl_1.0_all.buildinfo from 0xC7E7F3E4B48B3CE6 opi2b (Automatically generated key for signing .buildinfo files)
envstore_1.0_amd64.buildinfo from 0x1BD40F376E003684 profitbricks-build5-amd64 (Automatically generated key for signing .buildinfo files)

ability to search for buildinfo's by hash

As a correlary to:

https://buildinfo.debian.net/binaries/u-boot
https://buildinfo.debian.net/sources/u-boot

Where you can search for buildinfo files based on binary package or source package, it would be nice if I could search for all buildinfo's containing certain hashes:

e.g. https://buildinfo.debian.net/by-hash/c9070da17f8dd59c9c09698add69453b7c78270b

It would either autodetect the hash type based on length, or possibly an alternate URL form to specify the hash type:

https://buildinfo.debian.net/by-hash/sha1/c9070da17f8dd59c9c09698add69453b7c78270b

This should produce a list of links to .buildinfo that contain the hash of the binary packages (.deb) or "other checksums" (.dsc, .orig.tar., debian.tar.). The above example links should provide a link to the following buildinfo, and possibly others:

https://buildinfo.debian.net/71a17701cd1b1b49dcbee51ae364d3bf8d6f7c10/u-boot_2016.09+dfsg1-2_amd64

The output produced by the binary package view "Generated by source packages" section should be sufficient to convey the desired information. There's some theoretical possibility that differing source/binary packages would produce the same hashes, but it doesn't seem very likely... but would be very interesting to find if that were so!

Hopefully that's enough to go on!

Refactor schema to remove optimisations for duplicate submissions

Batch uploads/import

I've started uploading all of the .buildinfo files available on ftp.debian.org, but I can't help but wondering if batching such uploads would be able to be time, network and space efficient if it were possible to upload multiple .buildinfo files in a single http call or an upload widget or something similar.

Compression of batches of .buildinfo files tends to be quite high, so being able to submit a tarball of compressed .buildinfo files would both reduce network usage as well as the transaction costs of multiple http calls.

"Only Format: 0.1 is supported"

Sun 04 17:16 <         h01ger > lamby: i also see warnings when sending buildinfos to b.d.n: "Only Format: 0.1 is supported"

certificate on buildinfo.debian.net expired

The certificate for buildinfo.debian.net expired on 2017-09-11, or at least, that's what my web browser tells me.

Reject files with duplidate Installed-Build-Depends

Move to salsa.debian.org

Only accept signed .buildinfo files

add a way to query for .buildinfo files from official debian builds only

Hi,

thanks to Vagrant's 'hack' we now have .buildinfo files from official Debian builders uploaded to buildinfo.debian.net. (We still miss some (eg from security builds) and we should fix the hack but thats besides the point here.)

Now it would be nice to have an easy way to query for those .buildinfo files only (and ignore eg those .buildinfo files coming from tests.r-b.o) and easily (AFAIK it's possible to query by signing key but there are many different signing keys used by the Debian autobuilders, so that's not easy.)

As a consumer/rebuilder I need a way to say: gimme all/this .buildinfo file(s) which ended up in the Debian archive, without really knowing which keys were used to signed...

Hope this makes sense, else I'm happy to clarify.

report stronger checksums on web interface

As far as I can tell, the web interface reports the sha1 checksum of files, even if .buildinfo files have stronger checksums:

https://buildinfo.debian.net/60ab1ce26f749f685acf39fac7a804a2fe3baa9f/ruby-httpauth_0.2.1+gh-1_all

The Debian archive only supports MD5 and SHA256 checksums (at least in sid), so ideally at least the default would be SHA256 if available in the .buildinfo, to make it easier to compare a one-off package against what's in the archive.

Thanks for buildinfo.debian.net!

live well,
vagrant

Find more efficient storage technique for .buildinfo files.

Cache expensive query on homepage

Compress .buildinfo files

Store original .buildinfo files on S3?

Make node an instance, rather than pure metadata? (Or drop now we are signing...)

Include .buildinfo files from ftp.debian.org

hi,
for this to become more meaningful, .buildinfo files from packages from the main Debian archive should be fed into this service. Let's discuss how to achieve this!

Add DigitalOcean attribution

check for match against debian archive

Wild off-the-top-of-my head wishlist item here:

buildinfo.debian.net appears to check if there are buildinfo files that successfully reproduced the binaries:

https://buildinfo.debian.net/sources/u-boot/2018.07+dfsg-1

It would be interesting if it could compare the produced hashes against the in-archive packages, and see which .buildinfo files match, and ideally expose matching in-archive .deb with some api, so someone could ask to explicitly rebuild against the archive by querying buildinfo.debian.net.

I'm sure this requires a fair amount of additional parsing; e.g. downloading all the Packages files for target architectures on a regular basis, and then updating the whole database.

Maybe it's infeasible, or the wrong place to do this sort of thing.

The main advantage to this approach is it would allow to retroactively provide a database of .buildinfo files that match the in-archive files once we get around to publicly publishing the .buildinfo files that are currently uploaded to the official Debian archive... and also .buildinfo files that happened to match the archive from our test infrastructure.

Don't assume that all checkums are .deb packages

Several packages builds something that is not a .deb, going through the pretty common .udeb, to more exotic files. Please accept them too instead of failing with an error on the POST.

API endpoint for getting all new buildinfos since a particular timestamp

yashsriv> lamby h01ger : I'd like your views on this as you weren't present
during the meeting [1]: I need to schedule builds on independent infrastructure
so as to verify the builds independently. A solution to that was to use
buildinfo.d.n and trigger builds for every new entry. How would I poll for new
entries from buildinfo.d.n ... I would ideally prefer something like - all new
buildinfo files since timestamp x .

yashsriv> More context: [1] - http://meetbot.debian.net/reproducible-builds/2018/reproducible-builds.2018-06-19-16.01.log.html
#reproducible-builds log

lamby> yashsriv: Could even do a push API depending on what you need

lamby> yashsriv: But a "since" filter could work. Can you file a wishlist request? :) 

.onion service

please default to showing sha256 hashes

Currently, it looks like buildinfo.debian.net defaults to exposing sha1 hashes:

https://buildinfo.debian.net/f5c517c5f165b048b70158666f98ecf42cefb0f0/u-boot_2017.09+dfsg1-3_armhf

But it would be nicer to display sha256, which I think is the strongest hash that dpkg-buildpackage generates by default, is what is exposed in the Packages files in the debian.org package archives.

Thanks!