lamby / buildinfo.debian.net Goto Github PK
View Code? Open in Web Editor NEWProof-of-concept .buildinfo server
Home Page: http://buildinfo.debian.net
License: GNU Affero General Public License v3.0
Proof-of-concept .buildinfo server
Home Page: http://buildinfo.debian.net
License: GNU Affero General Public License v3.0
I'm geetting a "502 - Bad Gateway" when I click on a build-dependency on a buildinfo page.
For example this page returns a 502: https://buildinfo.debian.net/binaries/autoconf
As a correlary to:
https://buildinfo.debian.net/binaries/u-boot
https://buildinfo.debian.net/sources/u-boot
Where you can search for buildinfo files based on binary package or source package, it would be nice if I could search for all buildinfo's containing certain hashes:
e.g. https://buildinfo.debian.net/by-hash/c9070da17f8dd59c9c09698add69453b7c78270b
It would either autodetect the hash type based on length, or possibly an alternate URL form to specify the hash type:
https://buildinfo.debian.net/by-hash/sha1/c9070da17f8dd59c9c09698add69453b7c78270b
This should produce a list of links to .buildinfo that contain the hash of the binary packages (.deb) or "other checksums" (.dsc, .orig.tar., debian.tar.). The above example links should provide a link to the following buildinfo, and possibly others:
https://buildinfo.debian.net/71a17701cd1b1b49dcbee51ae364d3bf8d6f7c10/u-boot_2016.09+dfsg1-2_amd64
The output produced by the binary package view "Generated by source packages" section should be sufficient to convey the desired information. There's some theoretical possibility that differing source/binary packages would produce the same hashes, but it doesn't seem very likely... but would be very interesting to find if that were so!
Hopefully that's enough to go on!
Hi,
thanks to Vagrant's 'hack' we now have .buildinfo files from official Debian builders uploaded to buildinfo.debian.net. (We still miss some (eg from security builds) and we should fix the hack but thats besides the point here.)
Now it would be nice to have an easy way to query for those .buildinfo files only (and ignore eg those .buildinfo files coming from tests.r-b.o) and easily (AFAIK it's possible to query by signing key but there are many different signing keys used by the Debian autobuilders, so that's not easy.)
As a consumer/rebuilder I need a way to say: gimme all/this .buildinfo file(s) which ended up in the Debian archive, without really knowing which keys were used to signed...
Hope this makes sense, else I'm happy to clarify.
.Buildinfo submissions that have the exact same hash are accepted (and thus pollute the database with duplicate entries). Rejecting duplicate submissions (probably by making the hash of the buildinfo file by a primary key) may be in order.
Several packages builds something that is not a .deb, going through the pretty common .udeb, to more exotic files. Please accept them too instead of failing with an error on the POST.
h01ger: lamby: say i want the .buildinfo file for zsh 5.4.2-1 on amd64, how can i get that from buildinfo.debian.net?
lamby: h01ger: https://buildinfo.debian.net/sources/zsh/5.4.2-1 ?
lamby: Note that there isn't "the" .buildinfo - there are (potentially) multiple ones.
lamby: So there isn't a canonical URL for a (src, version, buildarch) tuple that leads to a single buildinfo, it's more (src, ver, arch) โ [.buildinfo]
h01ger: | lamby: i think we need such an API, or do you expect consumers (=tools) to scrape those webpages?
| h01ger: Obviously avoiding scraping. Please file a wishlist bug if you want an API endpoint for that :)
https://jenkins.debian.net/job/reproducible_compare_Debian_sha1sums/ now downloads all the .buildinfo files for all packages in sid (with the versions in sid), each.
It would be really cool if there was a way to download them all as one big .tgz or whatever.
Presumably due to the update of the .buildinfo file format, buildinfo.debian.net is somehow parsing the "Format" field as the "Version" field, and all .buildinfo files are registering as version "1.0". Just grabbing the first three examples from https://buildinfo.debian.net at the moment:
stress-ng_1.0_arm64.buildinfo from 0xAAA3AE5F86A5D0BA
stress-ng_1.0_arm64.buildinfo from 0x1D3EA4D86F2FB555
libcatalyst-model-adaptor-perl_1.0_all.buildinfo from 0xFAC0B94FFF2617A2 cbxi4b (Automatically generated key for signing .buildinfo files)
libcatalyst-model-adaptor-perl_1.0_all.buildinfo from 0xC7E7F3E4B48B3CE6 opi2b (Automatically generated key for signing .buildinfo files)
envstore_1.0_amd64.buildinfo from 0x1BD40F376E003684 profitbricks-build5-amd64 (Automatically generated key for signing .buildinfo files)
Wild off-the-top-of-my head wishlist item here:
buildinfo.debian.net appears to check if there are buildinfo files that successfully reproduced the binaries:
https://buildinfo.debian.net/sources/u-boot/2018.07+dfsg-1
It would be interesting if it could compare the produced hashes against the in-archive packages, and see which .buildinfo files match, and ideally expose matching in-archive .deb with some api, so someone could ask to explicitly rebuild against the archive by querying buildinfo.debian.net.
I'm sure this requires a fair amount of additional parsing; e.g. downloading all the Packages files for target architectures on a regular basis, and then updating the whole database.
Maybe it's infeasible, or the wrong place to do this sort of thing.
The main advantage to this approach is it would allow to retroactively provide a database of .buildinfo files that match the in-archive files once we get around to publicly publishing the .buildinfo files that are currently uploaded to the official Debian archive... and also .buildinfo files that happened to match the archive from our test infrastructure.
hi,
for this to become more meaningful, .buildinfo files from packages from the main Debian archive should be fed into this service. Let's discuss how to achieve this!
yashsriv> lamby h01ger : I'd like your views on this as you weren't present
during the meeting [1]: I need to schedule builds on independent infrastructure
so as to verify the builds independently. A solution to that was to use
buildinfo.d.n and trigger builds for every new entry. How would I poll for new
entries from buildinfo.d.n ... I would ideally prefer something like - all new
buildinfo files since timestamp x .
yashsriv> More context: [1] - http://meetbot.debian.net/reproducible-builds/2018/reproducible-builds.2018-06-19-16.01.log.html
#reproducible-builds log
lamby> yashsriv: Could even do a push API depending on what you need
lamby> yashsriv: But a "since" filter could work. Can you file a wishlist request? :)
As far as I can tell, the web interface reports the sha1 checksum of files, even if .buildinfo files have stronger checksums:
https://buildinfo.debian.net/60ab1ce26f749f685acf39fac7a804a2fe3baa9f/ruby-httpauth_0.2.1+gh-1_all
The Debian archive only supports MD5 and SHA256 checksums (at least in sid), so ideally at least the default would be SHA256 if available in the .buildinfo, to make it easier to compare a one-off package against what's in the archive.
Thanks for buildinfo.debian.net!
live well,
vagrant
Sun 04 17:16 < h01ger > lamby: i also see warnings when sending buildinfos to b.d.n: "Only Format: 0.1 is supported"
Currently, it looks like buildinfo.debian.net defaults to exposing sha1 hashes:
https://buildinfo.debian.net/f5c517c5f165b048b70158666f98ecf42cefb0f0/u-boot_2017.09+dfsg1-3_armhf
But it would be nicer to display sha256, which I think is the strongest hash that dpkg-buildpackage generates by default, is what is exposed in the Packages files in the debian.org package archives.
Thanks!
Hi,
once this is less of a PoC I think this should life together with our other repositories which currently can be found at https://anonscm.debian.org/git/reproducible
Thanks a lot for working on this prototype!
The certificate for buildinfo.debian.net expired on 2017-09-11, or at least, that's what my web browser tells me.
I've started uploading all of the .buildinfo files available on ftp.debian.org, but I can't help but wondering if batching such uploads would be able to be time, network and space efficient if it were possible to upload multiple .buildinfo files in a single http call or an upload widget or something similar.
Compression of batches of .buildinfo files tends to be quite high, so being able to submit a tarball of compressed .buildinfo files would both reduce network usage as well as the transaction costs of multiple http calls.
I wanted to submit a buildinfo file from my Debian unstable system, where the buildinfo format version has been bumped to 1.0, but the server was rejecting it:
Rejecting submission: Only Format: 0.2 is supported
I've been manually submitting .buildinfo files for packages I've uploaded to Debian. But after recent updates to my key, it appears to no longer accept the signatures:
$ cat ../u-boot_2018.09+dfsg-1_amd64.buildinfo | curl -X PUT --max-time 30 --data-binary @-
https://buildinfo.debian.net/api/submit
Rejecting submission: Could not determine GPG uid
My guess is either it has an outdated keyring, or does not support ed25519 signatures, or potentially both.
Thanks for maintaining buildinfo.debian.net!
Attached is the submitted .buildinfo (compressed, to make github happy).
u-boot_2018.09+dfsg-1_amd64.buildinfo.0.gz
live well,
vagrant
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.