debian / britney2 Goto Github PK

Sometimes the user might forget a file, data/yakkety/state/age-policy-dates for example. Instead of throwing FileNotFoundError and just freezing (I can Ctrl + C of course), it should create the file.

Exception handling should be better integrated so this type of error is less likely to ever happen.

For the freeze it would be great if we had a block-all new-source hint. It would block all source packages that are not already in testing (but existing ones would still be able to migrate)

Hi!
This is a slightly different form of the issue we recently discussed on IRC, but it likely has the same cause: arch:all package being available in different versions in arm64 vs amd64 for some reason.

Britney crashes with the following error:

I: [2018-06-06T19:54:54+0000] -   most: (92) .. kicad-templates kimagemapeditor -python-msrestazure kicad-footprints kicad-packages3d kopano-webapp libgitlab-api-v4-perl cross-toolchain-base-ports emacs-jabber pxz fonts-sawarabi-gothic ruby-sidekiq-limit-fetch golang-github-go-ini-ini apache-directory-server python-certbot-dns-digitalocean stenographer t4kcommon libsdl1.2 emacs-ctable refpolicy
I: [2018-06-06T19:54:54+0000] - trying: r-cran-bayesplot
I: [2018-06-06T19:54:54+0000] - skipped: r-cran-bayesplot (41, 0, 10)
I: [2018-06-06T19:54:54+0000] -     got: 17398+0: a-14680:a-2718
I: [2018-06-06T19:54:54+0000] -     * amd64: r-cran-bayesplot
I: [2018-06-06T19:54:54+0000] - trying: neutron
I: [2018-06-06T19:54:54+0000] - skipped: neutron (41, 1, 9)
I: [2018-06-06T19:54:54+0000] -     got: 17398+0: a-14680:a-2718
I: [2018-06-06T19:54:54+0000] -     * amd64: neutron-plugin-openvswitch-agent
I: [2018-06-06T19:54:54+0000] - trying: selinux-dbus selinux-python

Traceback (most recent call last):
  File "/srv/laniakea/dist/britney2/britney.py", line 2857, in <module>
    Britney().main()
  File "/srv/laniakea/dist/britney2/britney.py", line 2846, in main
    self.upgrade_testing()
  File "/srv/laniakea/dist/britney2/britney.py", line 2493, in upgrade_testing
    self.do_all()
  File "/srv/laniakea/dist/britney2/britney.py", line 2359, in do_all
    (nuninst_end, extra) = self.iter_packages(upgrade_me, selected, nuninst=nuninst_end, lundo=lundo)
  File "/srv/laniakea/dist/britney2/britney.py", line 2250, in iter_packages
    accepted, nuninst_after, comp_undo, failed_arch = self.try_migration(comp, nuninst_last_accepted, lundo)
  File "/srv/laniakea/dist/britney2/britney.py", line 2149, in try_migration
    allow_smooth_updates=False)
  File "/srv/laniakea/dist/britney2/britney.py", line 1855, in _compute_groups
    if binaries_t[parch][0][binary].source != source_name:
KeyError: 'policycoreutils-gui'

Given this configuration file (slightly reduced and simplified here):

NONINST_STATUS      = output/target/non-installable-status
EXCUSES_OUTPUT      = output/target/excuses.html
EXCUSES_YAML_OUTPUT = output/target/excuses.yaml
UPGRADE_OUTPUT      = output/target/output.txt
HEIDI_OUTPUT        = output/target/HeidiResult
STATIC_INPUT_DIR = input/
HINTSDIR         = input/hints
STATE_DIR        = state/
HINTS_LANIAKEA   = ALL
SMOOTH_UPDATES = libs oldlibs
IGNORE_CRUFT   = 1
UNSTABLE = /path/to/landing
TESTING  = /path/to/purple
COMPONENTS = main
ARCHITECTURES = amd64 arm64
NOBREAKALL_ARCHES = amd64
MINDAYS_LOW = 0
MINDAYS_EMERGENCY = 0
MINDAYS_CRITICAL = 0
MINDAYS_HIGH = 0
MINDAYS_MEDIUM = 0
DEFAULT_URGENCY   = medium
OUTOFSYNC_ARCHES  = amd64 arm64
BREAK_ARCHES      =
NEW_ARCHES        =

And using these Packages/Sources as input: https://people.debian.org/~mak/share/pureos-mindist-britney-crash.zip (file is a bit bigger than what could be uploaded here).

Britney2 is at commit d11152b36601bf81c2ab0f24a476654bb1e59eff when showing this crash.
A difference to the previous crash (at the same position) that did show up first is that this time, arm64 is not in BREAK_ARCHES.

If there is any further information you need or anything I can help with, please let me know!
Thank you!

Use case - "Suite multiplexer"

• Import from Debian testing
• Import from "own special sauce"-suite
• End result is a merged suite

Open question:

• How to handle conflicts (e.g. the two suites have the same packages). Possibly via policy

For proper dbgsym support, Britney needs to support multiple archives at the same time.

If we are going to support release files (see issue #10), we might as well pull the architecture list from the Release file of the target distribution.

Britney still has a number of data files stored in various "suites" or assumed to be a part of the mirror

Known cases:

• Dates (in $TESTING)
• Urgencies (in $TESTING)
• BugsV (in $UNSTABLE and $TESTING)
• Hints/* (in $UNSTABLE if hintsdir is not set - partially fixed)
• Faux packages (assumed to be a part of the Packages files). Issue #5

Raphaël Hertzog suggested that it would be nice to have a machine parsable file of the migration attempts (instead of or in addition to the current output.log). Requested features include:

• Separate packages that are directly broken from packages that are transitively broken. (Caveat: not all breakage happens at "first" level).
• In the list of old libraries, include a list of (versioned) reverse dependencies that prevents their removal.

Currently a d-i freeze involves blacklisting packages with udebs. However, Britney should have enough information to determine which packages have udebs and thereby support a "block-all udeb". This would enable us to freeze all udebs.

Use case:

• Import Debian suite (e.g. unstable)
• Tell Britney "I want package X, Y and Z plus everything required for them"
• Britney creates a target suite with exactly that.

With the new mirror layout support, it would make sense if Britney actually enforced that certain components are not allowed to depend on others (e.g. main vs. non-free)

So we can avoid needing to wait for the dinstall/dak lock.

Commit 4ced8a3 and 2e81e55 are both issues caused by the "src_data.binaries" field being a poor over-approximation of the packages related to a given migration item. While packages in a migration item should be based on that field, it is clear they are not exactly equal to the contents of that field.

Britney's current arch:all has work ok for a long while, but it is a hack and should be replaced by a proper solution (that does not require force-hints when Britney is wrong).

What is expected from Britney's arch:all handling?

• arch:all packages must be installable on at least 1 architecture.
• arch:all packages may be uninstallable on a strict subset of the architectures assuming they have no arch:any rdeps on that architecture.
• arch:all that used to be installable must not regress without a hint/manual approval except:
  • If the source previously provided arch:any binaries but no longer does for that architecture. (Assumption, if arch:any + arch:all are provided, the arch:all are assumed only to be relevant on architectures with arch:any binaries)

Reports are at: https://piuparts.debian.org/<suite|test>/summary.json (e.g. https://piuparts.debian.org/sid/summary.json)

Status is defined in[1]:

• "P" = Passed (or essential)
• "F" = Failed
• "W" = waiting for test
• "-" = Weird (doesn't exist)
• "X" = Stalled (dependency failed OR cannot be tested)

[1] https://anonscm.debian.org/cgit/piuparts/piuparts.git/tree/piupartslib/pkgsummary.py

Currently we have a helper script to generate and inject fake packages for various purposes. Britney should support these use-cases directly (which would also assist in fixing issue #2).

Known existing use cases:

• Avoid accidental removals (noremove.d - used by e.g. d-i)
  • Bonus for handling the "tasksel" case.
• Permit dependencies on certain missing packages (primarily for non-free packages that rely on third-party repositories)

Possibly new use-case:

• Ensure (co-)installability of certain package sets (https://bugs.debian.org/707219)

The test suite has issues in the break-arch live data test (see https://bugs.debian.org/803633). This is probably due to the essential set handling being wrong.

If an essential package goes from uninstallable to installable, we would need to re-test all broken packages to see if some of them starts being installable. At the moment, I do not think that is done.

Once the essential set is installable, Britney will keep it that way. This is presumably why only the break arch test stands out.

When bootstrapping a new architecture, it will not be in p-u and it will not appear until the next release. We need some decent way to handle that, which does not imply the new architecture being a(n implicit) "BREAK_ARCH"

Now that Britney (mostly) supports a native mirror layout, it would be great if she also supported "partial suites".

Examples include: Ubuntu's "$NAME-proprosed" suite is not self-contained and requires "$NAME" (similar to experimental vs. unstable in Debian, but that example is less relevant for Britney)

It would be great if Britney ensured that testing was self-contained build-dependency wise.

A couple of open questions:

• How to handle arch:all / Build-Depends-Indep?
  • Is one arch sufficient?
  • Is any arch sufficient?
  • Should it be the "no break arch:all arches"? (Caveat: They do not cover "arch:X-only" packages)