Define criteria for a job to be release-blocking about sig-release

[enisoc]

We cut v1.9.0-alpha.2 with the following jobs failing out of the current master-release-blocking set:

• gci-gce-serial
• gci-gke-serial
• gci-gce-alpha-features
• gce-scale-correctness
• soak-gce-gci
• kubeadm-gce

Once we define criteria, we will be following up to evaluate these for continued blocking status.

[spiffxp]

Still working on a proposal, but here's my rough take for now:

An ideal candidate job for release-blocking:

• Finishes a run in 60 minutes or less
• Runs at least every 2 hours
• Passes 90% of all runs in the past week (regardless of underlying commit)
• Is capable of passing 3 times in a row against the same commit
• Is owned by a SIG that is responsive to @kubernetes/sig-foo-test-failures
• Fails for no more than 10 runs in a row

[jdumars]

Aaron and I worked on some definitions around this in a Google doc, and I would like to reconcile that effort with this PR. The doc is here: https://docs.google.com/document/d/1kCDdmlpTnHPQt5z8JzODdFCc3T2D4MKR53twsDZu20c/edit We had intended to do the necessary pre-work and then submit it as a proposal, but the process for "process proposals" is not clear. (my head just exploded)

[jdumars]

Also, I don't think this should be in a KEP. These are procedural guard rails around the build ecosystem, and require a different audience and level of review.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[justaugustus]

/remove-lifecycle stale

[justaugustus]

/lifecycle frozen

[spiffxp]

I'm going to try and reframe this into a step by step guide on how to move a job from:

• non-blocking to release-blocking
• bring-your-own-ci to triggered-by-k8s-prow
• pr-optional to pr-non-blocking
• pr-non-blocking to pr-blocking

Criteria will include responsive humans and metrics on signal reliability that are available to us today

Then PR this into test-infra/docs

[justaugustus]

/area release-team
/milestone v1.15
/priority critical-urgent

[justaugustus]

/unassign @jdumars

[BenTheElder]

#346 did this I think?

[spiffxp]

/close
Criteria are documented but not yet fully enforced (re: metrics on duration and flakiness)

Will continue to work that against #347

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

/close
Criteria are documented but not yet fully enforced (re: metrics on duration and flakiness)

Will continue to work that against #347

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.