v1.9.0-beta.2 about sig-release

[spiffxp]

@enisoc asked for a status check today at contributor summit

• good news: all sig-release-master-blocking tests are passing!
• bad news: sig-release-1.9-blocking#verify is failing, and all sig-release-master-upgrade tests are failing

I'm getting a little worried now. We're moving in the wrong direction with upgrade tests. I am doubtful we will resolve all of these failures in the week we have left before code freeze, let alone any more that crop up.

[enisoc]

release-1.9 verify is still only broken due to kubernetes/release#470. We could fix it one of two ways: revert the branch to stubs, or disable the check for stubs on the branch. I'm still looking for guidance on which is right.

The consistently failing test cases within the upgrade jobs seem to be:

• dashboard test for which we just merged a PR into the 1.8 branch that we think might help.
• network partition test that we already agreed is non-blocking and should be disabled if it can't be fixed.

[enisoc]

Verify seems to be passing after kubernetes/kubernetes#56910.

https://k8s-testgrid.appspot.com/sig-release-1.9-blocking#verify-1.9

The network partition tests had already been disabled on master, but we are now cherrypicking that to the 1.8 branch (kubernetes/kubernetes#56900).

The dashboard test seems to be getting better since kubernetes/kubernetes#56793 merged, but on some jobs it hasn't run again yet.

https://k8s-testgrid.appspot.com/sig-release-master-upgrade#gce-1.8-master-upgrade-cluster-parallel
https://k8s-testgrid.appspot.com/sig-release-master-upgrade#gce-1.8-master-upgrade-master

[enisoc]

Verify is green.

The CP to disable the network partition tests on the 1.8 branch is on hold due to the 1.8.5 cut planned today.

The dashboard test is passing on the upgrade job now, but a bunch of others started failing:

https://k8s-testgrid.appspot.com/sig-release-master-upgrade#gce-1.8-master-upgrade-master

[enisoc]

The official release-blocking list is all green for HEAD of release-1.9:

  Job #                                   Run #   Build # Time/Status
= ======================================= =====   ======= ===========
* build-1.9                               #12     #114    [19:18 12/06]
* (--buildversion=v1.9.0-beta.1.114+3ac6d1f5c03bbc)
- verify-release-1.9                      #667    #114    PASSED
- test-go-release-1.9                     #3284   #114    PASSED
- bazel-build-1-9                         #211    #114    PASSED
- bazel-test-1-9                          #228    #114    PASSED
- periodic-kubernetes-bazel-build-1-9     #166    #114    PASSED
- periodic-kubernetes-bazel-test-1-9      #162    #114    PASSED
- e2e-kubeadm-gce-1-8-on-1-9              #212    #114    PASSED
- periodic-kubernetes-e2e-kubeadm-gce-1-9 #173    #114    PASSED
- e2e-gce-cos-k8sbeta-default             #477    #114    PASSED
- e2e-gce-cos-k8sbeta-reboot              #496    #114    PASSED
- e2e-gce-cos-k8sbeta-serial              #103    #114    PASSED
- e2e-gce-cos-k8sbeta-slow                #481    #114    PASSED
- e2e-gce-cos-k8sbeta-ingress             #488    #114    PASSED
- e2e-gce-cos-k8sbeta-alphafeatures       #478    #114    PASSED
- e2e-gke-cos-k8sbeta-default             #488    #114    PASSED
- e2e-gke-cos-k8sbeta-reboot              #441    #114    PASSED
- e2e-gke-cos-k8sbeta-serial              #105    #114    PASSED
- e2e-gke-cos-k8sbeta-slow                #491    #114    PASSED
- e2e-gke-cos-k8sbeta-ingress             #511    #114    PASSED
- e2e-gce-gpu-beta                        #201    #114    PASSED
- e2e-gce-device-plugin-gpu-beta          #205    #114    PASSED
- e2e-gke-device-plugin-gpu-beta          #203    #114    PASSED
- e2e-gke-device-plugin-gpu-p100-beta     #34     #114    PASSED
- e2e-kops-aws-beta                       #383    #114    PASSED
- node-kubelet-beta                       #369    #114    PASSED
- e2e-gci-gce-scalability-beta            #70     #114    PASSED

I suggest we cut beta.2 to get more mileage on the approved exceptions that have now merged. The upgrade/downgrade problems are unlikely to be worse than they were for beta.1. We just need to keep investigating.

cc @dashpole @spiffxp

[spiffxp]

@enisoc agreed

[dashpole]

beta.2 was cut yesterday