Detect when the nightly CI fails because of a cluster shutdown about ci-artifacts

[kpouget]

some steps and thoughs:

a) look at the test history in our slack #psap-ci-alerts and see that sometimes, the cluster gets disconnected in the middle of our testing
see this recent test for an example: step 002__entitlement__deploy failed because of the cluster being unreachable:

2021-06-08 23:39:24,514 p=630 u=psap-ci-runner n=ansible | <stderr> from server for: "STDIN": Get "https://api.ci-op-cw4km932-3a604.origin-ci-int-aws.dev.rhcloud.com:6443/apis/machineconfiguration.openshift.io/v1/machineconfigs/50-rhsm-conf": dial tcp 52.200.204.34:6443: connect: connection refused
2021-06-08 23:39:24,514 p=630 u=psap-ci-runner n=ansible | ----- FAILED ----

b) see in ci-dashboard that when this appear, the test is shown in :red_jenkins_circle: before the GPU Operator test failed

so to improve that:

1. in the nightly tests, add trap ERR, and test if the cluster is still reachable (oc version). If unreachable, touch $ARTIFACTS/CLUSTER_DISCONNECTED

2. in the ci-dashboard, if the test fail, test if $ARTIFACTS/CLUSTER_DISCONNECTED exists and if it does, indicate one way or another that the test could not complete

[kpouget]

/close

done, see https://github.com/openshift-psap/ci-artifacts/blob/master/testing/run#L100

[openshift-ci]

@kpouget: Closing this issue.

In response to this:

/close

done, see https://github.com/openshift-psap/ci-artifacts/blob/master/testing/run#L100

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.