Comments (3)
some steps and thoughs:
a) look at the test history in our slack #psap-ci-alerts and see that sometimes, the cluster gets disconnected in the middle of our testing
see this recent test for an example: step 002__entitlement__deploy
failed because of the cluster being unreachable:
2021-06-08 23:39:24,514 p=630 u=psap-ci-runner n=ansible | <stderr> from server for: "STDIN": Get "https://api.ci-op-cw4km932-3a604.origin-ci-int-aws.dev.rhcloud.com:6443/apis/machineconfiguration.openshift.io/v1/machineconfigs/50-rhsm-conf": dial tcp 52.200.204.34:6443: connect: connection refused
2021-06-08 23:39:24,514 p=630 u=psap-ci-runner n=ansible | ----- FAILED ----
b) see in ci-dashboard that when this appear, the test is shown in :red_jenkins_circle: before the GPU Operator test failed
so to improve that:
-
in the nightly tests, add trap ERR, and test if the cluster is still reachable (oc version). If unreachable, touch $ARTIFACTS/CLUSTER_DISCONNECTED
-
in the ci-dashboard, if the test fail, test if $ARTIFACTS/CLUSTER_DISCONNECTED exists and if it does, indicate one way or another that the test could not complete
from ci-artifacts.
/close
done, see https://github.com/openshift-psap/ci-artifacts/blob/master/testing/run#L100
from ci-artifacts.
@kpouget: Closing this issue.
In response to this:
/close
done, see https://github.com/openshift-psap/ci-artifacts/blob/master/testing/run#L100
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
from ci-artifacts.
Related Issues (20)
- Add the ability to entitle only GPU nodes HOT 2
- set_scale.sh: cannot specify the source machineset HOT 3
- entitlement: using the same content for the entitlement.pem and entitlement-key.pem isn't safe HOT 4
- Learn about ansible TAGS and refactor the roles to us it
- Test OpenShift upgrade with GPU workload running
- Force Ansible "connection: local" for running all the commands HOT 3
- Use Ansible role "template" files instead of custom sed replacement
- documentation: add roles/*/README.md descriptions
- GPU Operator: Prepare a disconnected driver-container POC HOT 1
- Prepare a 'release' checklist and scripts to properly cut `ci-artifacts` releases
- Rewrite the toolbox scripts with a proper CLI parameters handling framework HOT 1
- Include "p3.2xlarge" GPU instance during CI HOT 1
- Ansible-lint tests only modified files
- Create a full weekly suite for the PSAP operators suite HOT 3
- Create a bot to highlight weekly PR`s
- GPU Operator: test PROXY configuration
- GPU Operator: run gpu-operator test_operatorhub should not specify the exact operator version
- ci-artifacts maintenance overview HOT 6
- Generic command for installing operators from OperatorHub HOT 2
- Typos HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ci-artifacts.