kubernetes-sigs / hydrophone Goto Github PK

The current code base is using go v1.18. We should bump this to at least 1.20.

• Run a single test (quick!)
• Run conformance tests.

All of the prow end to end tests that we have use the same setup where we build the binary then run the script. Example: https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes-sigs/hydrophone/hydrophone-periodic.yaml#L32-L34

We could simplify this by adding a make command called test-e2e in the Makefile that runs build then invokes the hack/run-e2e.sh script.

If anyone wants to take a look at this, please feel free to assign yourself to the issue. And if there are any questions, reach out in this issue or to me on slack at @rjsadow

/good-first-issue

Currently hydrophone uses a spinner to indicate something is going on. This is, I think, because Ginkgo's default output is one long line, so the line-based log reading hydrophone does waits "forever" and no progress is visible.

Since there is already a --verbosity flag, I was wondering if it could be used to toggle between

• v <=4: show the spinner, like it is right now
• v >= 6: do not show the spinner, but configure E2E_EXTRA_GINKGO_ARGS=-v, which will lead to more line-based output.

Currently, setting the -v arg on Ginkgo by hacking hydrophone looks like

So a way to disable the spinner in favor of more verbose output would be nice.

See here for how RESULTS_DIR is created and populated:
https://cs.k8s.io/?q=RESULTS_DIR&i=nope&files=&excludeFiles=&repos=kubernetes/kubernetes

see how we do a docker exec to get the results in shell script based CI job:
https://github.com/kubernetes/test-infra/blob/8bd221589dbbdcbc93598d3a1f125f09d18428cc/experiment/kind-conformance-image-e2e.sh#L156

see how sonobuoy deals with it:
https://github.com/vmware-tanzu/sonobuoy/search?l=Go&q=RESULTS_DIR

Essentially, you need to write code that will get into the pod, and extract all files in the /tmp/results directory and drop it into the same --output directory you have in:
#2

thanks,
Dims

When a resource already exists, such as clusterrole etc, we log both:

21:54:37 INF clusterrole already exist conformance-serviceaccount
21:54:37 INF clusterrole created 

which is somewhat misleading. This should be fixed i.e. if a resource exists, we should only log already exists and if it gets created, we should log it got created.

Just a reminder for now... but we may have to figure out how to test

What's the issue?

The logs of hydrophone show time in different time zones. One seems to be local time and other in UTC.

How to reproduce?

bin/hydrophone --conformance

Migrate from github actions to prow

Periodic e2e Tests (periodic_e2e.yaml)

• Add prow config for periodic execution of test, scheduled every 6 hours.
• Ensure the build process and e2e tests run similarly to the current GitHub Actions workflow.
• Implement artifact upload for conformance results in Prow.

Periodic Conformance Tests (periodic_conformance.yml)

• Add prow config for periodic execution of full conformance test, scheduled every 6 hours.
• Adapt the steps for building, testing, and running full conformance tests.
• Include steps to archive conformance results.

Migrate presubmit (test.yml)

• Ensure linting is replicated in prow.
• Adapt build, unit test
• Implement checks for dry run duration and test execution duration.
• Set up artifact archiving for conformance results.

• https://github.com/kubernetes-sigs/windows-operational-readiness/
• https://kubernetes.io/blog/2024/04/03/intro-windows-ops-readiness/

Currently --parallel gives the impression of setting the concurrency level (thread/goroutine count). This makes sense when looking at the Dockerfile for the conformance tester:

ENV E2E_PARALLEL="1"

However when I grep through the k/k codebase, this env is only used in 2 places:

• The go-runner, which is the image's default entrypoint. But the go-runner does not treat the env var as an integer, but only interprets it as a boolean flag: https://github.com/kubernetes/kubernetes/blob/ec301a5cc76f48cdadc77bcfbd686cf40b124ecf/test/conformance/image/go-runner/cmd.go#L34-L40
• The run_e2e.sh, which is the only place that optionally considers the env to be a number: https://github.com/kubernetes/kubernetes/blob/ec301a5cc76f48cdadc77bcfbd686cf40b124ecf/test/conformance/image/run_e2e.sh#L60-L66

Since hydrophone does not set a custom command, setting --parallel=7 has no effect.

run_e2e.sh is deprecated and now only a shim to the go-runner, so I guess it would make sense to redefine --parallel to be a boolean. Alternatively maybe it could remain an int and set the --procs (--nodes is an alias for it) flag via E2E_EXTRA_GINKGO_ARGS, like the run_e2e.sh did?

Hey all, would like to start using this instead of sonobuoy, however you don't have a brew package yet. Would love to see that get added.

Something like https://github.com/containerd/containerd/blob/main/ADOPTERS.md ?

Sometimes control plane nodes are not schedulable by default, requiring the use of additional conformance flags like --allowed-not-ready-nodes=3 for tests. However, Hydrophone does not currently offer a comprehensive mapping to the conformance image options.

ref: https://kubernetes.slack.com/archives/C78F00H99/p1705427311573629
ref: https://github.com/kubernetes/kubernetes/blob/9b5e2dc54d2315402a462e75c168bfc4ce28ecf3/test/e2e/framework/test_context.go#L363

against kind cluster is fine! there may be other settings we have to do to ensure all the conformance tests pass that we don't know of yet.

An issue in our hack/run-e2e.sh script is causing periodic jobs to fail. The variable CHECK_DURATION should be initialized similar to the other variables around https://github.com/kubernetes-sigs/hydrophone/blob/main/hack/run-e2e.sh#L70-L82 or have a default value set on line 54.

hack/run-e2e.sh: line 54: CHECK_DURATION: unbound variable

If anyone wants to take a look at this, please feel free to assign yourself to the issue. And if there are any questions, reach out in this issue or to me on slack at @rjsadow

https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/hydrophone-e2e-check/1747715074929201152

The #hydrophone slack channel was recently created. We need to update the readme https://github.com/kubernetes-sigs/hydrophone/blob/main/README.md?plain=1#L121 with the new slack channel and make sure it's apparent that this is the preferred channel to reach out on. We can keep #sig-testing and #sig-conformance as they're very related and relevant to this project.

If anyone wants to take a look at this, please feel free to assign yourself to the issue. And if there are any questions, reach out in this issue or to me on slack at @rjsadow

/good-first-issue

A user might try to build the binary using go build . which will result in the creation of ./hydrophone. To avoid pushing this binary to git, we should add hydrophone to our .gitignore.

The current process of running ./hydrophone -conformance is cumbersome as it waits for tests to finish and then grabs the results. This could be more efficient. It may be more user friendly to refactor some of the testing code into separate sub-commands (e.g., run, get-logs, fetch-files, cleanup).

https://github.com/kubernetes/kubernetes/blob/master/test/conformance/testdata/conformance.yaml shows 388 tests. we seem to be running 380 only

Multiple mistake available in air-gapped.md

The error check on https://github.com/kubernetes-sigs/hydrophone/blob/main/pkg/client/check.go#L63 seems redundant since it will always be nil given that within the for loop a new err variable is initialized everytime https://github.com/kubernetes-sigs/hydrophone/blob/main/pkg/client/check.go#L55

There are two unused parameters cfg and config in the PrintListImages() function under pkg/service/list_images.go.
Since these are unused, it would be better to remove them from the function.

/good-first-issue

ref: https://kubernetes.slack.com/archives/C6L3G051C/p1703611973360109?thread_ts=1703359255.475159&cid=C6L3G051C

Waiting for pod to terminate...
2023/12/26 22:19:04 Pod terminated.
2023/12/26 22:19:04 downloading e2e.log to  /Users/ishankhare/work/hydrophone/e2e.log
2023/12/26 22:19:04 unable to download e2e.log: unable to upgrade connection: container not found ("output-container")
exit status 1

What's the issue?

While running the test we don't show any progress bar or any other info to know that the tests are running. Since the test suite takes a while to run, it sometimes feels like the process hangs and there's no progress at all.

(screenshot showing the same content even after 30 minutes)

Some ideas to tackle this might be adding a progress animation like the kind does when spinning up a cluster OR simply setting the user expectations that the process might take a while.

How to reproduce

Run the following command

bin/hydrophone --conformance

Hydrophone currently always selects the latest image version (v1.29.0) by default, regardless of the cluster version it is testing. This can lead to mismatches causing inconsistencies in testing and results. Users can specify the version image with --conformance-image, but requires additional user input, configuration, and knowledge of the command line options. The desired behavior is for Hydrophone to automatically detect and use a conformance image that matches the cluster version.

ref: https://kubernetes.slack.com/archives/C78F00H99/p1705427311573629

We are not setting default value for namespace flag when calling hydrophone --cleanup as a result its getting a blank namespace https://github.com/kubernetes-sigs/hydrophone/blob/main/pkg/service/init.go#L325

reeta@Reetas-MacBook-Pro homebrew-core % hydrophone --cleanup
17:35:42 INF API endpoint : https://127.0.0.1:50108
17:35:42 INF Server version : version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-19T15:42:59Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/arm64"}
17:35:42 ERR an empty namespace may not be set when a resource name is provided
reeta@Reetas-MacBook-Pro homebrew-core % 

What's the issue?

There're 2 broken links in the README.md which are not resolving to correct resources:

1. contributor guide: https://sigs.k8s.io/hydrophone/blob/main/CONTRIBUTING.md
2. filing an issue: https://sigs.k8s.io/hydrophone/issues/new

xref: kubernetes/org#4654

For moving the project into the kubernetes organization, the repo needs to follow a standard conformance template (https://github.com/kubernetes/kubernetes-template-project/). Items that need to be set:

• README.md outlining the project goals, sponsoring sig, and community contact information
• OWNERS with the project leads listed as approvers
• CONTRIBUTING.md outlining how to contribute to the project
• code-of-conduct.md copied from the template repo
• LICENSE which must be Apache 2.0
• SECURITY_CONTACTS with the contact points for the Product Security Team

Suppose you download Hydrophone and put it somewhere in your PATH. You navigate to your project of choice and take a look:

$ cd ~/myproject
$ hydrophone --help
...
      --output-dir string           directory for logs. (default "/home/xrstf/myproject")
...

Cool! So you run it to perform conformance tests:

$ export KUBECONFIG=...
$ hydrophone --conformance
...
13:30:53 INF Downloading e2e.log to /home/xrstf/myproject/e2e.log...
13:30:53 INF Downloading junit_01.xml to /home/xrstf/myproject/junit_01.xml...

Cool! Now let's run the conformance tests for another project:

$ cd ..
$ cd anotherproject
$ hydrophone --help
...
      --output-dir string           directory for logs. (default "/home/xrstf/anotherproject")
...

$ export KUBECONFIG=...
$ hydrophone --conformance
...
13:30:53 INF Downloading e2e.log to /home/xrstf/myproject/e2e.log...
13:30:53 INF Downloading junit_01.xml to /home/xrstf/myproject/junit_01.xml...

And now you'd be confused why your files ended up in a different location.

I think the way a config file works should be handled differently. Silently remembering default values that are path-dependent is IMHO not a good idea. I would never expect a tool to just take my current flags and use them for all future invocations if I don't specify a flag. Especially since I have no real way to opt out of this behaviour.