grafana / tempo-operator Goto Github PK
View Code? Open in Web Editor NEWGrafana Tempo Kubernetes operator
Home Page: https://grafana.com/docs/tempo/latest/setup/operator/
License: GNU Affero General Public License v3.0
Grafana Tempo Kubernetes operator
Home Page: https://grafana.com/docs/tempo/latest/setup/operator/
License: GNU Affero General Public License v3.0
When I execute targets like "make all" I get the following output which is from go-get-tool:
test -s /Users/kearls/sources/os-observability/tempo-operator/bin/controller-gen-v0.9.2 || @[ -f /Users/kearls/sources/os-observability/tempo-operator/bin/controller-gen-v0.9.2 ] || { set -e ; TMP_DIR=$(mktemp -d) ; cd $TMP_DIR ; go mod init tmp ; echo "Downloading sigs.k8s.io/controller-tools/cmd/controller-gen" ; go get -d sigs.k8s.io/controller-tools/cmd/[email protected] ; GOBIN=/Users/kearls/sources/os-observability/tempo-operator/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen ; APP=$(echo "/Users/kearls/sources/os-observability/tempo-operator/bin/controller-gen") ; APP_NAME=$(echo "$APP-v0.9.2") ; mv "$APP" "$APP_NAME" ; rm -rf $TMP_DIR ; }
bash: line 1: @[: command not found
Some constants (e.g. configVolumeName, portHTTPServer) are duplicated across the tempo manifests and should be moved to manifestutils/constants.go.
The PodAntiAffinity rule of the query-frontend pod prevents the deployment from adding a new pod (and terminating the old one) during reconciling if the cluster only has a single node: Warning FailedScheduling 22s (x11 over 20m) default-scheduler 0/1 nodes are available: 1 node(s) didn't match pod anti-affinity rules. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
Steps to reproduce:
.spec.resources.total.limits.cpu
Currently, the operator uses tempo-gossip-member: 'true'
label selector for the gossip.
If two tempo instances are created in the same namespace this will create issues, hence the label should be changed to e.g. tempo-${instance}-gossip-member
The status should show:
Create release make target and CI config to release the operator (e.g. all-in-one yaml with all objects).
The build of the Tempo Operator fails with the error:
/home/iblancas/gopath/src/github.com/tempo-operator/api/v1alpha1/microservices_types.go:60:2: invalid field type: any
Error: not all generators ran successfully
$ make build
test -s /home/iblancas/gopath/src/github.com/tempo-operator/bin/controller-gen || GOBIN=/home/iblancas/gopath/src/github.com/tempo-operator/bin go install sigs.k8s.io/controller-tools/cmd/[email protected]
go: downloading sigs.k8s.io/controller-tools v0.9.2
go: downloading github.com/spf13/cobra v1.4.0
go: downloading golang.org/x/tools v0.1.10-0.20220218145154-897bd77cd717
go: downloading gopkg.in/yaml.v2 v2.4.0
go: downloading github.com/fatih/color v1.12.0
go: downloading github.com/gobuffalo/flect v0.2.5
go: downloading k8s.io/apiextensions-apiserver v0.24.0
go: downloading k8s.io/api v0.24.0
go: downloading k8s.io/apimachinery v0.24.0
go: downloading gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b
go: downloading sigs.k8s.io/yaml v1.3.0
go: downloading github.com/mattn/go-colorable v0.1.8
go: downloading github.com/mattn/go-isatty v0.0.12
go: downloading github.com/spf13/pflag v1.0.5
go: downloading github.com/gogo/protobuf v1.3.2
go: downloading k8s.io/utils v0.0.0-20220210201930-3a6ce19ff2f9
go: downloading k8s.io/klog/v2 v2.60.1
go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.2.1
go: downloading github.com/google/gofuzz v1.1.0
go: downloading sigs.k8s.io/json v0.0.0-20211208200746-9f7c6b3444d2
go: downloading gopkg.in/inf.v0 v0.9.1
go: downloading golang.org/x/sys v0.0.0-20220209214540-3681064d5158
go: downloading github.com/go-logr/logr v1.2.0
go: downloading github.com/json-iterator/go v1.1.12
go: downloading golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd
go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go: downloading github.com/modern-go/reflect2 v1.0.2
go: downloading golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
go: downloading golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3
go: downloading golang.org/x/text v0.3.7
/home/iblancas/gopath/src/github.com/tempo-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
/home/iblancas/gopath/src/github.com/tempo-operator/api/v1alpha1/microservices_types.go:60:2: invalid field type: any
Error: not all generators ran successfully
run `controller-gen object:headerFile=hack/boilerplate.go.txt paths=./... -w` to see all available markers, or `controller-gen object:headerFile=hack/boilerplate.go.txt paths=./... -h` for usage
$ go version
go version go1.19.1 linux/amd64
Add tempo and Jaeger query images to the:
After adding the webhooks for #39 I got the following after deploying the operator:
E0921 13:57:47.355694 1 leaderelection.go:330] error retrieving resource lock tempo-operator-system/8b886b0f.grafana.com: Get "[https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/tempo-operator-system/leases/8b886b0f.grafana.com](https://10.96.0.1/apis/coordination.k8s.io/v1/namespaces/tempo-operator-system/leases/8b886b0f.grafana.com)": context canceled
1.6637686673557637e+09 ERROR setup problem running manager {"error": "open /tmp/k8s-webhook-server/serving-certs/tls.crt: no such file or directory"}
main.main
/workspace/main.go:100
runtime.main
/usr/local/go/src/runtime/proc.go:250
Change local filesystem backed to object storage.
TODOs:
All components should use tempo service account. The service account should be configurable in the CR (e.g. useful for image pull secret)
Tempo changed the default port from 3100 to 3200 in their documentation and examples, I wonder if we should do the same for the operator?
Ref. grafana/tempo#809
The defaulter webhook is not run if the webhook gets installed after the CR is already created.
This leaves the Spec.Images.Tempo
and Spec.Images.TempoQuery
to its default value (empty), which then results in an error once the operator tries to create the deployment.
I'm not sure though if this is a problem in practice, I just came across it during development.
https://github.com/os-observability/tempo-operator/pull/46 added ingestion deployments.
The objective of this ticket is to create query deployments - query-frontend and querier.
Docs: https://grafana.com/docs/tempo/latest/configuration/#ingester
As part of this ticket, we should learn how the replication factor relates to the number of ingester/distributor replicas and apply some validation based on that.
Shall the operator remove a component from the ring?
The operator should Own all its generated resources and reconcile if they change.
Ref. in the loki-operator: https://github.com/grafana/loki/blob/fb3ad058e0de6bb4e3041691085df8a2c10efafb/operator/controllers/loki/lokistack_controller.go#L170
go: downloading github.com/modern-go/reflect2 v1.0.2
go: downloading golang.org/x/text v0.3.7
/home/runner/work/tempo-operator/tempo-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
panic: interface conversion: types.Type is *types.Interface, not *types.Named
https://github.com/os-observability/tempo-operator/runs/8304372293?check_suite_focus=true#step:4:38
Created from https://github.com/os-observability/tempo-operator/pull/55#discussion_r985604686
Storage secret should be validated in the webhook.
The webhook should check if the secret exists and the right content is provided - e.g. the endpoint is a valid endpoint etc.
Sometimes the 03-run-check-services
test of the E2E jaeger smoketest fails with wget: server returned error: HTTP/1.1 500 Internal Server Error
. Restarting the job succeeds.
In an error case, the tempo-query logs contains error finding ingesters in Querier.SearchTagValues: empty ring
:
{"level":"error","ts":1671726688.7815628,"caller":"app/http_handler.go:495","msg":"HTTP handler, Internal Server Error","error":"plugin error: rpc error: code = Unknown desc = error finding ingesters in Querier.SearchTagValues: empty ring\n","stacktrace":"github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleError\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:495\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).getServices\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:166\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2084\ngithub.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5\n\tgithub.com/opentracing-contrib/[email protected]/nethttp/server.go:154\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2084\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2084\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\tgithub.com/gorilla/[email protected]/mux.go:210\ngithub.com/jaegertracing/jaeger/cmd/query/app.additionalHeadersHandler.func1\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/additional_headers_handler.go:28\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2084\ngithub.com/jaegertracing/jaeger/pkg/bearertoken.PropagationHandler.func1\n\tgithub.com/jaegertracing/jaeger/pkg/bearertoken/http.go:51\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2084\ngithub.com/gorilla/handlers.CompressHandlerLevel.func1\n\tgithub.com/gorilla/[email protected]/compress.go:100\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2084\ngithub.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\tgithub.com/gorilla/[email protected]/recovery.go:78\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2916\nnet/http.(*conn).serve\n\tnet/http/server.go:1966"}
All pods are started successfully however:
logger.go:42: 16:31:29 | smoketest-with-jaeger/3-run-check-services | NAME READY STATUS RESTARTS AGE
logger.go:42: 16:31:29 | smoketest-with-jaeger/3-run-check-services | tempo-simplest-compactor-f5888c49-9pb5h 1/1 Running 0 46s
logger.go:42: 16:31:29 | smoketest-with-jaeger/3-run-check-services | tempo-simplest-distributor-5ff56df548-b9d5n 1/1 Running 0 46s
logger.go:42: 16:31:29 | smoketest-with-jaeger/3-run-check-services | tempo-simplest-ingester-0 1/1 Running 0 46s
logger.go:42: 16:31:29 | smoketest-with-jaeger/3-run-check-services | tempo-simplest-querier-5c77cd7b7c-gz9kc 1/1 Running 0 46s
logger.go:42: 16:31:29 | smoketest-with-jaeger/3-run-check-services | tempo-simplest-query-frontend-74cd7948d7-rqss8 2/2 Running 0 46s
logger.go:42: 16:31:29 | smoketest-with-jaeger/3-run-check-services | tracegen-4hbxw 0/1 Completed 0 37s
I think sometimes the wget command is executed before the ingester gets registered in the ring. I assume #149 and asserting that all pods are ready might fix this flaky test.
Logs of the error case:
https://github.com/andreasgerstmayr/tempo-operator/actions/runs/3759424603/jobs/6388971648 (ignore the green checkmark)
I started minikube on my mac (sithout installing cert-manager), then did make install run
and got:
go run ./main.go
1.664956172949677e+09 INFO controller-runtime.metrics Metrics server is starting to listen {"addr": ":8080"}
1.664956172949868e+09 INFO controller-runtime.builder Registering a mutating webhook {"GVK": "tempo.grafana.com/v1alpha1, Kind=Microservices", "path": "/mutate-tempo-grafana-com-v1alpha1-microservices"}
1.6649561729499059e+09 INFO controller-runtime.webhook Registering webhook {"path": "/mutate-tempo-grafana-com-v1alpha1-microservices"}
1.6649561729499252e+09 INFO controller-runtime.builder Registering a validating webhook {"GVK": "tempo.grafana.com/v1alpha1, Kind=Microservices", "path": "/validate-tempo-grafana-com-v1alpha1-microservices"}
1.66495617294994e+09 INFO controller-runtime.webhook Registering webhook {"path": "/validate-tempo-grafana-com-v1alpha1-microservices"}
1.6649561729499671e+09 INFO setup starting manager
1.6649561729501529e+09 INFO Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.664956172950083e+09 INFO controller-runtime.webhook.webhooks Starting webhook server
1.664956172950231e+09 INFO Starting server {"kind": "health probe", "addr": "[::]:8081"}
1.6649561729502552e+09 INFO Stopping and waiting for non leader election runnables
1.6649561729502778e+09 INFO Stopping and waiting for leader election runnables
1.664956172950305e+09 INFO Starting EventSource {"controller": "microservices", "controllerGroup": "tempo.grafana.com", "controllerKind": "Microservices", "source": "kind source: *v1alpha1.Microservices"}
1.66495617295034e+09 INFO Starting Controller {"controller": "microservices", "controllerGroup": "tempo.grafana.com", "controllerKind": "Microservices"}
1.6649561729503438e+09 INFO Starting workers {"controller": "microservices", "controllerGroup": "tempo.grafana.com", "controllerKind": "Microservices", "worker count": 1}
1.664956172950349e+09 INFO Shutdown signal received, waiting for all workers to finish {"controller": "microservices", "controllerGroup": "tempo.grafana.com", "controllerKind": "Microservices"}
1.664956172950352e+09 INFO All workers finished {"controller": "microservices", "controllerGroup": "tempo.grafana.com", "controllerKind": "Microservices"}
1.664956172950355e+09 INFO Stopping and waiting for caches
1.66495617295036e+09 INFO Stopping and waiting for webhooks
1.664956172950363e+09 INFO Wait completed, proceeding to shutdown the manager
1.664956172950367e+09 ERROR setup problem running manager {"error": "open /var/folders/kg/6b37cdkx5l11j9sqp4v63dxc0000gn/T/k8s-webhook-server/serving-certs/tls.crt: no such file or directory"}
main.main
/Users/kearls/sources/os-observability/tempo-operator/main.go:100
runtime.main
/opt/homebrew/Cellar/go/1.19.1/libexec/src/runtime/proc.go:250
exit status 1
I also tried this after setting ENABLE_WEBHOOKS=false
and got the same result. This is a big impediment to developer productivity, so we should either fix it in the Makefile or figure out what steps are necessary and document it in CONTRIBUTING.md
Created from https://github.com/os-observability/tempo-operator/pull/56/#issuecomment-1263647995
k logs tempo-simplest-ingester-0 ploffay@fedora
level=info ts=2022-09-30T14:19:40.189855607Z caller=main.go:191 msg="initialising OpenTracing tracer"
level=warn ts=2022-09-30T14:19:40.190480636Z caller=config.go:155 msg="Local backend will not correctly retrieve traces with a distributed deployment unless all components have access to the same disk. You should probably be using object storage as a backend."
level=info ts=2022-09-30T14:19:40.190541339Z caller=main.go:106 msg="Starting Tempo" version="(version=, branch=HEAD, revision=fd5743d5d)"
level=info ts=2022-09-30T14:19:40.190706874Z caller=server.go:306 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
level=debug ts=2022-09-30T14:19:40.191743299Z caller=module_service.go:72 msg="module waiting for initialization" module=ingester waiting_for=server
level=info ts=2022-09-30T14:19:40.191765979Z caller=module_service.go:82 msg=initialising module=store
level=debug ts=2022-09-30T14:19:40.191767272Z caller=module_service.go:72 msg="module waiting for initialization" module=overrides waiting_for=server
level=debug ts=2022-09-30T14:19:40.191778596Z caller=module_service.go:72 msg="module waiting for initialization" module=usage-report waiting_for=memberlist-kv
ts=2022-09-30T14:19:40.19180243Z caller=memberlist_logger.go:74 level=debug msg="configured Transport is not a NodeAwareTransport and some features may not work as desired"
level=debug ts=2022-09-30T14:19:40.1917849Z caller=module_service.go:72 msg="module waiting for initialization" module=memberlist-kv waiting_for=server
level=info ts=2022-09-30T14:19:40.191801405Z caller=module_service.go:82 msg=initialising module=server
level=info ts=2022-09-30T14:19:40.191854654Z caller=module_service.go:82 msg=initialising module=memberlist-kv
level=info ts=2022-09-30T14:19:40.191868892Z caller=module_service.go:82 msg=initialising module=overrides
level=debug ts=2022-09-30T14:19:40.191882004Z caller=module_service.go:72 msg="module waiting for initialization" module=ingester waiting_for=store
level=debug ts=2022-09-30T14:19:40.191886806Z caller=module_service.go:72 msg="module waiting for initialization" module=usage-report waiting_for=server
level=debug ts=2022-09-30T14:19:40.191891367Z caller=module_service.go:72 msg="module waiting for initialization" module=ingester waiting_for=usage-report
level=info ts=2022-09-30T14:19:40.191895428Z caller=module_service.go:82 msg=initialising module=usage-report
level=debug ts=2022-09-30T14:19:40.192119498Z caller=module_service.go:72 msg="module waiting for initialization" module=ingester waiting_for=memberlist-kv
level=debug ts=2022-09-30T14:19:40.192442028Z caller=module_service.go:72 msg="module waiting for initialization" module=ingester waiting_for=overrides
level=info ts=2022-09-30T14:19:40.192452332Z caller=module_service.go:82 msg=initialising module=ingester
level=info ts=2022-09-30T14:19:40.192545729Z caller=ingester.go:328 msg="beginning wal replay"
level=warn ts=2022-09-30T14:19:40.192626314Z caller=rescan_blocks.go:24 msg="failed to open search wal directory" err="open /var/tempo/wal/search: no such file or directory"
level=info ts=2022-09-30T14:19:40.1929802Z caller=ingester.go:413 msg="wal replay complete"
level=info ts=2022-09-30T14:19:40.193022956Z caller=ingester.go:427 msg="reloading local blocks" tenants=0
level=debug ts=2022-09-30T14:19:40.193060457Z caller=tcp_transport.go:393 component="memberlist TCPTransport" msg=FinalAdvertiseAddr advertiseAddr=172.17.0.10 advertisePort=7946
level=info ts=2022-09-30T14:19:40.193078602Z caller=app.go:195 msg="Tempo started"
level=debug ts=2022-09-30T14:19:40.193384188Z caller=tcp_transport.go:393 component="memberlist TCPTransport" msg=FinalAdvertiseAddr advertiseAddr=172.17.0.10 advertisePort=7946
level=info ts=2022-09-30T14:19:40.193540931Z caller=lifecycler.go:576 msg="instance not found in ring, adding with no tokens" ring=ingester
level=debug ts=2022-09-30T14:19:40.19366803Z caller=lifecycler.go:412 msg="JoinAfter expired" ring=ingester
level=info ts=2022-09-30T14:19:40.193684836Z caller=lifecycler.go:416 msg="auto-joining cluster after timeout" ring=ingester
level=debug ts=2022-09-30T14:19:40.193870499Z caller=memberlist_client.go:842 msg="CAS attempt failed" err="no change detected" retry=true
ts=2022-09-30T14:19:40.600158068Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.1:14533"
level=debug ts=2022-09-30T14:19:41.194667986Z caller=broadcast.go:48 msg="Invalidating forwarded broadcast" key=collectors/ring version=2 oldVersion=1 content=[tempo-simplest-ingester-0] oldContent=[tempo-simplest-ingester-0]
level=debug ts=2022-09-30T14:19:41.599967326Z caller=broadcast.go:48 msg="Invalidating forwarded broadcast" key=collectors/usagestats_token version=2 oldVersion=1 content=[21165ff2-4a8d-415c-b522-2d418f840d6d] oldContent=[21165ff2-4a8d-415c-b522-2d418f840d6d]
ts=2022-09-30T14:19:50.194096438Z caller=memberlist_logger.go:74 level=debug msg="Failed to join 10.106.28.225:7946: dial tcp 10.106.28.225:7946: i/o timeout"
level=debug ts=2022-09-30T14:19:50.19411806Z caller=memberlist_client.go:542 msg="attempt to join memberlist cluster failed" retries=0 err="1 error occurred:\n\t* Failed to join 10.106.28.225:7946: dial tcp 10.106.28.225:7946: i/o timeout\n\n"
ts=2022-09-30T14:19:51.920216041Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: 10.106.28.225:7946"
level=info ts=2022-09-30T14:19:51.921613817Z caller=memberlist_client.go:563 msg="joined memberlist cluster" reached_nodes=1
ts=2022-09-30T14:20:16.650978517Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.13:38772"
level=debug ts=2022-09-30T14:20:20.198355734Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:20:21.218157391Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-distributor-64d9455df-s8rrp-cd90913d 172.17.0.13:7946"
ts=2022-09-30T14:20:46.653822706Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.13:60750"
ts=2022-09-30T14:21:16.655474821Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.13:36336"
ts=2022-09-30T14:21:21.220424449Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-distributor-64d9455df-s8rrp-cd90913d 172.17.0.13:7946"
ts=2022-09-30T14:21:46.660109753Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.13:40554"
ts=2022-09-30T14:21:51.222943226Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-distributor-64d9455df-s8rrp-cd90913d 172.17.0.13:7946"
ts=2022-09-30T14:22:16.662041502Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.13:42536"
ts=2022-09-30T14:22:21.225571434Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-distributor-64d9455df-s8rrp-cd90913d 172.17.0.13:7946"
ts=2022-09-30T14:22:46.664086303Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.13:49796"
ts=2022-09-30T14:22:51.227770528Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-distributor-64d9455df-s8rrp-cd90913d 172.17.0.13:7946"
ts=2022-09-30T14:23:16.667031549Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.13:37000"
ts=2022-09-30T14:23:21.230080434Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-distributor-64d9455df-s8rrp-cd90913d 172.17.0.13:7946"
ts=2022-09-30T14:23:46.669832176Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.13:59392"
ts=2022-09-30T14:23:51.232216395Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-distributor-64d9455df-s8rrp-cd90913d 172.17.0.13:7946"
ts=2022-09-30T14:24:16.672053596Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.13:45336"
ts=2022-09-30T14:24:21.235233088Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-distributor-64d9455df-s8rrp-cd90913d 172.17.0.13:7946"
ts=2022-09-30T14:24:46.674743316Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.13:52216"
ts=2022-09-30T14:24:51.237137586Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-distributor-64d9455df-s8rrp-cd90913d 172.17.0.13:7946"
ts=2022-09-30T14:25:21.238253302Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-distributor-64d9455df-s8rrp-cd90913d 172.17.0.13:7946"
ts=2022-09-30T14:25:46.677824896Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.13:52612"
ts=2022-09-30T14:25:51.240523901Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-distributor-64d9455df-s8rrp-cd90913d 172.17.0.13:7946"
ts=2022-09-30T14:26:16.680735215Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.13:53366"
and
k logs tempo-simplest-distributor-64d9455df-s8rrp ploffay@fedora
level=info ts=2022-09-30T14:19:40.596746275Z caller=main.go:191 msg="initialising OpenTracing tracer"
level=warn ts=2022-09-30T14:19:40.597338833Z caller=config.go:155 msg="Local backend will not correctly retrieve traces with a distributed deployment unless all components have access to the same disk. You should probably be using object storage as a backend."
level=info ts=2022-09-30T14:19:40.59739635Z caller=main.go:106 msg="Starting Tempo" version="(version=, branch=HEAD, revision=fd5743d5d)"
level=info ts=2022-09-30T14:19:40.59755628Z caller=server.go:306 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
ts=2022-09-30T14:19:40.598219047Z caller=memberlist_logger.go:74 level=debug msg="configured Transport is not a NodeAwareTransport and some features may not work as desired"
ts=2022-09-30T14:19:40Z level=info msg="OTel Shim Logger Initialized" component=tempo
level=debug ts=2022-09-30T14:19:40.598702442Z caller=tcp_transport.go:393 component="memberlist TCPTransport" msg=FinalAdvertiseAddr advertiseAddr=172.17.0.13 advertisePort=7946
level=debug ts=2022-09-30T14:19:40.599159339Z caller=tcp_transport.go:393 component="memberlist TCPTransport" msg=FinalAdvertiseAddr advertiseAddr=172.17.0.13 advertisePort=7946
level=debug ts=2022-09-30T14:19:40.599429759Z caller=module_service.go:72 msg="module waiting for initialization" module=memberlist-kv waiting_for=server
level=debug ts=2022-09-30T14:19:40.59944112Z caller=module_service.go:72 msg="module waiting for initialization" module=distributor waiting_for=memberlist-kv
level=debug ts=2022-09-30T14:19:40.599456055Z caller=module_service.go:72 msg="module waiting for initialization" module=overrides waiting_for=server
level=debug ts=2022-09-30T14:19:40.599472748Z caller=module_service.go:72 msg="module waiting for initialization" module=usage-report waiting_for=memberlist-kv
level=info ts=2022-09-30T14:19:40.599471231Z caller=module_service.go:82 msg=initialising module=server
level=info ts=2022-09-30T14:19:40.599848618Z caller=module_service.go:82 msg=initialising module=overrides
level=info ts=2022-09-30T14:19:40.599932225Z caller=module_service.go:82 msg=initialising module=memberlist-kv
level=debug ts=2022-09-30T14:19:40.600002146Z caller=module_service.go:72 msg="module waiting for initialization" module=distributor waiting_for=overrides
level=debug ts=2022-09-30T14:19:40.599483355Z caller=module_service.go:72 msg="module waiting for initialization" module=ring waiting_for=memberlist-kv
level=debug ts=2022-09-30T14:19:40.600022759Z caller=module_service.go:72 msg="module waiting for initialization" module=distributor waiting_for=ring
level=debug ts=2022-09-30T14:19:40.600035776Z caller=module_service.go:72 msg="module waiting for initialization" module=ring waiting_for=server
level=info ts=2022-09-30T14:19:40.600048934Z caller=module_service.go:82 msg=initialising module=ring
ts=2022-09-30T14:19:40.600105181Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: 10.106.28.225:7946"
level=debug ts=2022-09-30T14:19:40.600042837Z caller=module_service.go:72 msg="module waiting for initialization" module=usage-report waiting_for=server
level=info ts=2022-09-30T14:19:40.600126554Z caller=module_service.go:82 msg=initialising module=usage-report
level=info ts=2022-09-30T14:19:40.600390736Z caller=ring.go:263 msg="ring doesn't exist in KV store yet"
level=debug ts=2022-09-30T14:19:40.600454536Z caller=module_service.go:72 msg="module waiting for initialization" module=distributor waiting_for=server
level=debug ts=2022-09-30T14:19:40.600466976Z caller=module_service.go:72 msg="module waiting for initialization" module=distributor waiting_for=usage-report
level=info ts=2022-09-30T14:19:40.600472611Z caller=module_service.go:82 msg=initialising module=distributor
level=debug ts=2022-09-30T14:19:40.600217851Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=info ts=2022-09-30T14:19:40.601214952Z caller=memberlist_client.go:534 msg="joined memberlist cluster" reached_nodes=1
ts=2022-09-30T14:19:40Z level=info msg="Starting GRPC server on endpoint 0.0.0.0:4317" component=tempo
ts=2022-09-30T14:19:40Z level=info msg="Starting HTTP server on endpoint 0.0.0.0:4318" component=tempo
level=info ts=2022-09-30T14:19:40.601617818Z caller=app.go:195 msg="Tempo started"
level=debug ts=2022-09-30T14:19:40.732758823Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:19:41.132443129Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:19:41.681883022Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:19:42.194120538Z caller=broadcast.go:48 msg="Invalidating forwarded broadcast" key=collectors/ring version=2 oldVersion=1 content=[tempo-simplest-ingester-0] oldContent=[tempo-simplest-ingester-0]
level=debug ts=2022-09-30T14:19:42.659487846Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:19:45.757344091Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:19:48.97743145Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:19:51.920294631Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.1:1052"
level=debug ts=2022-09-30T14:19:55.773106279Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:20:03.085275277Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:20:11.687389226Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:20:16.650921283Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-ingester-0-415e44f4 172.17.0.10:7946"
level=debug ts=2022-09-30T14:20:20.968592248Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:20:21.218151452Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.10:43550"
level=debug ts=2022-09-30T14:20:29.569443884Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:20:39.339520319Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:20:46.514699493Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:20:46.653767379Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-ingester-0-415e44f4 172.17.0.10:7946"
level=debug ts=2022-09-30T14:20:53.436620122Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:20:59.861291571Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:21:07.556225991Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:21:16.56551341Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:21:16.655451545Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-ingester-0-415e44f4 172.17.0.10:7946"
ts=2022-09-30T14:21:21.220587578Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.10:46902"
level=debug ts=2022-09-30T14:21:23.17462959Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:21:30.807467513Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:21:38.125382773Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:21:46.660132064Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-ingester-0-415e44f4 172.17.0.10:7946"
level=debug ts=2022-09-30T14:21:47.546521482Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:21:51.222929956Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.10:48774"
level=debug ts=2022-09-30T14:21:54.425518925Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:22:03.142497285Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:22:12.177534398Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:22:16.662034921Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-ingester-0-415e44f4 172.17.0.10:7946"
level=debug ts=2022-09-30T14:22:20.4129592Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:22:21.225575939Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.10:35950"
level=debug ts=2022-09-30T14:22:28.453617915Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:22:35.983651148Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:22:45.011511924Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:22:46.664028129Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-ingester-0-415e44f4 172.17.0.10:7946"
ts=2022-09-30T14:22:51.227848743Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.10:55922"
level=debug ts=2022-09-30T14:22:54.505850997Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:23:02.152875872Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:23:08.81944818Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:23:16.667033806Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-ingester-0-415e44f4 172.17.0.10:7946"
level=debug ts=2022-09-30T14:23:18.41451966Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:23:21.230237305Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.10:32942"
level=debug ts=2022-09-30T14:23:26.670546844Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:23:35.044342127Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:23:42.204347516Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:23:46.669814926Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-ingester-0-415e44f4 172.17.0.10:7946"
level=debug ts=2022-09-30T14:23:49.134606996Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:23:51.232256358Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.10:54938"
level=debug ts=2022-09-30T14:23:57.30725Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:24:06.642179511Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:24:15.590287499Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:24:16.672037336Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-ingester-0-415e44f4 172.17.0.10:7946"
ts=2022-09-30T14:24:21.235281435Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.10:33194"
level=debug ts=2022-09-30T14:24:24.338517333Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:24:33.315454392Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:24:43.125279232Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:24:46.674730818Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-ingester-0-415e44f4 172.17.0.10:7946"
ts=2022-09-30T14:24:51.237148969Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.10:41260"
level=debug ts=2022-09-30T14:24:51.592802076Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:24:59.061433093Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:25:07.736890017Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:25:14.32339676Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:25:21.23827594Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.10:55532"
level=debug ts=2022-09-30T14:25:23.212392331Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:25:31.264146985Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:25:38.162407144Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:25:46.677824901Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-ingester-0-415e44f4 172.17.0.10:7946"
level=debug ts=2022-09-30T14:25:47.85560542Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:25:51.24055589Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.10:35988"
level=debug ts=2022-09-30T14:25:57.102426314Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:26:06.58151438Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:26:14.407902317Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:26:16.680640322Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-ingester-0-415e44f4 172.17.0.10:7946"
ts=2022-09-30T14:26:21.243456675Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.10:36380"
level=debug ts=2022-09-30T14:26:23.630248676Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:26:31.760417029Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:26:41.011049209Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:26:46.682623836Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-ingester-0-415e44f4 172.17.0.10:7946"
level=debug ts=2022-09-30T14:26:48.424980756Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
https://github.com/os-observability/tempo-operator/pull/46 creates tempo config from a string. This is inefficient and ugly to do handle if multiple config options need to be added conditionally.
Some questions:
i.e. "-mem-ballast-size-mbs=1024"
See #63
https://github.com/os-observability/tempo-operator/pull/63 adds query capabilities, however, it does not configure search - e.g. lookback for the query.
kubectl apply -f - <<EOF
apiVersion: tempo.grafana.com/v1alpha1
kind: Microservices
metadata:
name: simplest
spec:
limits:
global: {}
perTenant: {}
retention:
global: {}
perTenant: {}
storage:
secret: test
EOF
tempo-simplest-query-frontend ClusterIP 10.106.215.239 <none> 3100/TCP,9095/TCP,16686/TCP,16687/TCP 3m52s
tempo-simplest-query-frontend-discovery ClusterIP 10.97.166.203 <none> 3100/TCP,9095/TCP,16686/TCP,16687/TCP,9096/TCP 3m52s
I am also not sure if the discovery service should expose the Jaeger ports
Seems/tmp/.buildx-cache-new does not exist.
CI (main): https://github.com/os-observability/tempo-operator/actions/runs/3050914986/jobs/4918544129#step:12:9
Run rm -rf /tmp/.buildx-cache
mv: cannot stat '/tmp/.buildx-cache-new': No such file or directory
Error: Process completed with exit code 1.
cc @iblancasa
kubectl get microservices.tempo.grafana.com simplest -o yaml ploffay@fedora
apiVersion: tempo.grafana.com/v1alpha1
kind: Microservices
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"tempo.grafana.com/v1alpha1","kind":"Microservices","metadata":{"annotations":{},"name":"simplest","namespace":"default"},"spec":{"storage":{"secret":"minio-test"},"template":{"queryFrontend":{"jaegerQuery":{"enabled":true}}}}}
creationTimestamp: "2022-10-25T13:27:49Z"
generation: 1
name: simplest
namespace: default
resourceVersion: "378836"
uid: 9973ed88-a90d-4fcf-81c2-553a0a6171e9
spec:
limits:
global:
ingestion:
ingestionBurstSizeBytes: 0
ingestionRateLimitBytes: 0
maxBytesPerTrace: 0
maxTracesPerUser: 0
query:
maxSearchBytesPerTrace: 0
retention:
global:
traces: 172800000000000
storage:
secret: minio-test
storageSize: "0"
template:
queryFrontend:
component: {}
jaegerQuery:
enabled: true
------------------------------------------------------
traces: 172800000000000
could be shown nicely with 48h.
cc) @frzifus
The MicroservicesController should Own()
the tempo ConfigMap and restart pods if it changes.
add scripts and Makefile rules to generate a CRD markdown overview. Like in the OpenTelemetry and Jaeger Operator.
This is a follow up to #91 In that test I ran tracegen for 30 seconds to avoid a race condition in the verification step where traces may not yet be available.
If it's not possible to fix this via configuration then maybe add a retry loop to the verification step.
Publish the image per tag and merge to the main branch.
https://github.com/os-observability/tempo-operator/blob/main/api/v1alpha1/microservices_types.go#L46
The work depends on deploying the compactor
There are several situations (some of which I will list below) where either building locally changes generated code which then has to be reverted before committing, or developers need to run generate targets manually, and then that code needs to be checked in.
In the first case, building an image on a developer's machine will change config/manager/kustomization.yaml
and this change will need to be reverted before making a commit.
For the second, If this build is the same as the jaeger-operator or opentelemetry-collector-operator there are Makefile targets that such as api-docs
that must be run manually before a commit and whatever files they change need to be added to the commit.
We should decide what is the best way to deal with generated code, and have this automated and done by the build where possible
Tempo operator reuses some parts of the Loki operator. Namely https://github.com/os-observability/tempo-operator/blob/main/internal/manifests/mutate.go.
We should investigate if the shared codebase could be moved to a separate project/pollable library.
While the implementation of #15, the Docker cache was used to speedup the builds. There is an issue associated to the actions/cache@v3
GitHub Action that makes the cache grown pretty fast.
To avoid this issue, we have to add this hack to the end of the GitHub Actions Workflow
Add gossip service and label pods that participate with (tempo-gossip-member: 'true'
)
The operator should watch the storage secret and regenerate the tempo configuration if the secret changes.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.