grafana / tempo-operator Goto Github PK

View Code? Open in Web Editor NEW

51.0 9.0 24.0 2.43 MB

Grafana Tempo Kubernetes operator

Home Page: https://grafana.com/docs/tempo/latest/setup/operator/

License: GNU Affero General Public License v3.0

Dockerfile 0.36% Makefile 2.01% Go 97.00% Shell 0.64%

distributed-tracing grafana jaeger jaegertracing observability opentelemetry zipkin

tempo-operator's Introduction

Grafana Tempo operator

This is a Kubernetes operator for Grafana Tempo.

Features

Resource Limits - Specify overall resource requests and limits in the TempoStack CR; the operator assigns fractions of it to each component
AuthN and AuthZ - Supports OpenID Control (OIDC) and role-based access control (RBAC)
Managed upgrades - Updating the operator will automatically update all managed Tempo clusters
Multitenancy - Multiple tenants can send traces to the same Tempo cluster
mTLS - Communication between the Tempo components can be secured via mTLS
Jaeger UI - Traces can be visualized in Jaeger UI and exposed via Ingress or OpenShift Route
Observability - The operator and TempoStack operands expose telemetry (metrics, traces) and integrate with Prometheus ServiceMonitor and PrometheusRule

Documentation

Deploy

Install cert-manager and minio: make cert-manager deploy-minio
Build and deploy operator:

IMG_PREFIX=docker.io/${USER} OPERATOR_VERSION=$(date +%s).0.0 make docker-build docker-push deploy

Create a secret for minio in the namespace you are using:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: minio-test
stringData:
  endpoint: http://minio.minio.svc:9000
  bucket: tempo
  access_key_id: tempo
  access_key_secret: supersecret
type: Opaque
EOF

Create Tempo CR:

kubectl apply -f - <<EOF
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: simplest
spec:
  storage:
    secret:
      name: minio-test
      type: s3
  storageSize: 1Gi
  resources:
    total:
      limits:
        memory: 2Gi
        cpu: 2000m
  template:
    queryFrontend:
      jaegerQuery:
        enabled: true
EOF

Community

Grafana Slack #tempo-operator

tempo-operator's People

Contributors

Stargazers

Watchers

tempo-operator's Issues

Configure release

Create release make target and CI config to release the operator (e.g. all-in-one yaml with all objects).

Add gossip service

Add gossip service and label pods that participate with (tempo-gossip-member: 'true')

Use service account

All components should use tempo service account. The service account should be configurable in the CR (e.g. useful for image pull secret)

Tempo Operator build fails: not all generators ran successfully

Issue description

The build of the Tempo Operator fails with the error:

/home/iblancas/gopath/src/github.com/tempo-operator/api/v1alpha1/microservices_types.go:60:2: invalid field type: any
Error: not all generators ran successfully

Steps to reproduce:

$ make build
test -s /home/iblancas/gopath/src/github.com/tempo-operator/bin/controller-gen || GOBIN=/home/iblancas/gopath/src/github.com/tempo-operator/bin go install sigs.k8s.io/controller-tools/cmd/[email protected]
go: downloading sigs.k8s.io/controller-tools v0.9.2
go: downloading github.com/spf13/cobra v1.4.0
go: downloading golang.org/x/tools v0.1.10-0.20220218145154-897bd77cd717
go: downloading gopkg.in/yaml.v2 v2.4.0
go: downloading github.com/fatih/color v1.12.0
go: downloading github.com/gobuffalo/flect v0.2.5
go: downloading k8s.io/apiextensions-apiserver v0.24.0
go: downloading k8s.io/api v0.24.0
go: downloading k8s.io/apimachinery v0.24.0
go: downloading gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b
go: downloading sigs.k8s.io/yaml v1.3.0
go: downloading github.com/mattn/go-colorable v0.1.8
go: downloading github.com/mattn/go-isatty v0.0.12
go: downloading github.com/spf13/pflag v1.0.5
go: downloading github.com/gogo/protobuf v1.3.2
go: downloading k8s.io/utils v0.0.0-20220210201930-3a6ce19ff2f9
go: downloading k8s.io/klog/v2 v2.60.1
go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.2.1
go: downloading github.com/google/gofuzz v1.1.0
go: downloading sigs.k8s.io/json v0.0.0-20211208200746-9f7c6b3444d2
go: downloading gopkg.in/inf.v0 v0.9.1
go: downloading golang.org/x/sys v0.0.0-20220209214540-3681064d5158
go: downloading github.com/go-logr/logr v1.2.0
go: downloading github.com/json-iterator/go v1.1.12
go: downloading golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd
go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go: downloading github.com/modern-go/reflect2 v1.0.2
go: downloading golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
go: downloading golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3
go: downloading golang.org/x/text v0.3.7
/home/iblancas/gopath/src/github.com/tempo-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
/home/iblancas/gopath/src/github.com/tempo-operator/api/v1alpha1/microservices_types.go:60:2: invalid field type: any
Error: not all generators ran successfully
run `controller-gen object:headerFile=hack/boilerplate.go.txt paths=./... -w` to see all available markers, or `controller-gen object:headerFile=hack/boilerplate.go.txt paths=./... -h` for usage

$ go version
go version go1.19.1 linux/amd64

Own the tempo ConfigMap and restart pods on changes

The MicroservicesController should Own() the tempo ConfigMap and restart pods if it changes.

make run fails because it can't find cert

I started minikube on my mac (sithout installing cert-manager), then did make install run and got:

go run ./main.go
1.664956172949677e+09	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": ":8080"}
1.664956172949868e+09	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "tempo.grafana.com/v1alpha1, Kind=Microservices", "path": "/mutate-tempo-grafana-com-v1alpha1-microservices"}
1.6649561729499059e+09	INFO	controller-runtime.webhook	Registering webhook	{"path": "/mutate-tempo-grafana-com-v1alpha1-microservices"}
1.6649561729499252e+09	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "tempo.grafana.com/v1alpha1, Kind=Microservices", "path": "/validate-tempo-grafana-com-v1alpha1-microservices"}
1.66495617294994e+09	INFO	controller-runtime.webhook	Registering webhook	{"path": "/validate-tempo-grafana-com-v1alpha1-microservices"}
1.6649561729499671e+09	INFO	setup	starting manager
1.6649561729501529e+09	INFO	Starting server	{"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.664956172950083e+09	INFO	controller-runtime.webhook.webhooks	Starting webhook server
1.664956172950231e+09	INFO	Starting server	{"kind": "health probe", "addr": "[::]:8081"}
1.6649561729502552e+09	INFO	Stopping and waiting for non leader election runnables
1.6649561729502778e+09	INFO	Stopping and waiting for leader election runnables
1.664956172950305e+09	INFO	Starting EventSource	{"controller": "microservices", "controllerGroup": "tempo.grafana.com", "controllerKind": "Microservices", "source": "kind source: *v1alpha1.Microservices"}
1.66495617295034e+09	INFO	Starting Controller	{"controller": "microservices", "controllerGroup": "tempo.grafana.com", "controllerKind": "Microservices"}
1.6649561729503438e+09	INFO	Starting workers	{"controller": "microservices", "controllerGroup": "tempo.grafana.com", "controllerKind": "Microservices", "worker count": 1}
1.664956172950349e+09	INFO	Shutdown signal received, waiting for all workers to finish	{"controller": "microservices", "controllerGroup": "tempo.grafana.com", "controllerKind": "Microservices"}
1.664956172950352e+09	INFO	All workers finished	{"controller": "microservices", "controllerGroup": "tempo.grafana.com", "controllerKind": "Microservices"}
1.664956172950355e+09	INFO	Stopping and waiting for caches
1.66495617295036e+09	INFO	Stopping and waiting for webhooks
1.664956172950363e+09	INFO	Wait completed, proceeding to shutdown the manager
1.664956172950367e+09	ERROR	setup	problem running manager	{"error": "open /var/folders/kg/6b37cdkx5l11j9sqp4v63dxc0000gn/T/k8s-webhook-server/serving-certs/tls.crt: no such file or directory"}
main.main
	/Users/kearls/sources/os-observability/tempo-operator/main.go:100
runtime.main
	/opt/homebrew/Cellar/go/1.19.1/libexec/src/runtime/proc.go:250
exit status 1

I also tried this after setting ENABLE_WEBHOOKS=false and got the same result. This is a big impediment to developer productivity, so we should either fix it in the Makefile or figure out what steps are necessary and document it in CONTRIBUTING.md

Implement status

The status should show:

the health of the instance
available (ingestion) APIs
version

ensure dependencies and tools are only downloaded when needed

Create shared library/operator with Loki

Tempo operator reuses some parts of the Loki operator. Namely https://github.com/os-observability/tempo-operator/blob/main/internal/manifests/mutate.go.

We should investigate if the shared codebase could be moved to a separate project/pollable library.

Add ballast options across components

i.e. "-mem-ballast-size-mbs=1024"

See #63

Configure historical search

https://github.com/os-observability/tempo-operator/pull/63 adds query capabilities, however, it does not configure search - e.g. lookback for the query.

go-get-tool in Makefile gets "command not found" on mac

When I execute targets like "make all" I get the following output which is from go-get-tool:

test -s /Users/kearls/sources/os-observability/tempo-operator/bin/controller-gen-v0.9.2 || @[ -f /Users/kearls/sources/os-observability/tempo-operator/bin/controller-gen-v0.9.2 ] || { set -e ; TMP_DIR=$(mktemp -d) ; cd $TMP_DIR ; go mod init tmp ; echo "Downloading sigs.k8s.io/controller-tools/cmd/controller-gen" ; go get -d sigs.k8s.io/controller-tools/cmd/[email protected] ; GOBIN=/Users/kearls/sources/os-observability/tempo-operator/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen ; APP=$(echo "/Users/kearls/sources/os-observability/tempo-operator/bin/controller-gen") ; APP_NAME=$(echo "$APP-v0.9.2") ; mv "$APP" "$APP_NAME" ; rm -rf $TMP_DIR ; }
bash: line 1: @[: command not found

Run scorecard tests in CI

Reduce test times for e2e tests

This is a follow up to #91 In that test I ran tracegen for 30 seconds to avoid a race condition in the verification step where traces may not yet be available.

If it's not possible to fix this via configuration then maybe add a retry loop to the verification step.

query-frontend pod reconciliation doesn't work due to PodAntiAffinity role

The PodAntiAffinity rule of the query-frontend pod prevents the deployment from adding a new pod (and terminating the old one) during reconciling if the cluster only has a single node: Warning FailedScheduling 22s (x11 over 20m) default-scheduler 0/1 nodes are available: 1 node(s) didn't match pod anti-affinity rules. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.

Steps to reproduce:

Create a CR
Change a value triggering new deployments, for example modify .spec.resources.total.limits.cpu

Create query deployment

https://github.com/os-observability/tempo-operator/pull/46 added ingestion deployments.

The objective of this ticket is to create query deployments - query-frontend and querier.

Move manifest constants to shared file

Some constants (e.g. configVolumeName, portHTTPServer) are duplicated across the tempo manifests and should be moved to manifestutils/constants.go.

Fix certificate issues

After adding the webhooks for #39 I got the following after deploying the operator:

E0921 13:57:47.355694       1 leaderelection.go:330] error retrieving resource lock tempo-operator-system/8b886b0f.grafana.com: Get "[https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/tempo-operator-system/leases/8b886b0f.grafana.com](https://10.96.0.1/apis/coordination.k8s.io/v1/namespaces/tempo-operator-system/leases/8b886b0f.grafana.com)": context canceled
1.6637686673557637e+09	ERROR	setup	problem running manager	{"error": "open /tmp/k8s-webhook-server/serving-certs/tls.crt: no such file or directory"}
main.main
	/workspace/main.go:100
runtime.main
	/usr/local/go/src/runtime/proc.go:250

Watch the storage secret and regenerate the tempo config if required

The operator should watch the storage secret and regenerate the tempo configuration if the secret changes.

Avoid fast cache growing in publish-images GitHub Actions Workflow

While the implementation of #15, the Docker cache was used to speedup the builds. There is an issue associated to the actions/cache@v3 GitHub Action that makes the cache grown pretty fast.

To avoid this issue, we have to add this hack to the end of the GitHub Actions Workflow

Add validation of storage secret to webhook

Created from https://github.com/os-observability/tempo-operator/pull/55#discussion_r985604686

Storage secret should be validated in the webhook.

The webhook should check if the secret exists and the right content is provided - e.g. the endpoint is a valid endpoint etc.

Configure readiness probes

See https://github.com/grafana/tempo/blob/0ed9ddcdbb38f2a6348f665ccd52aea3dc7a3b8e/operations/jsonnet-compiled/Deployment-querier.yaml#L39-L44 for example

Image field not set if the CR is created before the webhook is installed

The defaulter webhook is not run if the webhook gets installed after the CR is already created.

This leaves the Spec.Images.Tempo and Spec.Images.TempoQuery to its default value (empty), which then results in an error once the operator tries to create the deployment.

I'm not sure though if this is a problem in practice, I just came across it during development.

Create Tempo config programmatically

https://github.com/os-observability/tempo-operator/pull/46 creates tempo config from a string. This is inefficient and ugly to do handle if multiple config options need to be added conditionally.

Some questions:

if the config is generated programmatically will it work for older Tempo versions?

Jaeger query ports should not be exposed if jaeger-query is disabled

kubectl apply -f - <<EOF
apiVersion: tempo.grafana.com/v1alpha1
kind: Microservices
metadata:
  name: simplest
spec:
  limits:
    global: {}
    perTenant: {}
  retention:
    global: {}
    perTenant: {}
  storage:
    secret: test
EOF

tempo-simplest-query-frontend             ClusterIP   10.106.215.239   <none>        3100/TCP,9095/TCP,16686/TCP,16687/TCP            3m52s
tempo-simplest-query-frontend-discovery   ClusterIP   10.97.166.203    <none>        3100/TCP,9095/TCP,16686/TCP,16687/TCP,9096/TCP   3m52s

I am also not sure if the discovery service should expose the Jaeger ports

build fails - new cache does not exist

Seems/tmp/.buildx-cache-new does not exist.

CI (main): https://github.com/os-observability/tempo-operator/actions/runs/3050914986/jobs/4918544129#step:12:9

Run rm -rf /tmp/.buildx-cache
mv: cannot stat '/tmp/.buildx-cache-new': No such file or directory
Error: Process completed with exit code 1.

cc @iblancasa

Investigate health of a component on the ring

Shall the operator remove a component from the ring?

A component should remove itself and restart the pod
Should a health check use the information from the ring?
If pod restarts is/should it be explicitly removed from the ring
Should we trigger alerts if such an issue occurs?

generate crd markdown overview

add scripts and Makefile rules to generate a CRD markdown overview. Like in the OpenTelemetry and Jaeger Operator.

Create e2e tests

Update default tempo port to 3200

Tempo changed the default port from 3100 to 3200 in their documentation and examples, I wonder if we should do the same for the operator?

Ref. grafana/tempo#809

Support replication factor

https://github.com/os-observability/tempo-operator/blob/main/apis/tempo/v1alpha1/microservices_types.go#L77

Docs: https://grafana.com/docs/tempo/latest/configuration/#ingester

As part of this ticket, we should learn how the replication factor relates to the number of ingester/distributor replicas and apply some validation based on that.

Run scorecard tests on CI

Generated code should be created by the build

There are several situations (some of which I will list below) where either building locally changes generated code which then has to be reverted before committing, or developers need to run generate targets manually, and then that code needs to be checked in.

In the first case, building an image on a developer's machine will change config/manager/kustomization.yaml and this change will need to be reverted before making a commit.

For the second, If this build is the same as the jaeger-operator or opentelemetry-collector-operator there are Makefile targets that such as api-docs that must be run manually before a commit and whatever files they change need to be added to the commit.

We should decide what is the best way to deal with generated code, and have this automated and done by the build where possible

Create API/CRD docs

Fix failed to read cluster seed file, related to gossip?

Created from https://github.com/os-observability/tempo-operator/pull/56/#issuecomment-1263647995

k logs tempo-simplest-ingester-0                                                                                                                                                          ploffay@fedora
level=info ts=2022-09-30T14:19:40.189855607Z caller=main.go:191 msg="initialising OpenTracing tracer"
level=warn ts=2022-09-30T14:19:40.190480636Z caller=config.go:155 msg="Local backend will not correctly retrieve traces with a distributed deployment unless all components have access to the same disk. You should probably be using object storage as a backend."
level=info ts=2022-09-30T14:19:40.190541339Z caller=main.go:106 msg="Starting Tempo" version="(version=, branch=HEAD, revision=fd5743d5d)"
level=info ts=2022-09-30T14:19:40.190706874Z caller=server.go:306 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
level=debug ts=2022-09-30T14:19:40.191743299Z caller=module_service.go:72 msg="module waiting for initialization" module=ingester waiting_for=server
level=info ts=2022-09-30T14:19:40.191765979Z caller=module_service.go:82 msg=initialising module=store
level=debug ts=2022-09-30T14:19:40.191767272Z caller=module_service.go:72 msg="module waiting for initialization" module=overrides waiting_for=server
level=debug ts=2022-09-30T14:19:40.191778596Z caller=module_service.go:72 msg="module waiting for initialization" module=usage-report waiting_for=memberlist-kv
ts=2022-09-30T14:19:40.19180243Z caller=memberlist_logger.go:74 level=debug msg="configured Transport is not a NodeAwareTransport and some features may not work as desired"
level=debug ts=2022-09-30T14:19:40.1917849Z caller=module_service.go:72 msg="module waiting for initialization" module=memberlist-kv waiting_for=server
level=info ts=2022-09-30T14:19:40.191801405Z caller=module_service.go:82 msg=initialising module=server
level=info ts=2022-09-30T14:19:40.191854654Z caller=module_service.go:82 msg=initialising module=memberlist-kv
level=info ts=2022-09-30T14:19:40.191868892Z caller=module_service.go:82 msg=initialising module=overrides
level=debug ts=2022-09-30T14:19:40.191882004Z caller=module_service.go:72 msg="module waiting for initialization" module=ingester waiting_for=store
level=debug ts=2022-09-30T14:19:40.191886806Z caller=module_service.go:72 msg="module waiting for initialization" module=usage-report waiting_for=server
level=debug ts=2022-09-30T14:19:40.191891367Z caller=module_service.go:72 msg="module waiting for initialization" module=ingester waiting_for=usage-report
level=info ts=2022-09-30T14:19:40.191895428Z caller=module_service.go:82 msg=initialising module=usage-report
level=debug ts=2022-09-30T14:19:40.192119498Z caller=module_service.go:72 msg="module waiting for initialization" module=ingester waiting_for=memberlist-kv
level=debug ts=2022-09-30T14:19:40.192442028Z caller=module_service.go:72 msg="module waiting for initialization" module=ingester waiting_for=overrides
level=info ts=2022-09-30T14:19:40.192452332Z caller=module_service.go:82 msg=initialising module=ingester
level=info ts=2022-09-30T14:19:40.192545729Z caller=ingester.go:328 msg="beginning wal replay"
level=warn ts=2022-09-30T14:19:40.192626314Z caller=rescan_blocks.go:24 msg="failed to open search wal directory" err="open /var/tempo/wal/search: no such file or directory"
level=info ts=2022-09-30T14:19:40.1929802Z caller=ingester.go:413 msg="wal replay complete"
level=info ts=2022-09-30T14:19:40.193022956Z caller=ingester.go:427 msg="reloading local blocks" tenants=0
level=debug ts=2022-09-30T14:19:40.193060457Z caller=tcp_transport.go:393 component="memberlist TCPTransport" msg=FinalAdvertiseAddr advertiseAddr=172.17.0.10 advertisePort=7946
level=info ts=2022-09-30T14:19:40.193078602Z caller=app.go:195 msg="Tempo started"
level=debug ts=2022-09-30T14:19:40.193384188Z caller=tcp_transport.go:393 component="memberlist TCPTransport" msg=FinalAdvertiseAddr advertiseAddr=172.17.0.10 advertisePort=7946
level=info ts=2022-09-30T14:19:40.193540931Z caller=lifecycler.go:576 msg="instance not found in ring, adding with no tokens" ring=ingester
level=debug ts=2022-09-30T14:19:40.19366803Z caller=lifecycler.go:412 msg="JoinAfter expired" ring=ingester
level=info ts=2022-09-30T14:19:40.193684836Z caller=lifecycler.go:416 msg="auto-joining cluster after timeout" ring=ingester
level=debug ts=2022-09-30T14:19:40.193870499Z caller=memberlist_client.go:842 msg="CAS attempt failed" err="no change detected" retry=true
ts=2022-09-30T14:19:40.600158068Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.1:14533"
level=debug ts=2022-09-30T14:19:41.194667986Z caller=broadcast.go:48 msg="Invalidating forwarded broadcast" key=collectors/ring version=2 oldVersion=1 content=[tempo-simplest-ingester-0] oldContent=[tempo-simplest-ingester-0]
level=debug ts=2022-09-30T14:19:41.599967326Z caller=broadcast.go:48 msg="Invalidating forwarded broadcast" key=collectors/usagestats_token version=2 oldVersion=1 content=[21165ff2-4a8d-415c-b522-2d418f840d6d] oldContent=[21165ff2-4a8d-415c-b522-2d418f840d6d]
ts=2022-09-30T14:19:50.194096438Z caller=memberlist_logger.go:74 level=debug msg="Failed to join 10.106.28.225:7946: dial tcp 10.106.28.225:7946: i/o timeout"
level=debug ts=2022-09-30T14:19:50.19411806Z caller=memberlist_client.go:542 msg="attempt to join memberlist cluster failed" retries=0 err="1 error occurred:\n\t* Failed to join 10.106.28.225:7946: dial tcp 10.106.28.225:7946: i/o timeout\n\n"
ts=2022-09-30T14:19:51.920216041Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with:  10.106.28.225:7946"
level=info ts=2022-09-30T14:19:51.921613817Z caller=memberlist_client.go:563 msg="joined memberlist cluster" reached_nodes=1
ts=2022-09-30T14:20:16.650978517Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.13:38772"
level=debug ts=2022-09-30T14:20:20.198355734Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:20:21.218157391Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-distributor-64d9455df-s8rrp-cd90913d 172.17.0.13:7946"
ts=2022-09-30T14:20:46.653822706Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.13:60750"
ts=2022-09-30T14:21:16.655474821Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.13:36336"
ts=2022-09-30T14:21:21.220424449Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-distributor-64d9455df-s8rrp-cd90913d 172.17.0.13:7946"
ts=2022-09-30T14:21:46.660109753Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.13:40554"
ts=2022-09-30T14:21:51.222943226Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-distributor-64d9455df-s8rrp-cd90913d 172.17.0.13:7946"
ts=2022-09-30T14:22:16.662041502Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.13:42536"
ts=2022-09-30T14:22:21.225571434Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-distributor-64d9455df-s8rrp-cd90913d 172.17.0.13:7946"
ts=2022-09-30T14:22:46.664086303Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.13:49796"
ts=2022-09-30T14:22:51.227770528Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-distributor-64d9455df-s8rrp-cd90913d 172.17.0.13:7946"
ts=2022-09-30T14:23:16.667031549Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.13:37000"
ts=2022-09-30T14:23:21.230080434Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-distributor-64d9455df-s8rrp-cd90913d 172.17.0.13:7946"
ts=2022-09-30T14:23:46.669832176Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.13:59392"
ts=2022-09-30T14:23:51.232216395Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-distributor-64d9455df-s8rrp-cd90913d 172.17.0.13:7946"
ts=2022-09-30T14:24:16.672053596Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.13:45336"
ts=2022-09-30T14:24:21.235233088Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-distributor-64d9455df-s8rrp-cd90913d 172.17.0.13:7946"
ts=2022-09-30T14:24:46.674743316Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.13:52216"
ts=2022-09-30T14:24:51.237137586Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-distributor-64d9455df-s8rrp-cd90913d 172.17.0.13:7946"
ts=2022-09-30T14:25:21.238253302Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-distributor-64d9455df-s8rrp-cd90913d 172.17.0.13:7946"
ts=2022-09-30T14:25:46.677824896Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.13:52612"
ts=2022-09-30T14:25:51.240523901Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-distributor-64d9455df-s8rrp-cd90913d 172.17.0.13:7946"
ts=2022-09-30T14:26:16.680735215Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.13:53366"

and

k logs tempo-simplest-distributor-64d9455df-s8rrp                                                                                                                                         ploffay@fedora
level=info ts=2022-09-30T14:19:40.596746275Z caller=main.go:191 msg="initialising OpenTracing tracer"
level=warn ts=2022-09-30T14:19:40.597338833Z caller=config.go:155 msg="Local backend will not correctly retrieve traces with a distributed deployment unless all components have access to the same disk. You should probably be using object storage as a backend."
level=info ts=2022-09-30T14:19:40.59739635Z caller=main.go:106 msg="Starting Tempo" version="(version=, branch=HEAD, revision=fd5743d5d)"
level=info ts=2022-09-30T14:19:40.59755628Z caller=server.go:306 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
ts=2022-09-30T14:19:40.598219047Z caller=memberlist_logger.go:74 level=debug msg="configured Transport is not a NodeAwareTransport and some features may not work as desired"
ts=2022-09-30T14:19:40Z level=info msg="OTel Shim Logger Initialized" component=tempo
level=debug ts=2022-09-30T14:19:40.598702442Z caller=tcp_transport.go:393 component="memberlist TCPTransport" msg=FinalAdvertiseAddr advertiseAddr=172.17.0.13 advertisePort=7946
level=debug ts=2022-09-30T14:19:40.599159339Z caller=tcp_transport.go:393 component="memberlist TCPTransport" msg=FinalAdvertiseAddr advertiseAddr=172.17.0.13 advertisePort=7946
level=debug ts=2022-09-30T14:19:40.599429759Z caller=module_service.go:72 msg="module waiting for initialization" module=memberlist-kv waiting_for=server
level=debug ts=2022-09-30T14:19:40.59944112Z caller=module_service.go:72 msg="module waiting for initialization" module=distributor waiting_for=memberlist-kv
level=debug ts=2022-09-30T14:19:40.599456055Z caller=module_service.go:72 msg="module waiting for initialization" module=overrides waiting_for=server
level=debug ts=2022-09-30T14:19:40.599472748Z caller=module_service.go:72 msg="module waiting for initialization" module=usage-report waiting_for=memberlist-kv
level=info ts=2022-09-30T14:19:40.599471231Z caller=module_service.go:82 msg=initialising module=server
level=info ts=2022-09-30T14:19:40.599848618Z caller=module_service.go:82 msg=initialising module=overrides
level=info ts=2022-09-30T14:19:40.599932225Z caller=module_service.go:82 msg=initialising module=memberlist-kv
level=debug ts=2022-09-30T14:19:40.600002146Z caller=module_service.go:72 msg="module waiting for initialization" module=distributor waiting_for=overrides
level=debug ts=2022-09-30T14:19:40.599483355Z caller=module_service.go:72 msg="module waiting for initialization" module=ring waiting_for=memberlist-kv
level=debug ts=2022-09-30T14:19:40.600022759Z caller=module_service.go:72 msg="module waiting for initialization" module=distributor waiting_for=ring
level=debug ts=2022-09-30T14:19:40.600035776Z caller=module_service.go:72 msg="module waiting for initialization" module=ring waiting_for=server
level=info ts=2022-09-30T14:19:40.600048934Z caller=module_service.go:82 msg=initialising module=ring
ts=2022-09-30T14:19:40.600105181Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with:  10.106.28.225:7946"
level=debug ts=2022-09-30T14:19:40.600042837Z caller=module_service.go:72 msg="module waiting for initialization" module=usage-report waiting_for=server
level=info ts=2022-09-30T14:19:40.600126554Z caller=module_service.go:82 msg=initialising module=usage-report
level=info ts=2022-09-30T14:19:40.600390736Z caller=ring.go:263 msg="ring doesn't exist in KV store yet"
level=debug ts=2022-09-30T14:19:40.600454536Z caller=module_service.go:72 msg="module waiting for initialization" module=distributor waiting_for=server
level=debug ts=2022-09-30T14:19:40.600466976Z caller=module_service.go:72 msg="module waiting for initialization" module=distributor waiting_for=usage-report
level=info ts=2022-09-30T14:19:40.600472611Z caller=module_service.go:82 msg=initialising module=distributor
level=debug ts=2022-09-30T14:19:40.600217851Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=info ts=2022-09-30T14:19:40.601214952Z caller=memberlist_client.go:534 msg="joined memberlist cluster" reached_nodes=1
ts=2022-09-30T14:19:40Z level=info msg="Starting GRPC server on endpoint 0.0.0.0:4317" component=tempo
ts=2022-09-30T14:19:40Z level=info msg="Starting HTTP server on endpoint 0.0.0.0:4318" component=tempo
level=info ts=2022-09-30T14:19:40.601617818Z caller=app.go:195 msg="Tempo started"
level=debug ts=2022-09-30T14:19:40.732758823Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:19:41.132443129Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:19:41.681883022Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:19:42.194120538Z caller=broadcast.go:48 msg="Invalidating forwarded broadcast" key=collectors/ring version=2 oldVersion=1 content=[tempo-simplest-ingester-0] oldContent=[tempo-simplest-ingester-0]
level=debug ts=2022-09-30T14:19:42.659487846Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:19:45.757344091Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:19:48.97743145Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:19:51.920294631Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.1:1052"
level=debug ts=2022-09-30T14:19:55.773106279Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:20:03.085275277Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:20:11.687389226Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:20:16.650921283Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-ingester-0-415e44f4 172.17.0.10:7946"
level=debug ts=2022-09-30T14:20:20.968592248Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:20:21.218151452Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.10:43550"
level=debug ts=2022-09-30T14:20:29.569443884Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:20:39.339520319Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:20:46.514699493Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:20:46.653767379Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-ingester-0-415e44f4 172.17.0.10:7946"
level=debug ts=2022-09-30T14:20:53.436620122Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:20:59.861291571Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:21:07.556225991Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:21:16.56551341Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:21:16.655451545Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-ingester-0-415e44f4 172.17.0.10:7946"
ts=2022-09-30T14:21:21.220587578Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.10:46902"
level=debug ts=2022-09-30T14:21:23.17462959Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:21:30.807467513Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:21:38.125382773Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:21:46.660132064Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-ingester-0-415e44f4 172.17.0.10:7946"
level=debug ts=2022-09-30T14:21:47.546521482Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:21:51.222929956Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.10:48774"
level=debug ts=2022-09-30T14:21:54.425518925Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:22:03.142497285Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:22:12.177534398Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:22:16.662034921Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-ingester-0-415e44f4 172.17.0.10:7946"
level=debug ts=2022-09-30T14:22:20.4129592Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:22:21.225575939Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.10:35950"
level=debug ts=2022-09-30T14:22:28.453617915Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:22:35.983651148Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:22:45.011511924Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:22:46.664028129Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-ingester-0-415e44f4 172.17.0.10:7946"
ts=2022-09-30T14:22:51.227848743Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.10:55922"
level=debug ts=2022-09-30T14:22:54.505850997Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:23:02.152875872Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:23:08.81944818Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:23:16.667033806Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-ingester-0-415e44f4 172.17.0.10:7946"
level=debug ts=2022-09-30T14:23:18.41451966Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:23:21.230237305Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.10:32942"
level=debug ts=2022-09-30T14:23:26.670546844Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:23:35.044342127Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:23:42.204347516Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:23:46.669814926Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-ingester-0-415e44f4 172.17.0.10:7946"
level=debug ts=2022-09-30T14:23:49.134606996Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:23:51.232256358Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.10:54938"
level=debug ts=2022-09-30T14:23:57.30725Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:24:06.642179511Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:24:15.590287499Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:24:16.672037336Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-ingester-0-415e44f4 172.17.0.10:7946"
ts=2022-09-30T14:24:21.235281435Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.10:33194"
level=debug ts=2022-09-30T14:24:24.338517333Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:24:33.315454392Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:24:43.125279232Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:24:46.674730818Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-ingester-0-415e44f4 172.17.0.10:7946"
ts=2022-09-30T14:24:51.237148969Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.10:41260"
level=debug ts=2022-09-30T14:24:51.592802076Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:24:59.061433093Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:25:07.736890017Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:25:14.32339676Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:25:21.23827594Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.10:55532"
level=debug ts=2022-09-30T14:25:23.212392331Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:25:31.264146985Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:25:38.162407144Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:25:46.677824901Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-ingester-0-415e44f4 172.17.0.10:7946"
level=debug ts=2022-09-30T14:25:47.85560542Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:25:51.24055589Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.10:35988"
level=debug ts=2022-09-30T14:25:57.102426314Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:26:06.58151438Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:26:14.407902317Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:26:16.680640322Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-ingester-0-415e44f4 172.17.0.10:7946"
ts=2022-09-30T14:26:21.243456675Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=172.17.0.10:36380"
level=debug ts=2022-09-30T14:26:23.630248676Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:26:31.760417029Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
level=debug ts=2022-09-30T14:26:41.011049209Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"
ts=2022-09-30T14:26:46.682623836Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: tempo-simplest-ingester-0-415e44f4 172.17.0.10:7946"
level=debug ts=2022-09-30T14:26:48.424980756Z caller=reporter.go:182 msg="failed to read cluster seed file" err="does not exist"

Publish operator image to GHCR

Publish the image per tag and merge to the main branch.

First unit test run failed

go: downloading github.com/modern-go/reflect2 v1.0.2
go: downloading golang.org/x/text v0.3.7
/home/runner/work/tempo-operator/tempo-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
panic: interface conversion: types.Type is *types.Interface, not *types.Named

https://github.com/os-observability/tempo-operator/runs/8304372293?check_suite_focus=true#step:4:38

Support node selector and tolerations

https://github.com/os-observability/tempo-operator/blob/main/apis/tempo/v1alpha1/microservices_types.go#L159

Add suppoort for retention spec

https://github.com/os-observability/tempo-operator/blob/main/api/v1alpha1/microservices_types.go#L46

The work depends on deploying the compactor

Add object storage configuration

Change local filesystem backed to object storage.

TODOs:

make target to deploy minio
object storage configuration in Tempo configmap
document how secret for the object storage should be created - similar to loki operator https://loki-operator.dev/docs/object_storage.md/

Ensure generated files are up to date

provide a compactor e2e test

Fix flaky e2e test

Sometimes the 03-run-check-services test of the E2E jaeger smoketest fails with wget: server returned error: HTTP/1.1 500 Internal Server Error. Restarting the job succeeds.

In an error case, the tempo-query logs contains error finding ingesters in Querier.SearchTagValues: empty ring:

{"level":"error","ts":1671726688.7815628,"caller":"app/http_handler.go:495","msg":"HTTP handler, Internal Server Error","error":"plugin error: rpc error: code = Unknown desc = error finding ingesters in Querier.SearchTagValues: empty ring\n","stacktrace":"github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleError\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:495\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).getServices\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:166\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2084\ngithub.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5\n\tgithub.com/opentracing-contrib/[email protected]/nethttp/server.go:154\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2084\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2084\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\tgithub.com/gorilla/[email protected]/mux.go:210\ngithub.com/jaegertracing/jaeger/cmd/query/app.additionalHeadersHandler.func1\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/additional_headers_handler.go:28\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2084\ngithub.com/jaegertracing/jaeger/pkg/bearertoken.PropagationHandler.func1\n\tgithub.com/jaegertracing/jaeger/pkg/bearertoken/http.go:51\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2084\ngithub.com/gorilla/handlers.CompressHandlerLevel.func1\n\tgithub.com/gorilla/[email protected]/compress.go:100\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2084\ngithub.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\tgithub.com/gorilla/[email protected]/recovery.go:78\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2916\nnet/http.(*conn).serve\n\tnet/http/server.go:1966"}

All pods are started successfully however:

    logger.go:42: 16:31:29 | smoketest-with-jaeger/3-run-check-services | NAME                                             READY   STATUS      RESTARTS   AGE
    logger.go:42: 16:31:29 | smoketest-with-jaeger/3-run-check-services | tempo-simplest-compactor-f5888c49-9pb5h          1/1     Running     0          46s
    logger.go:42: 16:31:29 | smoketest-with-jaeger/3-run-check-services | tempo-simplest-distributor-5ff56df548-b9d5n      1/1     Running     0          46s
    logger.go:42: 16:31:29 | smoketest-with-jaeger/3-run-check-services | tempo-simplest-ingester-0                        1/1     Running     0          46s
    logger.go:42: 16:31:29 | smoketest-with-jaeger/3-run-check-services | tempo-simplest-querier-5c77cd7b7c-gz9kc          1/1     Running     0          46s
    logger.go:42: 16:31:29 | smoketest-with-jaeger/3-run-check-services | tempo-simplest-query-frontend-74cd7948d7-rqss8   2/2     Running     0          46s
    logger.go:42: 16:31:29 | smoketest-with-jaeger/3-run-check-services | tracegen-4hbxw                                   0/1     Completed   0          37s

I think sometimes the wget command is executed before the ingester gets registered in the ring. I assume #149 and asserting that all pods are ready might fix this flaky test.

Logs of the error case:
https://github.com/andreasgerstmayr/tempo-operator/actions/runs/3759424603/jobs/6388971648 (ignore the green checkmark)

Improve retention duration showed in the cluster

kubectl get microservices.tempo.grafana.com simplest -o yaml                                                                                                                                                                                                                             ploffay@fedora
apiVersion: tempo.grafana.com/v1alpha1
kind: Microservices
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"tempo.grafana.com/v1alpha1","kind":"Microservices","metadata":{"annotations":{},"name":"simplest","namespace":"default"},"spec":{"storage":{"secret":"minio-test"},"template":{"queryFrontend":{"jaegerQuery":{"enabled":true}}}}}
  creationTimestamp: "2022-10-25T13:27:49Z"
  generation: 1
  name: simplest
  namespace: default
  resourceVersion: "378836"
  uid: 9973ed88-a90d-4fcf-81c2-553a0a6171e9
spec:
  limits:
    global:
      ingestion:
        ingestionBurstSizeBytes: 0
        ingestionRateLimitBytes: 0
        maxBytesPerTrace: 0
        maxTracesPerUser: 0
      query:
        maxSearchBytesPerTrace: 0
  retention:
    global:
      traces: 172800000000000
  storage:
    secret: minio-test
  storageSize: "0"
  template:
    queryFrontend:
      component: {}
      jaegerQuery:
        enabled: true
------------------------------------------------------

traces: 172800000000000 could be shown nicely with 48h.

cc) @frzifus

CR
operator config

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

grafana / tempo-operator Goto Github PK

tempo-operator's Introduction

Grafana Tempo operator

Features

Documentation

Deploy

Community

tempo-operator's People

Contributors

Stargazers

Watchers

Forkers

tempo-operator's Issues

Issue description

Steps to reproduce:

Recommend Projects

Recommend Topics

Recommend Org