Can't query traces with Grafana 10.1.5+ about qryn

[akvlad]

@Xantorero I have just inserted your issue into the next release issue.

[akvlad]

Tested the qryn collector. All the metrics, traces and logs get ingested successfully.
The environment:
latest version of qryn-collector: https://github.com/metrico/otel-collector/pkgs/container/qryn-otel-collector
The configuration:

receivers:
  zipkin:
  loki:
    protocols:
      http:
  hostmetrics:
    collection_interval: 15s
    initial_delay: 15s
    root_path:
    scrapers:
      cpu:
      disk:
      load:
processors:
  batch:
    send_batch_size: 10000
    timeout: 5s
  memory_limiter:
    check_interval: 2s
    limit_mib: 1800
    spike_limit_mib: 500

exporters:
  qryn:
    dsn: tcp://localhost:9000/qryn2?username=default
    clustered_clickhouse: true
    timeout: 10s
    sending_queue:
      queue_size: 100
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
    logs:
      format: json

extensions:
  health_check:
  pprof:
  zpages:
  memory_ballast:
    size_mib: 1000

service:
  extensions: [pprof, zpages, health_check]
  pipelines:
    logs:
      receivers: [loki]
      processors: [memory_limiter, batch]
      exporters: [qryn]
    metrics:
      receivers: [hostmetrics]
      processors: [memory_limiter, batch]
      exporters: [qryn]
    traces:
      receivers: [zipkin]
      processors: [memory_limiter, batch]
      exporters: [qryn]

please note exporters.qryn.clustered_clickhouse should be true if you have more than one clickhouse node behind a load balancer.

[Xantorero]

Sorry, my long post was a little bit chaotic with information. My problem is not with the data ingestion with Oryn-otel-collector into Clickhouse, all the data metrics, logs and traces are properly inserted. I have problem on the line:

Grafana [Tempo datasource] -> Qryn

I did some research and the issue is connected to Grafana version. Under version 10, so let's say the latest 9.5.13 everything works fine. Bit since Grafana version 10+ I can't query traces. The reason is that underlaying API calls from Grafana to Tempo datasource did change, I can see that in Qryn logs:

Grafana 9.5.13 calls to Qryn look like this:
"GET /api/search/tag/values"

But Grafana 10+ calls to Qryn like this:
"GET /api/v2/search/tag/name/values"

So basically if I use new Grafana releases I can't access the traces stored qith Qryn in Clickhouse.

[akvlad]

Currently the latest tested working version is Grafana v10.0.1 (5a30620b85) . It works ok.

[Xantorero]

You are right, 10.0.1 works fine - by 10+ I meant 10.1.5 and 10.2.0 that I have tested and they did not work. Do You have some plans regarding Grafana updates?

[Xantorero]

Thank You very much!

[lmangani]

Resolved in 3.x feel free to reopen if needed.