suggest changing the insert operation to directly write to the local table instead of the distributed table about qryn

[akvlad]

Hello @Tokoy
We don't ingest line by line. We have a timeout based microbatching. I believe the timeout is 0.1s.
We use distributed table in order to make fingerprint based distribution. It's important for clustering.
Feel free to try using our binary writer https://github.com/metrico/otel-collector . It's more suitable for high load writes.

[deathalt]

@akvlad From my observations 20-30 instances of qryn write to clickhouse is much more efficient than otel collector.

for some reason scaling collectors doesn't give such a boost for my case.

[Tokoy]

Hello @Tokoy We don't ingest line by line. We have a timeout based microbatching. I believe the timeout is 0.1s. We use distributed table in order to make fingerprint based distribution. It's important for clustering. Feel free to try using our binary writer https://github.com/metrico/otel-collector . It's more suitable for high load writes.

I achieve the distribution of fingerprints by using SLB or Nginx for even distribution, current rate is writing 500k records to the each local table per second.The distributed table itself is recommended for read operations only. If frequent write operations are performed, it can easily result in the related table being in a readonly state, leading to write failures.

Additionally, I have noticed that the main query for the time_series_dist table performs a full table scan each time. If the time_series table is very large, the retrieval time will be long and it will consume a significant amount of memory. Here is my query statement:
WITH sel_a AS ( SELECT samples.string AS string, samples.fingerprint AS fingerprint, samples.timestamp_ns AS timestamp_ns FROM cloki.samples_v3_dist AS samples WHERE ((samples.timestamp_ns >= 1704440225031000000) AND (samples.timestamp_ns <= 1704440525031000000)) AND (samples.fingerprint IN ( SELECT sel_1.fingerprint FROM ( SELECT fingerprint FROM cloki.time_series_gin WHERE (key = 'app') AND (val = 'gate-v1') ) AS sel_1 ANY INNER JOIN ( SELECT fingerprint FROM cloki.time_series_gin WHERE (key = 'cluster') AND (val = 'eu-prod') ) AS sel_2 ON sel_1.fingerprint = sel_2.fingerprint )) ORDER BY timestamp_ns DESC LIMIT 100 ) SELECT JSONExtractKeysAndValues(time_series.labels, 'String') AS labels, sel_a.* FROM sel_a ANY LEFT JOIN cloki.time_series_dist AS time_series ON sel_a.fingerprint = time_series.fingerprint ORDER BY labels DESC, timestamp_ns DESC

[akvlad]

@Tokoy I'm not sure I understand how to achieve even distribution of fingerprints using nginx.
Every request may have fingerprints for multiple servers.

The recommended number of fingerprints is 1M per shard. You may start experiencing OOM errors above the number.
Of course it depends on your clickhouse server RAM.

[lmangani]

Closing as stale. Please reopen anytime if needed or useful.