`otlp_config` has no effect about loki

[sandeepsukhani]

Hey @zarbis, sorry for the inconvenience and thanks for reporting the issue.
It seems there is some issue with applying the otlp_config set in the global limits_config, i.e. in the overall Loki config file.
It is working fine as expected when set as per tenant config. While I get this fixed, can you please give it a try with the per-tenant config and use that to avoid getting blocked? I will update here when the fix gets merged.

[JStickler]

@zarbis, updating your configuration is only one part of the implementation, how are you Attaching the structured metatdata to your log lines?

[zarbis]

@JStickler I expected it to be ingested by OTLP endpoint naturally due to how it's documented:

Structured Metadata: Anything which can’t be stored in Index labels and LogLine would be stored as Structured Metadata.

[JStickler]

So you're using the OpenTelemetry Collector, which means you must use the otlphttp exporter, which does not support structured metadata.

The docs need to be updated, as we have two OLTP solutions.

[zarbis]

Indeed I do. The entire Ingesting logs to Loki using OpenTelemetry Collector doc reads like:

1. run otlphttp exporter in your OTEL Collector
2. customize otlp_config in Loki (optional)

If otlphttp indeed is incompatible with structured metadata, why does the entire doc is literally the opposite story? Sounds really confusing...

Just in case, this is relevant part of my Collector config (I run entire LGTM stack through it so showing only logs part of it):

config:
  receivers:
        otlp:
          protocols:
            http: ...
  exporters:
    otlphttp/loki:
        endpoint: http://loki-gateway/otlp
        auth: ...
        sending_queue: ...
    loki:
        endpoint: http://loki-gateway/loki/api/v1/push
        auth: ...
        sending_queue: ...

  service:
    pipelines:
      logs:
        receivers:
        - otlp
        processors:
        - memory_limiter
        - attributes/loki
        - resource/loki
        - batch
        exporters:
        - debug/normal
        # - otlphttp/loki
        - loki

I've tried running otlphttp/loki while also disabling attributes/loki and resource/loki since I supposed they won't be needed in native OTLP mode.

[zarbis]

@sandeepsukhani I've moved otlp_config under runtimeConfig.overrides.<org> and it kinda works, but not really:

otlp_config:
  resource_attributes:
    attributes_config:
    - action: drop
      attributes:
      - k8s.pod.name
      - k8s.deployment.name
  log_attributes:
    - action: drop
      regex: .*
  scope_attributes:
    - action: drop
      regex: .*

log_attributes are dropped, scope_attributes I've had none to begin with, but resource_attributes are still present.
I'm not sure whether the difference in different treatment of resource_attributes and other ones, or regex vs attributes list, but labels I've tried to drop are still there:

logcli --org-id <org> series --from="$(date --utc --iso-8601=seconds --date='5 minutes ago')" --analyze-labels '{}'
2024/04/30 21:08:26 http://localhost:3100/loki/api/v1/series?end=1714500506642625448&match=%7B%7D&start=1714500206000000000
Total Streams:  1653
Unique Labels:  8

Label Name                 Unique Values  Found In Streams
k8s_pod_name               1608           1626
k8s_deployment_name        268            745
k8s_namespace_name         40             1626
__stream_shard__           5              23
exporter                   1              21
service_name               1              1653