Quarkus opentelemetry extension WARNING when exporting to dynatrace about quarkus

[quarkus-bot]

/cc @brunobat (opentelemetry,tracing), @radcortez (opentelemetry,tracing)

[geoand]

How can I reproduce this?

[mnguetsa]

How can I reproduce this?

I am thinking how to best reproduce this because it depends on Dynatrace. Basically you need a quarkus application with the open telemetry extension and a dynatrace instance to which quarkus app will connect. My feeling is that the Quarkus opentelemetry extension is considering empty HTTP responses from Dyntrace API as a failure, hence the warning. But it is just a guess

[geoand]

My feeling is that the Quarkus opentelemetry extension is considering empty HTTP responses from Dyntrace API as a failure, hence the warning. But it is just a guess

Could very well be the case, but without being able to test, we can't be sure.

One more question: Does Dynatrace show the traces that Quarkus has warned failed?

[mnguetsa]

@geoand unfortunately I cannot see those traces. Has the Quarkus opentelemetry extension been proven to work with Dynatrace already ? Here is the doc from dynatrace how to integrate with Quarkus https://docs.dynatrace.com/docs/setup-and-configuration/technology-support/application-software/java/quarkus

[geoand]

Has the Quarkus opentelemetry extension been proven to work with Dynatrace already

That's a question for @brunobat

[mnguetsa]

Hi @geoand , I came up with a reproducer. First I realized that since the default is grpc, I need to change the exporter protocol in the code to http: quarkus.otel.exporter.otlp.traces.protocol = http The documentation states that two protocols are supported: grpc and http. The dynatrace API works over http/protobuf.

Here is the error I get now when starting the Quarkus application:
You can reproduce with this: https://github.com/mnguetsa/quarkus-reproducer. branch dynatrace. I also added more details on the reproducer section of this issue description.

2024-02-23 14:53:23,599 ERROR [io.qua.run.Application] (Quarkus Main Thread) Failed to start application (with profile [dev]): java.lang.RuntimeException: Failed to start quarkus
	at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
	at io.quarkus.runtime.Application.start(Application.java:101)
	at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:111)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:124)
	at io.quarkus.runner.GeneratedMain.main(Unknown Source)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at io.quarkus.runner.bootstrap.StartupActionImpl$1.run(StartupActionImpl.java:113)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: jakarta.enterprise.inject.CreationException: Error creating synthetic bean [Le6zQbzkojAYO_OiKIQWJf4lGa4]: jakarta.enterprise.inject.CreationException: Error creating synthetic bean [shT2wDm9BfI89PAXjVYmrqS0z6k]: java.lang.IllegalStateException: Unable to install OTLP Exporter
	at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.doCreate(Unknown Source)
	at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.create(Unknown Source)
	at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.create(Unknown Source)
	at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
	at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:32)
	at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
	at io.quarkus.arc.impl.ComputingCacheContextInstances.computeIfAbsent(ComputingCacheContextInstances.java:19)
	at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
	at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.get(Unknown Source)
	at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.get(Unknown Source)
	at io.quarkus.arc.impl.InstanceImpl.getBeanInstance(InstanceImpl.java:325)
	at io.quarkus.arc.impl.InstanceImpl.getInternal(InstanceImpl.java:309)
	at io.quarkus.arc.impl.InstanceImpl.get(InstanceImpl.java:190)
	at io.quarkus.arc.runtime.BeanContainerImpl.beanInstance(BeanContainerImpl.java:26)
	at io.quarkus.opentelemetry.runtime.tracing.intrumentation.InstrumentationRecorder.setupVertxTracer(InstrumentationRecorder.java:43)
	at io.quarkus.deployment.steps.OpenTelemetryProcessor$setupVertx126580776.deploy_0(Unknown Source)
	at io.quarkus.deployment.steps.OpenTelemetryProcessor$setupVertx126580776.deploy(Unknown Source)
	... 11 more
Caused by: jakarta.enterprise.inject.CreationException: Error creating synthetic bean [shT2wDm9BfI89PAXjVYmrqS0z6k]: java.lang.IllegalStateException: Unable to install OTLP Exporter
	at io.quarkus.opentelemetry.runtime.exporter.otlp.LateBoundBatchSpanProcessor_shT2wDm9BfI89PAXjVYmrqS0z6k_Synthetic_Bean.doCreate(Unknown Source)
	at io.quarkus.opentelemetry.runtime.exporter.otlp.LateBoundBatchSpanProcessor_shT2wDm9BfI89PAXjVYmrqS0z6k_Synthetic_Bean.create(Unknown Source)
	at io.quarkus.opentelemetry.runtime.exporter.otlp.LateBoundBatchSpanProcessor_shT2wDm9BfI89PAXjVYmrqS0z6k_Synthetic_Bean.create(Unknown Source)
	at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
	at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:32)
	at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
	at io.quarkus.arc.impl.ComputingCacheContextInstances.computeIfAbsent(ComputingCacheContextInstances.java:19)
	at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
	at io.quarkus.opentelemetry.runtime.exporter.otlp.LateBoundBatchSpanProcessor_shT2wDm9BfI89PAXjVYmrqS0z6k_Synthetic_Bean.get(Unknown Source)
	at io.quarkus.opentelemetry.runtime.exporter.otlp.LateBoundBatchSpanProcessor_shT2wDm9BfI89PAXjVYmrqS0z6k_Synthetic_Bean.get(Unknown Source)
	at io.quarkus.arc.impl.Instances.getBeanInstance(Instances.java:115)
	at io.quarkus.arc.impl.Instances.listOf(Instances.java:75)
	at io.quarkus.arc.impl.ListProvider.get(ListProvider.java:50)
	at io.quarkus.arc.impl.ListProvider.get(ListProvider.java:15)
	at io.quarkus.opentelemetry.runtime.AutoConfiguredOpenTelemetrySdkBuilderCustomizer_TracerProviderCustomizer_Bean.doCreate(Unknown Source)
	at io.quarkus.opentelemetry.runtime.AutoConfiguredOpenTelemetrySdkBuilderCustomizer_TracerProviderCustomizer_Bean.create(Unknown Source)
	at io.quarkus.opentelemetry.runtime.AutoConfiguredOpenTelemetrySdkBuilderCustomizer_TracerProviderCustomizer_Bean.create(Unknown Source)
	at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
	at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:32)
	at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
	at io.quarkus.arc.impl.ComputingCacheContextInstances.computeIfAbsent(ComputingCacheContextInstances.java:19)
	at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
	at io.quarkus.opentelemetry.runtime.AutoConfiguredOpenTelemetrySdkBuilderCustomizer_TracerProviderCustomizer_Bean.get(Unknown Source)
	at io.quarkus.opentelemetry.runtime.AutoConfiguredOpenTelemetrySdkBuilderCustomizer_TracerProviderCustomizer_Bean.get(Unknown Source)
	at io.quarkus.arc.impl.InstanceImpl.getBeanInstance(InstanceImpl.java:325)
	at io.quarkus.arc.impl.InstanceImpl$InstanceIterator.next(InstanceImpl.java:363)
	at io.quarkus.opentelemetry.runtime.OpenTelemetryRecorder$1.apply(OpenTelemetryRecorder.java:74)
	at io.quarkus.opentelemetry.runtime.OpenTelemetryRecorder$1.apply(OpenTelemetryRecorder.java:51)
	at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.createSynthetic(Unknown Source)
	... 30 more
Caused by: java.lang.IllegalStateException: Unable to install OTLP Exporter
	at io.quarkus.opentelemetry.runtime.exporter.otlp.OTelExporterRecorder$1.apply(OTelExporterRecorder.java:75)
	at io.quarkus.opentelemetry.runtime.exporter.otlp.OTelExporterRecorder$1.apply(OTelExporterRecorder.java:48)
	at io.quarkus.opentelemetry.runtime.exporter.otlp.LateBoundBatchSpanProcessor_shT2wDm9BfI89PAXjVYmrqS0z6k_Synthetic_Bean.createSynthetic(Unknown Source)
	... 63 more
Caused by: java.lang.IllegalArgumentException: Unsupported OTLP protocol http specified. Please check `quarkus.otel.exporter.otlp.traces.protocol` property
	at io.quarkus.opentelemetry.runtime.exporter.otlp.OTelExporterRecorder$1.createSpanExporter(OTelExporterRecorder.java:94)
	at io.quarkus.opentelemetry.runtime.exporter.otlp.OTelExporterRecorder$1.apply(OTelExporterRecorder.java:63)
	... 65 more

[mnguetsa]

UPDATE:
@geoand
@brunobat

When setting the protocol to http/protobuf the exception disappears. The documentation was not clear about it.

[quarkus.otel.exporter.otlp.traces.protocol](https://quarkus.io/guides/opentelemetry#quarkus-opentelemetry_quarkus-otel-exporter-otlp-traces-protocol)

OTLP defines the encoding of telemetry data and the protocol used to exchange data between the client and the server. Depending on the exporter, the available protocols will be different.

Currently, only grpc and http are allowed.

Maybe that should be adjusted.

Now that it is set up, sending traces over http/protobuf results in the following error: Failed to export spans. Server responded with HTTP status code 404. Error message: Unable to parse response body, HTTP status message: Not Found
Which is still surprising because I can contact the API normally with curl or similar without getting any HTTP 404

Which still leads to the question whether the integration of Quarkus with http/protobuf Otel exporter like for dynatrace in this case is already confirmed to be mature?

[mnguetsa]

Adding this so it could help someone.
When configuring the http traces endpoint, do not add the /v1/traces suffix . opentelemetry-java will append it automatically.
Here is the explanation in opentelemetry-java repo README:

The OTLP traces, metrics, and logs endpoint to connect to. Must be a URL with a scheme of either http or https based on the use of TLS.
 If protocol is http/protobuf the version and signal will be appended to the path (e.g. v1/traces, v1/metrics, or v1/logs). 
Default is http://localhost:4317 when protocol is grpc, and http://localhost:4318/v1/{signal} when protocol is http/protobuf.

Closing this issue.

[brunobat]

Correct @mnguetsa. Those paths are never referred in the quarkus documentation here: https://quarkus.io/guides/opentelemetry
Will soon improve the docs around the endpoint to help with that and the protocol.