Very long "cold run" with AWS OT Java Agent about aws-otel-java-instrumentation

[anuraaga]

Hi @mat-rumian - the aws java agent is almost the same binary as the OpenTelemetry upstream one so it's really surprising if they have different performance. I tried with the upstream one a while ago and found it to have similar 40-50s overhead with normal functions, and around 10s with provisioned concurrency. Can you confirm the function configuration was the same when trying the two agents? I'll try to confirm on my end too.

[anuraaga]

@mat-rumian - I tried the upstream agent and had similar performance profiles to the AWS distro. No provisioning, ~60s, with provisioning, ~10s. This is unfortunately just the nature of doing significant bytecode manipulation at runtime I'm afraid.

One trick to try could be to disable instrumentation by default and only enable the ones you need, using -Dotel.instrumentation.common.default-enabled=false. For example, this enables only annotatinon processing

https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/suppressing-instrumentation.md#enable-manual-instrumentation-only

[drissamri]

Will this be improved going forward? Are there any plans to do more things at build time instead of runtime? (as Quarkus, Micronaut and such are doing)

[anuraaga]

@drissamri I have heard that Quarkus team is working on being able to run a java agent during compile time which would probably make the Java agent practical on lambda, even without provisioned concurrency! I'm not sure of a roadmap or any issues to follow related to it, my understanding is the effort just started.

[github-actions]

This issue is stale because it has been open 90 days with no activity. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled

[github-actions]

This issue was closed because it has been marked as stale for 30 days with no activity.