Performance/Throughput Impact with auto instrumentation in Spring 5 applications about aws-otel-java-instrumentation

[stnor]

What sampler are you using? Why 100% sampling? That will have a perf impact.

I used -Dotel.traces.sampler.traceidratio=true -Dotel.traces.sampler.arg=0.005 and had perf issues (cpu util rose w/ 50%).

Switched to the parent based ratio sampler parentbased_traceidratio and that had a huge impact for me, but I am using a lot of internal spans.

[kurvatch]

@stnor parentbased_always_on is the sampler. I am testing things out with default configurations, it was expected there will be a performance impact with 100% sampling but was shocked to see more than 50% degradation of throughput. I am doing another round of testing with parentbased_traceidratio and -Dotel.traces.sampler.arg=0.25 for the same application.

[stnor]

25% is a very high sampling frequency in my experience.

[kurvatch]

Is there a standards recommendation available. It would defiantly help to publish benchmark performance with different samplers and ratios with a demo application interacting database and a message system

[anuraaga]

Hi @kurvatch - I agree that the performance impact seems much larger than we'd expect. Sampling rate is great for reducing load on backends, but we wouldn't expect such that much overhead at hundreds of QPS.

I have filed open-telemetry/opentelemetry-java-instrumentation#3047 as that repo is where the actual code is and the performance bottlenecks can be investigated.

[github-actions]

This issue is stale because it has been open 90 days with no activity. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled

[github-actions]

This issue was closed because it has been marked as stale for 30 days with no activity.