Comments (15)
mx-psi
commented on August 27, 2024
1
Hi @wyTrivail, thanks for taking the time to provide the pprof results back to us, this is very helpful. It looks like the problem still is the same we identified and tried to ameliorate in #190, but we still need to do some work to improve it.
We will try to get back with a fix in a timely manner. We are still working on setting up the performance tests to aid in reproducing and testing.
from aws-otel-collector.
wyTrivail
commented on August 27, 2024
1
@mx-psi sorry for that! closing. Thank you for the quick response all the time, let's keep in touch!
from aws-otel-collector.
mx-psi
commented on August 27, 2024
Thanks for reporting this. cc @ericmustin since this seems to affect the traces exporter only
from aws-otel-collector.
ericmustin
commented on August 27, 2024
@wyTrivail 👋 Hi there. i work with @mx-psi at Datadog.
We've made a PR to address some areas that we believe, after observing some profiles of the trace exporter under load, were written inefficiently. open-telemetry/opentelemetry-collector-contrib#1706
We're continuing to iterate here and still working on setting up an AWS test environment described here https://github.com/aws-observability/aws-otel-test-framework/blob/terraform/docs/get-performance-model.md#step-2-build-aws-otel-collector-package, that will allow us to validate some of the work we've done. However, we think this above PR should begin to address some of the issues you're seeing and didn't want to delay any potential improvement.
One thing worth getting some clarification on, is that we noticed there was a recent PR here that removed the batch processor from our tracing pipeline setup https://github.com/aws-observability/aws-otel-test-framework/pull/111/files#diff-a8ffbd6c07b2d581b17da61afc441dfcb1cbdb2bb03833dcaccff251a80d3f60L26
We'd like to understand a bit more the decision making process here, as we are currently recommending that the batch processor be used by end-users here and it's an essential part of ensuring that our trace related statistics are generated accurately. Removing it in tests would create a delta between the test environment and a typical production environment
from aws-otel-collector.
mxiamxia
commented on August 27, 2024
We'd like to understand a bit more the decision making process here, as we are currently recommending that the batch processor be used by end-users here and it's an essential part of ensuring that our trace related statistics are generated accurately. Removing it in tests would create a delta between the test environment and a typical production environment
Hi @mx-psi , Thanks for taking the response quickly. Since we want to do "apple to apple" comparison on all the Traces exporters. Some exporters doesn't apply Batch processor or with different batch parameters in the config. So we decided to remove the batch processor in the pipeline make sure we evaluate all the exporters in the same setup. We can also consider add batch back to all the exporters with the same batch parameters to run the perf result again.
Thanks.
from aws-otel-collector.
mx-psi
commented on August 27, 2024
I opened #190 to include our latest changes that should improve performance into the AWS OpenTelemetry Collector.
from aws-otel-collector.
ericmustin
commented on August 27, 2024
@mxiamxia @wyTrivail thanks for the explanation here, see pr by @mx-psi
from aws-otel-collector.
mx-psi
commented on August 27, 2024
@wyTrivail Do we consider this solved given the new report in #194? If I am not mistaken this test included #190, and the new numbers are as follows
| Test Case | Data Rate(tps) | Data Type | Instance Type | Avg CPU Usage (Percent Per Core) | Avg Memory Usage (Megabytes) |
|---|---|---|---|---|---|
| datadog_exporter_trace_mock | 100 | otlp | m5.2xlarge | 3.84 | 67.66 |
| datadog_exporter_trace_mock | 1000 | otlp | m5.2xlarge | 8.58 | 66.98 |
| datadog_exporter_trace_mock | 5000 | otlp | m5.2xlarge | 21.82 | 70.14 |
from aws-otel-collector.
wyTrivail
commented on August 27, 2024
@mx-psi thx for your patients!
we finally figured out the report in #194 is invalid as we added a batch span processor in the load generator recently, after removing the batching logic, we still see the abnormal cpu and mem usage: #200.
could you take a look?
from aws-otel-collector.
wyTrivail
commented on August 27, 2024
Below are some of my researches, hope it helps.
pprof memory results(looks like the calculator was creating too many objects?):
pprof file
profile pdf
profile alloc space pdf
pprof goroutine result (looks like there're too many goroutines):
memory dump (looks like the major consumption is in heap, while stack also takes around 250 mb):
[ec2-user@ip-10-0-101-91 tmp]$ sudo pmap -x 3624
3624: /opt/aws/aws-otel-collector/bin/aws-otel-collector --config /opt/aws/aws-otel-collector/etc/config.yaml
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 35928 22488 0 r-x-- aws-otel-collector
0000000002716000 34780 22876 0 r---- aws-otel-collector
000000000490d000 652 652 260 rw--- aws-otel-collector
00000000049b0000 312 152 152 rw--- [ anon ]
000000c000000000 2048 1900 1900 rw--- [ anon ]
000000c000200000 55296 54984 54984 rw--- [ anon ]
000000c003800000 4096 4096 4096 rw--- [ anon ]
000000c003c00000 737280 737280 737280 rw--- [ anon ]
000000c030c00000 2048 2048 2048 rw--- [ anon ]
000000c030e00000 4096 4096 4096 rw--- [ anon ]
000000c031200000 2048 2048 2048 rw--- [ anon ]
000000c031400000 14336 14336 14336 rw--- [ anon ]
000000c032200000 2048 2048 2048 rw--- [ anon ]
000000c032400000 14336 14336 14336 rw--- [ anon ]
000000c033200000 2048 2048 2048 rw--- [ anon ]
000000c033400000 8192 8192 8192 rw--- [ anon ]
000000c033c00000 2048 2048 2048 rw--- [ anon ]
000000c033e00000 10240 10240 10240 rw--- [ anon ]
000000c034800000 4096 4096 4096 rw--- [ anon ]
000000c034c00000 6144 6144 6144 rw--- [ anon ]
000000c035200000 6144 6144 6144 rw--- [ anon ]
000000c035800000 2048 2048 2048 rw--- [ anon ]
000000c035a00000 2048 2048 2048 rw--- [ anon ]
000000c035c00000 2048 2048 2048 rw--- [ anon ]
000000c035e00000 18432 18432 18432 rw--- [ anon ]
000000c037000000 8192 8192 8192 rw--- [ anon ]
000000c037800000 4096 4096 4096 rw--- [ anon ]
000000c037c00000 75776 75776 75776 rw--- [ anon ]
000000c03c600000 2048 2048 2048 rw--- [ anon ]
000000c03c800000 196608 196608 196608 rw--- [ anon ]
000000c048800000 2048 2048 2048 rw--- [ anon ]
000000c048a00000 40960 40960 40960 rw--- [ anon ]
000000c04b200000 2048 2048 2048 rw--- [ anon ]
000000c04b400000 16384 16384 16384 rw--- [ anon ]
000000c04c400000 2048 2048 2048 rw--- [ anon ]
000000c04c600000 135168 135168 135168 rw--- [ anon ]
000000c054a00000 2048 2048 2048 rw--- [ anon ]
000000c054c00000 2048 2048 2048 rw--- [ anon ]
000000c054e00000 2048 2048 2048 rw--- [ anon ]
000000c055000000 12288 12288 12288 rw--- [ anon ]
000000c055c00000 2048 2048 2048 rw--- [ anon ]
000000c055e00000 4096 4096 4096 rw--- [ anon ]
000000c056200000 2048 2048 2048 rw--- [ anon ]
000000c056400000 12288 12288 12288 rw--- [ anon ]
000000c057000000 2048 2048 2048 rw--- [ anon ]
000000c057200000 14336 14336 14336 rw--- [ anon ]
000000c058000000 2048 2048 2048 rw--- [ anon ]
000000c058200000 2048 2048 2048 rw--- [ anon ]
000000c058400000 2048 2048 2048 rw--- [ anon ]
000000c058600000 2048 2048 2048 rw--- [ anon ]
000000c058800000 2048 2048 2048 rw--- [ anon ]
000000c058a00000 8192 8192 8192 rw--- [ anon ]
000000c059200000 2048 2048 2048 rw--- [ anon ]
000000c059400000 2048 2048 2048 rw--- [ anon ]
000000c059600000 24576 24576 24576 rw--- [ anon ]
000000c05ae00000 16384 16384 16384 rw--- [ anon ]
000000c05be00000 2048 2048 2048 rw--- [ anon ]
000000c05c000000 159744 159744 159744 rw--- [ anon ]
000000c065c00000 2048 2048 2048 rw--- [ anon ]
000000c065e00000 2048 2048 2048 rw--- [ anon ]
000000c066000000 2048 2048 2048 rw--- [ anon ]
000000c066200000 6144 6144 6144 rw--- [ anon ]
000000c066800000 4096 4096 4096 rw--- [ anon ]
000000c066c00000 2048 2048 2048 rw--- [ anon ]
000000c066e00000 6144 6144 6144 rw--- [ anon ]
000000c067400000 10240 10240 10240 rw--- [ anon ]
000000c067e00000 2048 2048 2048 rw--- [ anon ]
000000c068000000 53248 53248 53248 rw--- [ anon ]
000000c06b400000 2048 2048 2048 rw--- [ anon ]
000000c06b600000 2048 2048 2048 rw--- [ anon ]
000000c06b800000 4096 4096 4096 rw--- [ anon ]
000000c06bc00000 4096 4096 4096 rw--- [ anon ]
000000c06c000000 8192 8192 8192 rw--- [ anon ]
000000c06c800000 2048 2048 2048 rw--- [ anon ]
000000c06ca00000 2048 2048 2048 rw--- [ anon ]
000000c06cc00000 4096 4096 4096 rw--- [ anon ]
000000c06d000000 2048 2048 2048 rw--- [ anon ]
000000c06d200000 2048 2048 2048 rw--- [ anon ]
000000c06d400000 2048 2048 2048 rw--- [ anon ]
000000c06d600000 2048 2048 2048 rw--- [ anon ]
000000c06d800000 4096 4096 4096 rw--- [ anon ]
000000c06dc00000 6144 6144 6144 rw--- [ anon ]
000000c06e200000 10240 10240 10240 rw--- [ anon ]
000000c06ec00000 4096 4096 4096 rw--- [ anon ]
000000c06f000000 4096 4096 4096 rw--- [ anon ]
000000c06f400000 2048 2048 2048 rw--- [ anon ]
000000c06f600000 12288 12288 12288 rw--- [ anon ]
000000c070200000 10240 10240 10240 rw--- [ anon ]
000000c070c00000 4096 4096 4096 rw--- [ anon ]
000000c071000000 2048 2048 2048 rw--- [ anon ]
000000c071200000 2048 2048 2048 rw--- [ anon ]
000000c071400000 16384 16384 16384 rw--- [ anon ]
000000c072400000 4096 4096 4096 rw--- [ anon ]
000000c072800000 26624 26624 26624 rw--- [ anon ]
000000c074200000 2048 2048 2048 rw--- [ anon ]
000000c074400000 12288 12288 12288 rw--- [ anon ]
000000c075000000 4096 4096 4096 rw--- [ anon ]
000000c075400000 2048 2048 2048 rw--- [ anon ]
000000c075600000 4096 4096 4096 rw--- [ anon ]
000000c075a00000 4096 4096 4096 rw--- [ anon ]
000000c075e00000 2048 2048 2048 rw--- [ anon ]
000000c076000000 4096 4096 4096 rw--- [ anon ]
000000c076400000 8192 8192 8192 rw--- [ anon ]
000000c076c00000 4096 4096 4096 rw--- [ anon ]
000000c077000000 2048 2048 2048 rw--- [ anon ]
000000c077200000 2048 2048 2048 rw--- [ anon ]
000000c077400000 2048 2048 2048 rw--- [ anon ]
000000c077600000 4096 4096 4096 rw--- [ anon ]
000000c077a00000 2048 2048 2048 rw--- [ anon ]
000000c077c00000 4096 4096 4096 rw--- [ anon ]
000000c078000000 6144 6144 6144 rw--- [ anon ]
000000c078600000 2048 2048 2048 rw--- [ anon ]
000000c078800000 2048 2048 2048 rw--- [ anon ]
000000c078a00000 2048 2048 2048 rw--- [ anon ]
000000c078c00000 2048 2048 2048 rw--- [ anon ]
000000c078e00000 20480 20480 20480 rw--- [ anon ]
000000c07a200000 2048 2048 2048 rw--- [ anon ]
000000c07a400000 4096 4096 4096 rw--- [ anon ]
000000c07a800000 2048 2048 2048 rw--- [ anon ]
000000c07aa00000 4096 4096 4096 rw--- [ anon ]
000000c07ae00000 2048 2048 2048 rw--- [ anon ]
000000c07b000000 8192 8192 8192 rw--- [ anon ]
000000c07b800000 4096 4096 4096 rw--- [ anon ]
000000c07bc00000 8192 8192 8192 rw--- [ anon ]
000000c07c400000 2048 2048 2048 rw--- [ anon ]
000000c07c600000 2048 2048 2048 rw--- [ anon ]
000000c07c800000 6144 6144 6144 rw--- [ anon ]
000000c07ce00000 8192 8192 8192 rw--- [ anon ]
000000c07d600000 2048 2048 2048 rw--- [ anon ]
000000c07d800000 18432 18432 18432 rw--- [ anon ]
000000c07ea00000 2048 2048 2048 rw--- [ anon ]
000000c07ec00000 4096 4096 4096 rw--- [ anon ]
000000c07f000000 8192 8192 8192 rw--- [ anon ]
000000c07f800000 12288 12288 12288 rw--- [ anon ]
000000c080400000 2048 2048 2048 rw--- [ anon ]
000000c080600000 4096 4096 4096 rw--- [ anon ]
000000c080a00000 2048 2048 2048 rw--- [ anon ]
000000c080c00000 6144 6144 6144 rw--- [ anon ]
000000c081200000 2048 2048 2048 rw--- [ anon ]
000000c081400000 6144 6144 6144 rw--- [ anon ]
000000c081a00000 2048 2048 2048 rw--- [ anon ]
000000c081c00000 8192 8192 8192 rw--- [ anon ]
000000c082400000 2048 2048 2048 rw--- [ anon ]
000000c082600000 8192 8192 8192 rw--- [ anon ]
000000c082e00000 2048 2048 2048 rw--- [ anon ]
000000c083000000 6144 6144 6144 rw--- [ anon ]
000000c083600000 2048 2048 2048 rw--- [ anon ]
000000c083800000 24576 24576 24576 rw--- [ anon ]
000000c085000000 2048 2048 2048 rw--- [ anon ]
000000c085200000 53248 53248 53248 rw--- [ anon ]
000000c088600000 2048 2048 2048 rw--- [ anon ]
000000c088800000 2048 2048 2048 rw--- [ anon ]
000000c088a00000 2048 2048 2048 rw--- [ anon ]
000000c088c00000 4096 4096 4096 rw--- [ anon ]
000000c089000000 2048 2048 2048 rw--- [ anon ]
000000c089200000 2048 2048 2048 rw--- [ anon ]
000000c089400000 2048 2048 2048 rw--- [ anon ]
000000c089600000 8192 8192 8192 rw--- [ anon ]
000000c089e00000 2048 2048 2048 rw--- [ anon ]
000000c08a000000 8192 8192 8192 rw--- [ anon ]
000000c08a800000 2048 2048 2048 rw--- [ anon ]
000000c08aa00000 4096 4096 4096 rw--- [ anon ]
000000c08ae00000 2048 2048 2048 rw--- [ anon ]
000000c08b000000 6144 6144 6144 rw--- [ anon ]
000000c08b600000 2048 2048 2048 rw--- [ anon ]
000000c08b800000 2048 2048 2048 rw--- [ anon ]
000000c08ba00000 6144 6144 6144 rw--- [ anon ]
000000c08c000000 2048 2048 2048 rw--- [ anon ]
000000c08c200000 2048 2048 2048 rw--- [ anon ]
000000c08c400000 2048 2048 2048 rw--- [ anon ]
000000c08c600000 20480 20480 20480 rw--- [ anon ]
000000c08da00000 2048 2048 2048 rw--- [ anon ]
000000c08dc00000 2048 2048 2048 rw--- [ anon ]
000000c08de00000 4096 4096 4096 rw--- [ anon ]
000000c08e200000 10240 10240 10240 rw--- [ anon ]
000000c08ec00000 2048 2048 2048 rw--- [ anon ]
000000c08ee00000 4096 4096 4096 rw--- [ anon ]
000000c08f200000 16384 16384 16384 rw--- [ anon ]
000000c090200000 14336 14336 14336 rw--- [ anon ]
000000c091000000 49152 49152 49152 rw--- [ anon ]
000000c094000000 2048 2048 2048 rw--- [ anon ]
000000c094200000 2048 2048 2048 rw--- [ anon ]
000000c094400000 2048 2048 2048 rw--- [ anon ]
000000c094600000 4096 4096 4096 rw--- [ anon ]
000000c094a00000 2048 2048 2048 rw--- [ anon ]
000000c094c00000 6144 6144 6144 rw--- [ anon ]
000000c095200000 4096 4096 4096 rw--- [ anon ]
000000c095600000 2048 2048 2048 rw--- [ anon ]
000000c095800000 2048 2048 2048 rw--- [ anon ]
000000c095a00000 4096 4096 4096 rw--- [ anon ]
000000c095e00000 2048 2048 2048 rw--- [ anon ]
000000c096000000 4096 4096 4096 rw--- [ anon ]
000000c096400000 2048 2048 2048 rw--- [ anon ]
000000c096600000 8192 8192 8192 rw--- [ anon ]
000000c096e00000 2048 2048 2048 rw--- [ anon ]
000000c097000000 10240 10240 10240 rw--- [ anon ]
000000c097a00000 6144 6144 6144 rw--- [ anon ]
000000c098000000 2048 2048 2048 rw--- [ anon ]
000000c098200000 2048 2048 2048 rw--- [ anon ]
000000c098400000 2048 2048 2048 rw--- [ anon ]
000000c098600000 2048 2048 2048 rw--- [ anon ]
000000c098800000 4096 4096 4096 rw--- [ anon ]
000000c098c00000 2048 2048 2048 rw--- [ anon ]
000000c098e00000 2048 2048 2048 rw--- [ anon ]
000000c099000000 2048 2048 2048 rw--- [ anon ]
000000c099200000 2048 2048 2048 rw--- [ anon ]
000000c099400000 4096 4096 4096 rw--- [ anon ]
000000c099800000 10240 10240 10240 rw--- [ anon ]
000000c09a200000 14336 14336 14336 rw--- [ anon ]
000000c09b000000 4096 4096 4096 rw--- [ anon ]
000000c09b400000 2048 2048 2048 rw--- [ anon ]
000000c09b600000 2048 2048 2048 rw--- [ anon ]
000000c09b800000 8192 8192 8192 rw--- [ anon ]
000000c09c000000 2048 2048 2048 rw--- [ anon ]
000000c09c200000 2048 2048 2048 rw--- [ anon ]
000000c09c400000 2048 2048 2048 rw--- [ anon ]
000000c09c600000 6144 6144 6144 rw--- [ anon ]
000000c09cc00000 8192 8192 8192 rw--- [ anon ]
000000c09d400000 10240 10240 10240 rw--- [ anon ]
000000c09de00000 2048 2048 2048 rw--- [ anon ]
000000c09e000000 6144 6144 6144 rw--- [ anon ]
000000c09e600000 2048 2048 2048 rw--- [ anon ]
000000c09e800000 10240 10240 10240 rw--- [ anon ]
000000c09f200000 2048 2048 2048 rw--- [ anon ]
000000c09f400000 16384 16384 16384 rw--- [ anon ]
000000c0a0400000 12288 12288 12288 rw--- [ anon ]
000000c0a1000000 2048 2048 2048 rw--- [ anon ]
000000c0a1200000 2048 2048 2048 rw--- [ anon ]
000000c0a1400000 2048 2048 2048 rw--- [ anon ]
000000c0a1600000 6144 6144 6144 rw--- [ anon ]
000000c0a1c00000 4096 4096 4096 rw--- [ anon ]
000000c0a2000000 2048 2048 2048 rw--- [ anon ]
000000c0a2200000 2048 2048 2048 rw--- [ anon ]
000000c0a2400000 2048 2048 2048 rw--- [ anon ]
000000c0a2600000 18432 18432 18432 rw--- [ anon ]
000000c0a3800000 2048 2048 2048 rw--- [ anon ]
000000c0a3a00000 4096 4096 4096 rw--- [ anon ]
000000c0a3e00000 6144 6144 6144 rw--- [ anon ]
000000c0a4400000 2048 2048 2048 rw--- [ anon ]
000000c0a4600000 2048 2048 2048 rw--- [ anon ]
000000c0a4800000 2048 2048 2048 rw--- [ anon ]
000000c0a4a00000 2048 2048 2048 rw--- [ anon ]
000000c0a4c00000 4096 4096 4096 rw--- [ anon ]
000000c0a5000000 2048 2048 2048 rw--- [ anon ]
000000c0a5200000 6144 6144 6144 rw--- [ anon ]
000000c0a5800000 2048 2048 2048 rw--- [ anon ]
000000c0a5a00000 2048 2048 2048 rw--- [ anon ]
000000c0a5c00000 10240 10240 10240 rw--- [ anon ]
000000c0a6600000 4096 4096 4096 rw--- [ anon ]
000000c0a6a00000 2048 2048 2048 rw--- [ anon ]
000000c0a6c00000 2048 2048 2048 rw--- [ anon ]
000000c0a6e00000 2048 2048 2048 rw--- [ anon ]
000000c0a7000000 6144 6144 6144 rw--- [ anon ]
000000c0a7600000 2048 2048 2048 rw--- [ anon ]
000000c0a7800000 8192 8192 8192 rw--- [ anon ]
000000c0a8000000 2048 2048 2048 rw--- [ anon ]
000000c0a8200000 2048 2048 2048 rw--- [ anon ]
000000c0a8400000 14336 14336 14336 rw--- [ anon ]
000000c0a9200000 2048 2048 2048 rw--- [ anon ]
000000c0a9400000 2048 2048 2048 rw--- [ anon ]
000000c0a9600000 4096 4096 4096 rw--- [ anon ]
000000c0a9a00000 4096 4096 4096 rw--- [ anon ]
000000c0a9e00000 6144 6144 6144 rw--- [ anon ]
000000c0aa400000 6144 6144 6144 rw--- [ anon ]
000000c0aaa00000 2048 2048 2048 rw--- [ anon ]
000000c0aac00000 2048 2048 2048 rw--- [ anon ]
000000c0aae00000 2048 2048 2048 rw--- [ anon ]
000000c0ab000000 20480 20456 20456 rw--- [ anon ]
000000c0ac400000 4096 4096 4096 rw--- [ anon ]
000000c0ac800000 10240 10240 10240 rw--- [ anon ]
000000c0ad200000 2048 2048 2048 rw--- [ anon ]
000000c0ad400000 16384 16384 16384 rw--- [ anon ]
000000c0ae400000 2048 2048 2048 rw--- [ anon ]
000000c0ae600000 4096 4096 4096 rw--- [ anon ]
000000c0aea00000 2048 2048 2048 rw--- [ anon ]
000000c0aec00000 2048 2048 2048 rw--- [ anon ]
000000c0aee00000 2048 2048 2048 rw--- [ anon ]
000000c0af000000 14336 14336 14336 rw--- [ anon ]
000000c0afe00000 2048 2048 2048 rw--- [ anon ]
000000c0b0000000 4096 4096 4096 rw--- [ anon ]
000000c0b0400000 2048 2048 2048 rw--- [ anon ]
000000c0b0600000 49152 49152 49152 rw--- [ anon ]
000000c0b3600000 4096 4096 4096 rw--- [ anon ]
000000c0b3a00000 2048 2048 2048 rw--- [ anon ]
000000c0b3c00000 2048 2048 2048 rw--- [ anon ]
000000c0b3e00000 2048 2048 2048 rw--- [ anon ]
000000c0b4000000 2048 2048 2048 rw--- [ anon ]
000000c0b4200000 2048 2048 2048 rw--- [ anon ]
000000c0b4400000 16384 16384 16384 rw--- [ anon ]
000000c0b5400000 2048 2048 2048 rw--- [ anon ]
000000c0b5600000 12288 12288 12288 rw--- [ anon ]
000000c0b6200000 2048 2048 2048 rw--- [ anon ]
000000c0b6400000 2048 2048 2048 rw--- [ anon ]
000000c0b6600000 2048 2048 2048 rw--- [ anon ]
000000c0b6800000 2048 2048 2048 rw--- [ anon ]
000000c0b6a00000 2048 2048 2048 rw--- [ anon ]
000000c0b6c00000 10240 10240 10240 rw--- [ anon ]
000000c0b7600000 2048 2048 2048 rw--- [ anon ]
000000c0b7800000 2048 2048 2048 rw--- [ anon ]
000000c0b7a00000 2048 2048 2048 rw--- [ anon ]
000000c0b7c00000 2048 2048 2048 rw--- [ anon ]
000000c0b7e00000 2048 2048 2048 rw--- [ anon ]
000000c0b8000000 4096 4096 4096 rw--- [ anon ]
000000c0b8400000 2048 2048 2048 rw--- [ anon ]
000000c0b8600000 61440 61440 61440 rw--- [ anon ]
000000c0bc200000 2048 2048 2048 rw--- [ anon ]
000000c0bc400000 38912 38912 38912 rw--- [ anon ]
000000c0bea00000 4096 4096 4096 rw--- [ anon ]
000000c0bee00000 67584 67584 67584 rw--- [ anon ]
000000c0c3000000 2048 2048 2048 rw--- [ anon ]
000000c0c3200000 16384 16384 16384 rw--- [ anon ]
000000c0c4200000 2048 2048 2048 rw--- [ anon ]
000000c0c4400000 106496 106496 106496 rw--- [ anon ]
000000c0cac00000 2048 2048 2048 rw--- [ anon ]
000000c0cae00000 8192 8192 8192 rw--- [ anon ]
000000c0cb600000 2048 2048 2048 rw--- [ anon ]
000000c0cb800000 88064 87656 87656 rw--- [ anon ]
000000c0d0e00000 2048 2048 2048 rw--- [ anon ]
000000c0d1000000 53248 53248 53248 rw--- [ anon ]
000000c0d4400000 12288 12288 12288 rw--- [ anon ]
000000c0d5000000 16384 16384 16384 rw--- [ anon ]
000000c0d6000000 2048 2048 2048 rw--- [ anon ]
000000c0d6200000 10240 10240 10240 rw--- [ anon ]
000000c0d6c00000 2048 2048 2048 rw--- [ anon ]
000000c0d6e00000 81920 81920 81920 rw--- [ anon ]
000000c0dbe00000 2048 2048 2048 rw--- [ anon ]
000000c0dc000000 34816 34816 34816 rw--- [ anon ]
000000c0de200000 2048 2048 2048 rw--- [ anon ]
000000c0de400000 6144 4912 4912 rw--- [ anon ]
000000c0dea00000 2048 0 0 rw--- [ anon ]
000000c0dec00000 2048 0 0 rw--- [ anon ]
000000c0dee00000 2048 0 0 rw--- [ anon ]
000000c0df000000 8192 0 0 rw--- [ anon ]
000000c0df800000 4096 0 0 rw--- [ anon ]
000000c0dfc00000 8192 0 0 rw--- [ anon ]
000000c0e0400000 4096 0 0 rw--- [ anon ]
000000c0e0800000 16384 0 0 rw--- [ anon ]
000000c0e1800000 2048 0 0 rw--- [ anon ]
000000c0e1a00000 2048 0 0 rw--- [ anon ]
000000c0e1c00000 2048 0 0 rw--- [ anon ]
000000c0e1e00000 8192 0 0 rw--- [ anon ]
000000c0e2600000 2048 0 0 rw--- [ anon ]
000000c0e2800000 4096 0 0 rw--- [ anon ]
000000c0e2c00000 12288 0 0 rw--- [ anon ]
000000c0e3800000 2048 0 0 rw--- [ anon ]
000000c0e3a00000 6144 0 0 rw--- [ anon ]
000000c0e4000000 12288 0 0 rw--- [ anon ]
000000c0e4c00000 4096 0 0 rw--- [ anon ]
000000c0e5000000 2048 0 0 rw--- [ anon ]
000000c0e5200000 2048 0 0 rw--- [ anon ]
000000c0e5400000 4096 0 0 rw--- [ anon ]
000000c0e5800000 2048 0 0 rw--- [ anon ]
000000c0e5a00000 2048 0 0 rw--- [ anon ]
000000c0e5c00000 2048 0 0 rw--- [ anon ]
000000c0e5e00000 4096 0 0 rw--- [ anon ]
000000c0e6200000 4096 0 0 rw--- [ anon ]
000000c0e6600000 2048 0 0 rw--- [ anon ]
000000c0e6800000 10240 0 0 rw--- [ anon ]
000000c0e7200000 2048 0 0 rw--- [ anon ]
000000c0e7400000 2048 0 0 rw--- [ anon ]
000000c0e7600000 2048 0 0 rw--- [ anon ]
000000c0e7800000 2048 0 0 rw--- [ anon ]
000000c0e7a00000 2048 0 0 rw--- [ anon ]
000000c0e7c00000 6144 0 0 rw--- [ anon ]
000000c0e8200000 6144 0 0 rw--- [ anon ]
000000c0e8800000 2048 0 0 rw--- [ anon ]
000000c0e8a00000 2048 0 0 rw--- [ anon ]
000000c0e8c00000 8192 0 0 rw--- [ anon ]
000000c0e9400000 8192 0 0 rw--- [ anon ]
000000c0e9c00000 2048 0 0 rw--- [ anon ]
000000c0e9e00000 4096 0 0 rw--- [ anon ]
000000c0ea200000 2048 0 0 rw--- [ anon ]
000000c0ea400000 2048 0 0 rw--- [ anon ]
000000c0ea600000 2048 0 0 rw--- [ anon ]
000000c0ea800000 12288 0 0 rw--- [ anon ]
000000c0eb400000 2048 0 0 rw--- [ anon ]
000000c0eb600000 2048 0 0 rw--- [ anon ]
000000c0eb800000 2048 0 0 rw--- [ anon ]
000000c0eba00000 2048 0 0 rw--- [ anon ]
000000c0ebc00000 10240 0 0 rw--- [ anon ]
000000c0ec600000 2048 0 0 rw--- [ anon ]
000000c0ec800000 2048 0 0 rw--- [ anon ]
000000c0eca00000 2048 0 0 rw--- [ anon ]
000000c0ecc00000 6144 0 0 rw--- [ anon ]
000000c0ed200000 2048 0 0 rw--- [ anon ]
000000c0ed400000 2048 0 0 rw--- [ anon ]
000000c0ed600000 4096 0 0 rw--- [ anon ]
000000c0eda00000 2048 0 0 rw--- [ anon ]
000000c0edc00000 4096 0 0 rw--- [ anon ]
000000c0ee000000 20480 0 0 rw--- [ anon ]
000000c0ef400000 2048 0 0 rw--- [ anon ]
000000c0ef600000 4096 0 0 rw--- [ anon ]
000000c0efa00000 2048 0 0 rw--- [ anon ]
000000c0efc00000 6144 0 0 rw--- [ anon ]
000000c0f0200000 2048 0 0 rw--- [ anon ]
000000c0f0400000 8192 0 0 rw--- [ anon ]
000000c0f0c00000 2048 0 0 rw--- [ anon ]
000000c0f0e00000 4096 0 0 rw--- [ anon ]
000000c0f1200000 2048 0 0 rw--- [ anon ]
000000c0f1400000 2048 0 0 rw--- [ anon ]
000000c0f1600000 2048 0 0 rw--- [ anon ]
000000c0f1800000 2048 0 0 rw--- [ anon ]
000000c0f1a00000 2048 0 0 rw--- [ anon ]
000000c0f1c00000 2048 0 0 rw--- [ anon ]
000000c0f1e00000 2048 0 0 rw--- [ anon ]
000000c0f2000000 4096 0 0 rw--- [ anon ]
000000c0f2400000 2048 0 0 rw--- [ anon ]
000000c0f2600000 12288 0 0 rw--- [ anon ]
000000c0f3200000 2048 0 0 rw--- [ anon ]
000000c0f3400000 4096 0 0 rw--- [ anon ]
000000c0f3800000 2048 0 0 rw--- [ anon ]
000000c0f3a00000 10240 0 0 rw--- [ anon ]
000000c0f4400000 2048 0 0 rw--- [ anon ]
000000c0f4600000 10240 0 0 rw--- [ anon ]
000000c0f5000000 2048 0 0 rw--- [ anon ]
000000c0f5200000 16384 0 0 rw--- [ anon ]
000000c0f6200000 2048 0 0 rw--- [ anon ]
000000c0f6400000 2048 0 0 rw--- [ anon ]
000000c0f6600000 6144 0 0 rw--- [ anon ]
000000c0f6c00000 24576 0 0 rw--- [ anon ]
000000c0f8400000 2048 0 0 rw--- [ anon ]
000000c0f8600000 8192 0 0 rw--- [ anon ]
000000c0f8e00000 2048 0 0 rw--- [ anon ]
000000c0f9000000 4096 0 0 rw--- [ anon ]
000000c0f9400000 2048 0 0 rw--- [ anon ]
000000c0f9600000 2048 0 0 rw--- [ anon ]
000000c0f9800000 2048 0 0 rw--- [ anon ]
000000c0f9a00000 43008 0 0 rw--- [ anon ]
000000c0fc400000 2048 0 0 rw--- [ anon ]
000000c0fc600000 6144 0 0 rw--- [ anon ]
000000c0fcc00000 2048 0 0 rw--- [ anon ]
000000c0fce00000 12288 0 0 rw--- [ anon ]
000000c0fda00000 2048 0 0 rw--- [ anon ]
000000c0fdc00000 43008 0 0 rw--- [ anon ]
000000c100600000 2048 0 0 rw--- [ anon ]
000000c100800000 2048 0 0 rw--- [ anon ]
000000c100a00000 2048 0 0 rw--- [ anon ]
000000c100c00000 8192 0 0 rw--- [ anon ]
000000c101400000 2048 0 0 rw--- [ anon ]
000000c101600000 40960 0 0 rw--- [ anon ]
000000c103e00000 2048 0 0 rw--- [ anon ]
000000c104000000 10240 0 0 rw--- [ anon ]
000000c104a00000 2048 0 0 rw--- [ anon ]
000000c104c00000 6144 0 0 rw--- [ anon ]
000000c105200000 2048 0 0 rw--- [ anon ]
000000c105400000 20480 0 0 rw--- [ anon ]
000000c106800000 2048 0 0 rw--- [ anon ]
000000c106a00000 100352 0 0 rw--- [ anon ]
000000c10cc00000 4096 0 0 rw--- [ anon ]
000000c10d000000 16384 0 0 rw--- [ anon ]
000000c10e000000 2048 0 0 rw--- [ anon ]
000000c10e200000 26624 0 0 rw--- [ anon ]
000000c10fc00000 2048 0 0 rw--- [ anon ]
000000c10fe00000 79872 0 0 rw--- [ anon ]
000000c114c00000 2048 0 0 rw--- [ anon ]
000000c114e00000 8192 0 0 rw--- [ anon ]
000000c115600000 2048 0 0 rw--- [ anon ]
000000c115800000 14336 0 0 rw--- [ anon ]
000000c116600000 2048 0 0 rw--- [ anon ]
000000c116800000 43008 0 0 rw--- [ anon ]
000000c119200000 2048 0 0 rw--- [ anon ]
000000c119400000 45056 0 0 rw--- [ anon ]
000000c11c000000 2048 0 0 rw--- [ anon ]
000000c11c200000 221184 0 0 rw--- [ anon ]
000000c129a00000 2048 0 0 rw--- [ anon ]
000000c129c00000 2048 0 0 rw--- [ anon ]
000000c129e00000 2048 0 0 rw--- [ anon ]
000000c12a000000 73728 0 0 rw--- [ anon ]
000000c12e800000 2048 0 0 rw--- [ anon ]
000000c12ea00000 55296 0 0 rw--- [ anon ]
000000c132000000 2048 0 0 rw--- [ anon ]
000000c132200000 4096 0 0 rw--- [ anon ]
000000c132600000 2048 0 0 rw--- [ anon ]
000000c132800000 24576 0 0 rw--- [ anon ]
00007ff3c4230000 63616 61220 61220 rw--- [ anon ]
00007ff3c8052000 33068 32860 32860 rw--- [ anon ]
00007ff3ca0a4000 5632 5560 5560 rw--- [ anon ]
00007ff3ca62e000 24808 24728 24728 rw--- [ anon ]
00007ff3cbe69000 68476 68444 68444 rw--- [ anon ]
00007ff3d014e000 3140 3132 3132 rw--- [ anon ]
00007ff3d0463000 3716 3696 3696 rw--- [ anon ]
00007ff3d080c000 44872 11180 11180 rw--- [ anon ]
00007ff3d33de000 263680 0 0 ----- [ anon ]
00007ff3e355e000 12 12 12 rw--- [ anon ]
00007ff3e3561000 293556 0 0 ----- [ anon ]
00007ff3f540e000 4 4 4 rw--- [ anon ]
00007ff3f540f000 36692 0 0 ----- [ anon ]
00007ff3f77e4000 4 4 4 rw--- [ anon ]
00007ff3f77e5000 4580 0 0 ----- [ anon ]
00007ff3f7c5e000 4 4 4 rw--- [ anon ]
00007ff3f7c5f000 508 0 0 ----- [ anon ]
00007ff3f7cde000 384 64 64 rw--- [ anon ]
00007ffc564b3000 132 16 16 rw--- [ stack ]
00007ffc5651d000 12 0 0 r---- [ anon ]
00007ffc56520000 8 4 0 r-x-- [ anon ]
ffffffffff600000 4 0 0 r-x-- [ anon ]
---------------- ------- ------- -------
total kB 5964852 3902460 3856700
from aws-otel-collector.
wyTrivail
commented on August 27, 2024
just an update, i will try to run the performance test with adding batch processor in collector as it's more like a real customer user case. will let you know the result.
from aws-otel-collector.
mx-psi
commented on August 27, 2024
👍, that sounds reasonable to me. Without batching we would make thousands of requests per second to our endpoints so it's likely that we would get rate-limited in any case in that situation.
In any case, we are in the process of opening one more PR which improves performance both with and without batching.
from aws-otel-collector.
ericmustin
commented on August 27, 2024
Things look much improved here, it seems? Was the Friday perf run a non batching or batching test?
- https://github.com/aws-observability/aws-otel-collector/blob/main/docs/performance_model.md#trace-tps-1000
- https://github.com/aws-observability/aws-otel-collector/blob/main/docs/performance_model.md#trace-tps-5000
from aws-otel-collector.
wyTrivail
commented on August 27, 2024
@ericmustin the Friday runs on the batching one, with batching we can see much better performance there.
Since this is a more realistic user case, i don't think this is an issue any more to block us to release the exporter. You can resolve this issue if you want, or you can use it to track the performance improvement.
At the same time, we are more than happy to merge any improvement you are going to make for the performance and run the performance test continuously.
from aws-otel-collector.
mx-psi
commented on August 27, 2024
@wyTrivail we have decided we want to close this issue and track further improvements internally to avoid confusion. Could you close it for us? (We don't have write permissions so we can't do it ourselves)
I will make sure to keep updating the version of our exporter used in the AWS to include performance-related and other improvements.
from aws-otel-collector.
Related Issues (20)
- /healthcheck doesn't work in ECS HOT 4
- Toolchain in Dockerfile too old to build AWS collector HOT 5
- spanmetricsprocessor (spanmetricsconnector) support HOT 3
- Add FluentForward receiver HOT 2
- ADOT EKS addon (v0.92.1-eksbuild.1) - Using otlphttp exporters when using the Add-on configuration schema HOT 3
- OTEL_OTLP_ENDPOINT seems to get ignored HOT 1
- Aws X-Ray sampling rules are being ignored HOT 4
- Not getting AWS Lambda service and the AWS Lambda function trace in custom OTEL collector HOT 3
- OpenTelemetry X-Ray: Traces without hierarchy HOT 2
- Add support for windowseventlog receiver for ADOT collector HOT 3
- ADOT EKS add-on documentation is missing important parts HOT 2
- Default image for autoinstrumentation dotnet is not setting the right path for Startup Hook HOT 2
- Attention: ADOT Collector upcoming breaking changes - third party exporter deprecation HOT 10
- err: open /opt/aws/aws-otel-collector/etc/extracfg.txt: no such file or directory HOT 2
- Error: NoCredentialProviders when using AWS X-Ray Exporter HOT 2
- ADOT EC2 not listening for traffic with IPv4, defaults to IPv6. Unable to communicate to the instance from anywhere. HOT 1
- CVE-2024-36129 in upstream OTEL Collector HOT 2
- AWS Distro for OpenTelemetry EKS add-on for kubernetes 1.30? HOT 18
- Can't filter out noisy events from Lambda due to outdated OTel collector version (v0.39.0, last released 3 years ago)
- NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors when using AWS X-Ray Exporter
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aws-otel-collector.