[NewRelic Exporter] Performance test anomaly about aws-otel-collector

[justinfoote]

I'm not convinced that there's actually a performance problem here, I think this anomaly may have been caused by background noise on the test machine. There has been a new performance test run (see #194).

There were no changes in the New Relic exporter between this report and this report but the numbers for the New Relic exporter changed from this:

Test Case	Data Rate(tps)	Data Type	Instance Type	Avg CPU Usage (Percent Per Core)	Avg Memory Usage (Megabytes)
newrelic_exporter_metric_mock	100	otlp	m5.2xlarge	0.04	59.12
newrelic_exporter_metric_mock	1000	otlp	m5.2xlarge	0.03	57.92
newrelic_exporter_metric_mock	5000	otlp	m5.2xlarge	0.04	57.94
newrelic_exporter_trace_mock	100	otlp	m5.2xlarge	23.10	65.86
newrelic_exporter_trace_mock	1000	otlp	m5.2xlarge	386.52	4294.13
newrelic_exporter_trace_mock	5000	otlp	m5.2xlarge	553.62	22009.65

To this:

Test Case	Data Rate	Data Type	Instance Type	Avg CPU Usage (Percent)	Avg Memory Usage (Megabytes)
newrelic_exporter_metric_mock	100	otlp	m5.2xlarge	0.04	57.22
newrelic_exporter_metric_mock	1000	otlp	m5.2xlarge	0.04	59.18
newrelic_exporter_metric_mock	5000	otlp	m5.2xlarge	0.04	57.79
newrelic_exporter_trace_mock	100	otlp	m5.2xlarge	2.74	61.71
newrelic_exporter_trace_mock	1000	otlp	m5.2xlarge	8.18	65.24
newrelic_exporter_trace_mock	5000	otlp	m5.2xlarge	28.06	72.28

[mx-psi]

It might be that the version bump I did in #190 included a change on the NewRelic exporter that fixed this for you (this last report is based off commit 7495b4b, so it would include my PR)

[a-feld]

@MrAlias

[elucus]

@wyTrivail can you please provide details on the 'realistic level' so we know what targets we are trying to hit?

/cc @alolita

[wyTrivail]

@justinfoote #194 has some problems that the load was not able to put onto collector, so that the cpu and memory looks good. could be possible you run the performance test locally?

[wyTrivail]

i will double confirm if that's true result of performance, and let you know.

[wyTrivail]

@mx-psi @justinfoote thx for your patients.

we finally figured out the report in #194 is invalid as we added a batch span processor in the load generator recently which decreased the tps upon collector , after removing the batching logic, we still see the abnormal cpu and mem usage: #200.

could you take a look?

[wyTrivail]

just an update, i will try to run the performance test with adding batch processor in collector as it's more like a real customer user case. will let you know the result.

[wyTrivail]

the performance test runs with batching produce much better result there: https://github.com/aws-observability/aws-otel-collector/blob/main/docs/performance_model.md.

Since this is a more realistic user case, i don't think this is an issue any more to block us to release the exporter. I'm closing this issue.

At the same time, we are more than happy to merge any improvement you are going to make for the performance and run the performance test continuously.

thx
Ying