xray tracing stops after some hours about aws-otel-java-instrumentation

[rapphil]

Hi, does the traces that you find prior to this error message look correct?

Is there anything strange in theses traces, like high number of specific spans, or is is it deeply nested?

[DavidSchwarz2]

Hi rapphil,
the traces prior to the error are looking ok.
Could it be that there are problems with parallel processing? We are using kotlin coroutines to do things in parallel.

Here are the complete logs of one example call:

info TracesExporter {"kind": "exporter", "data_type": "traces", "name": "logging", "resource spans": 1, "spans": 19}
2023-07-03T09:40:25.135Z debug [email protected]/awsxray.go:68 TracesExporter {"kind": "exporter", "data_type": "traces", "name": "awsxray", "type": "awsxray", "name": "awsxray", "#spans": 7}
2023-07-03T09:40:25.135Z debug [email protected]/awsxray.go:80 request: {
TraceSegmentDocuments: [
"{\"name\":\"<snip...s3 call>\"type\":\"subsegment\"}\n",
"{\"name\":\"<snip...s3 call>",
"{\"name\":\"<snip...database call>",
"{\"name\":\"<snip...>",
"{\"name\":\"HealthController.health\",\"id\":\"d2336dcff1eec3d9\",\"start_time\":1688377221.398077,\"service\":{\"version\":\"7a083a57c8ec9cecb2bcc7b6e3d64ca29332a1e44ffc331e99043c9f649435a6\"},\"origin\":\"AWS::ECS::Fargate\",\"trace_id\":\"1-64a28428-03698151c619e1014355530a\",\"end_time\":1688377221.4458563,\"fault\":false,\"error\":false,\"throttle\":false,\"aws\":{\"cloudwatch_logs\":[{\"log_group\":\"<snip...>\"parent_id\":\"e7f85d8dc4ed1cf1\",\"type\":\"subsegment\"}\n",
"{\"name\":\"<snip...>"type\":\"subsegment\"}\n",
"{\"name\":\"<snip...>\"type\":\"subsegment\"}\n"
]
} {"kind": "exporter", "data_type": "traces", "name": "awsxray"}
2023-07-03T09:40:25.155Z debug [email protected]/awsxray.go:90 response: {
UnprocessedTraceSegments: [
{
ErrorCode: "InvalidSegment",
Id: "09eb84f2fd77cb63",
Message: "Trace Size Exceeded. Reduce the number of segments per trace and the size of your segment documents."
},
{
ErrorCode: "InvalidSegment",
Id: "e4032e122cd6575b",
Message: "Trace Size Exceeded. Reduce the number of segments

[atshaw43]

This can occur by you having a large number of traces and you get rate limited.
Any idea how many spans and traces you are creating per second?
Can you try a percentage based sampling rate instead of always on?

[DavidSchwarz2]

Hi,
I tried it with OTEL_TRACES_SAMPLER "traceidratio" and OTEL_TRACES_SAMPLER_ARG “0.25”.
This had no effect.
I tried the plain aws sdk with the aws x-ray daemon and it seems that there are problems with multithreading.
I removed our multithreading code for the test and it seems to work.
I will try the otel solution without multithreading and see if it is a difference.
(We are using kotlin coroutines and java parallelStream).

[atshaw43]

This looks like a bug around instrumentation then.
Can you make a minimal code sample that reproduces the issue?

[github-actions]

This issue is stale because it has been open 90 days with no activity. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled

[github-actions]

This issue was closed because it has been marked as stale for 30 days with no activity.