Lambda traces not being sent to data-prepper about aws-otel-lambda

[anuraaga]

Hi @imanol-uribe - the issue may be that you need to remove the batch processor - we currently need additional support from the Lambda runtime to enable asynchronous flushing of data, so in the meantime all data needs to be flushed for each lambda request. This means the batch processor can't be used, which implies an asynchronous flush outside the scope of a lambda request, which generally wouldn't happen while the lambda runtime is frozen between requests.

Also to confirm, are you seeing the traces in the Lambda logs (via the logging exporter you have configured)? That will confirm your file is getting picked up.

[imanol-uribe]

Hi @anuraaga. I removed the batch processor and tried the memory-limiter processor also without success. Do you recommend any processors?
I see the X-Ray traces in the Lambda UI (when I select my Lambda in the Lambda dashboard and then click monitor).
When I trigger the lambda I can see using debug print statements, in the cloudwatch logs, that the file is in the path that I specified for the environment variables. Where else could the logging exporter show up?
Thanks for your help, I really appreciate it.

[anuraaga]

@imanol-uribe With the logging exporter, we would expect to see the actual payloads for the spans printed to the Cloudwatch logs. We'd expect this to work independent of the setup of the data prepper. Did you also try without any processors at all? The memory limiter also introduces a queue so may cause the flushing to not work correctly - it's not really needed in our current setup because we flush on every request.

Just to confirm, you've added debug logging to verify the path /var/task/collector.yaml/collector.template.yaml exists? I tried a custom configuration yesterday with the documented environment variable and it worked OK, and also if setting the environment variable to a non-existent path it fails properly. I can't see anything wrong with your config except the possibility of the memory-limiter.

[anuraaga]

By the way I sent open-telemetry/opentelemetry-lambda#128 so hopefully this will be a bit more transparent in the next release of the layer.

[imanol-uribe]

@anuraaga I have tried without the processor as well, and nothing related to open telemetry traces or spans appears in the Cloudwatch Logs.

Yes I add debug logging to verify the path /var/task/collector.yaml/collector.template.yaml exists.

That's interesting that it works on your end with the custom config file. Just to make sure, you are using this layer: arn:aws:lambda:us-west-2:901920570463:layer:aws-otel-java-wrapper-ver-1-2-0:2.

Thank you for the continued support.

[anuraaga]

We debugged offline and found this to be caused by open-telemetry/opentelemetry-lambda#88 - changing the runtime fixed it.