Not seeing the metrics using the lambda layers or the traces about aws-otel-lambda

[anuraaga]

Hi @mfolivas - we're currently working on official support for JS with Lambda and can provide more examples for you as we get more done on it. Most of our lambda work has moved upstream including a newer version of the collector.

https://github.com/open-telemetry/opentelemetry-lambda

It would be great if you can stay tuned for this, including some layer build scripts for the JS SDK itself we'll provide.

[mfolivas]

Thanks @anuraaga , I am confused, I am using the following example: https://github.com/aws-observability/aws-otel-lambda/tree/main/extensions/aoc-extension ...does this mean that this is not working?

[mxiamxia]

The OTel metrics support in Lambda hasn't been certified yet (We're still working on it). To help us to understand your problem, could you attach your lambda logs (w/ the collector logs) here?

[mfolivas]

This is my package.json:

"dependencies": {
    "@aws/otel-aws-xray-id-generator": "^0.13.1",
    "@aws/otel-aws-xray-propagator": "^0.13.0",
    "@opentelemetry/api": "^0.18.1",
    "@opentelemetry/core": "^0.16.0",
    "@opentelemetry/exporter-collector-grpc": "^0.18.0",
    "@opentelemetry/instrumentation": "^0.18.0",
    "@opentelemetry/metrics": "^0.18.0",
    "@opentelemetry/node": "^0.17.0",
    "@opentelemetry/plugin-express": "^0.14.0",
    "@opentelemetry/plugin-http": "^0.18.0",
    "@opentelemetry/plugin-https": "^0.18.0",
    "@opentelemetry/tracing": "^0.17.0",
    "aws-xray-sdk-core": "^3.2.0"
  }

My serverless.yml:

service: aws-serverless-nodejs-opentelemetry

frameworkVersion: '2'

plugins:
  - serverless-pseudo-parameters

provider:
  name: aws
  runtime: nodejs12.x
  lambdaHashingVersion: 20201221

  stage: dev
  tracing: true # enable tracing
  iamRoleStatements:
    - Effect: "Allow" # xray permissions (required)
      Action:
        - "xray:PutTraceSegments"
        - "xray:PutTelemetryRecords"
      Resource:
        - "*"

functions:
  hello:
    handler: src/handler.hello
    layers:
      - arn:aws:lambda:#{AWS::Region}:#{AWS::AccountId}:layer:aws-otel-collector:1

Here are some of the errors that I am getting:

2021-03-24 00:35:15.435 (+00:00)        undefined       ERROR   Uncaught Exception      {"errorType":"Error","errorMessage":"Failed to load gRPC binary module because it was not installed for the current system\nExpected directory: node-v72-linux-x64-glibc\nFound: [node-v79-darwin-x64-unknown]\nThis problem can often be fixed by running \"npm rebuild\" on the current system\nOriginal error: Cannot find module '/var/task/node_modules/grpc/src/node/extension_binary/node-v72-linux-x64-glibc/grpc_node.node'\nRequire stack:\n- /var/task/node_modules/grpc/src/grpc_extension.js\n- /var/task/node_modules/grpc/src/client_interceptors.js\n- /var/task/node_modules/grpc/src/client.js\n- /var/task/node_modules/grpc/index.js\n- /var/task/node_modules/@opentelemetry/exporter-collector-grpc/build/src/util.js\n- /var/task/node_modules/@opentelemetry/exporter-collector-grpc/build/src/CollectorExporterNodeBase.js\n- /var/task/node_modules/@opentelemetry/exporter-collector-grpc/build/src/CollectorTraceExporter.js\n- /var/task/node_modules/@opentelemetry/exporter-collector-grpc/build/src/index.js\n- /var/task/tracer.js\n- /var/task/handler.js\n- /var/runtime/UserFunction.js\n- /var/runtime/index.js","code":"MODULE_NOT_FOUND","stack":["Error: Failed to load gRPC binary module because it was not installed for the current system","Expected directory: node-v72-linux-x64-glibc","Found: [node-v79-darwin-x64-unknown]","This problem can often be fixed by running \"npm rebuild\" on the current system","Original error: Cannot find module '/var/task/node_modules/grpc/src/node/extension_binary/node-v72-linux-x64-glibc/grpc_node.node'","Require stack:","- /var/task/node_modules/grpc/src/grpc_extension.js","- /var/task/node_modules/grpc/src/client_interceptors.js","- /var/task/node_modules/grpc/src/client.js","- /var/task/node_modules/grpc/index.js","- /var/task/node_modules/@opentelemetry/exporter-collector-grpc/build/src/util.js","- /var/task/node_modules/@opentelemetry/exporter-collector-grpc/build/src/CollectorExporterNodeBase.js","- /var/task/node_modules/@opentelemetry/exporter-collector-grpc/build/src/CollectorTraceExporter.js","- /var/task/node_modules/@opentelemetry/exporter-collector-grpc/build/src/index.js","- /var/task/tracer.js","- /var/task/handler.js","- /var/runtime/UserFunction.js","- /var/runtime/index.js","    at Object.<anonymous> (/var/task/node_modules/grpc/src/grpc_extension.js:53:17)","    at Module._compile (internal/modules/cjs/loader.js:999:30)","    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)","    at Module.load (internal/modules/cjs/loader.js:863:32)","    at Function.Module._load (internal/modules/cjs/loader.js:708:14)","    at Module.require (internal/modules/cjs/loader.js:887:19)","    at require (internal/modules/cjs/helpers.js:74:18)","    at Object.<anonymous> (/var/task/node_modules/grpc/src/client_interceptors.js:144:12)","    at Module._compile (internal/modules/cjs/loader.js:999:30)","    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)"]}
[aoc-lambda-extension] Received event: {
        "eventType": "INVOKE",
        "deadlineMs": 1616546121438,
        "requestId": "5d8ae3e9-7c98-4556-aeec-fb1e5123f207",
        "invokedFunctionArn": "arn:aws:lambda:us-east-1:999999:function:aws-serverless-nodejs-opentelemetry-dev-hello",
        "tracing": {
                "type": "X-Amzn-Trace-Id",
                "value": "Root=1-605a8941-1410784b6410ceb25c9443f7;Parent=7134a6be39dc918f;Sampled=0"
        }
}
[aoc-lambda-extension] Waiting for event...
2021-03-24 00:35:15.442 (+00:00)        5d8ae3e9-7c98-4556-aeec-fb1e5123f207    INFO    Calling counter
2021-03-24 00:35:15.444 (+00:00)        5d8ae3e9-7c98-4556-aeec-fb1e5123f207    INFO    emit metric with http request size 678.1309371386055 byte, /outgoing-http-call
2021-03-24 00:35:15.445 (+00:00)        5d8ae3e9-7c98-4556-aeec-fb1e5123f207    INFO    emit metric with return time 2, /outgoing-http-call, 200
2021-03-24 00:35:15.446 (+00:00)        5d8ae3e9-7c98-4556-aeec-fb1e5123f207    INFO    Counter called
END RequestId: 5d8ae3e9-7c98-4556-aeec-fb1e5123f207
REPORT RequestId: 5d8ae3e9-7c98-4556-aeec-fb1e5123f207  Duration: 18.34 ms      Billed Duration: 19 ms  Memory Size: 1024 MB    Max Memory Used: 143 MB Init Duration: 1745.93 ms

START RequestId: 7f80bbbe-5e4f-4837-8e15-61f9ba003f08 Version: $LATEST
[aoc-lambda-extension] Received event: {
        "eventType": "INVOKE",
        "deadlineMs": 1616546130411,
        "requestId": "7f80bbbe-5e4f-4837-8e15-61f9ba003f08",
        "invokedFunctionArn": "arn:aws:lambda:us-east-1:999999:function:aws-serverless-nodejs-opentelemetry-dev-hello",
        "tracing": {
                "type": "X-Amzn-Trace-Id",
                "value": "Root=1-605a894c-1e7eb5652431738828181a1d;Parent=208bdabb6ed3886a;Sampled=0"
        }
}
[aoc-lambda-extension] Waiting for event...
2021-03-24 00:35:24.412 (+00:00)        7f80bbbe-5e4f-4837-8e15-61f9ba003f08    INFO    Calling counter
2021-03-24 00:35:24.412 (+00:00)        7f80bbbe-5e4f-4837-8e15-61f9ba003f08    INFO    emit metric with http request size 907.7307594332686 byte, /outgoing-http-call
2021-03-24 00:35:24.412 (+00:00)        7f80bbbe-5e4f-4837-8e15-61f9ba003f08    INFO    emit metric with return time 0, /outgoing-http-call, 200
2021-03-24 00:35:24.412 (+00:00)        7f80bbbe-5e4f-4837-8e15-61f9ba003f08    INFO    Counter called
2021-03-24 00:35:24.416 (+00:00)        7f80bbbe-5e4f-4837-8e15-61f9ba003f08    DEBUG   items to be sent [
  {
    descriptor: {
      name: 'apiBytesSent',
      description: 'Metric for counting request payload size',
      unit: '1',
      metricKind: 0,
      valueType: 1
    },
    labels: { apiName: '/outgoing-http-call', statusCode: 200 },
    aggregator: SumAggregator {
      kind: 0,
      _current: 1585.8616965718743,
      _lastUpdateTime: [Array]
    },
    resource: Resource { attributes: [Object] },
    instrumentationLibrary: { name: 'aws-otel', version: undefined }
  },
  {
    descriptor: {
      name: 'latency',
      description: 'Metric for record request latency',
      unit: '1',
      metricKind: 2,
      valueType: 1
    },
    labels: { apiName: '/outgoing-http-call', statusCode: 200 },
    aggregator: HistogramAggregator {
      kind: 2,
      _boundaries: [Array],
      _current: [Object],
      _lastUpdateTime: [Array]
    },
    resource: Resource { attributes: [Object] },
    instrumentationLibrary: { name: 'aws-otel', version: undefined }
  }
]
[aoc-lambda-extension] Received event: {
        "eventType": "SHUTDOWN",
        "deadlineMs": 1616546126422,
        "requestId": "",
        "invokedFunctionArn": "",
        "tracing": {
                "type": "",
                "value": ""
        }
}
2021-03-24 00:35:24.422 (+00:00)        INFO    service/service.go:282  Received stop test request
2021-03-24 00:35:24.422 (+00:00)        INFO    service/service.go:447  Starting shutdown...
2021-03-24 00:35:24.422 (+00:00)        INFO    service/service.go:380  Stopping receivers...
2021-03-24 00:35:24.422 (+00:00)        INFO    service/service.go:386  Stopping processors...
2021-03-24 00:35:24.422 (+00:00)        INFO    builder/pipelines_builder.go:69 Pipeline is shutting down...    {"pipeline_name": "traces", "pipeline_datatype": "traces"}
2021-03-24 00:35:24.422 (+00:00)        INFO    builder/pipelines_builder.go:75 Pipeline is shutdown.   {"pipeline_name": "traces", "pipeline_datatype": "traces"}
2021-03-24 00:35:24.422 (+00:00)        INFO    builder/pipelines_builder.go:69 Pipeline is shutting down...    {"pipeline_name": "metrics", "pipeline_datatype": "metrics"}
2021-03-24 00:35:24.422 (+00:00)        INFO    builder/pipelines_builder.go:75 Pipeline is shutdown.   {"pipeline_name": "metrics", "pipeline_datatype": "metrics"}
2021-03-24 00:35:24.422 (+00:00)        INFO    service/service.go:392  Stopping exporters...
2021-03-24 00:35:24.422 (+00:00)        INFO    service/service.go:402  Stopping extensions...
2021-03-24 00:35:24.422 (+00:00)        INFO    service/service.go:469  Shutdown complete.
Error: failed to shutdown pipelines: failed to shutdown exporters: sync /dev/stderr: invalid argument
[aoc-lambda-extension] Received SHUTDOWN event
[aoc-lambda-extension] Exiting
END RequestId: 7f80bbbe-5e4f-4837-8e15-61f9ba003f08
REPORT RequestId: 7f80bbbe-5e4f-4837-8e15-61f9ba003f08  Duration: 12.15 ms      Billed Duration: 13 ms  Memory Size: 1024 MB    Max Memory Used: 143 MB

RequestId: 7f80bbbe-5e4f-4837-8e15-61f9ba003f08 Error: Runtime exited with error: exit status 129

[mfolivas]

I have even tried with the layers provided here and got the following error:

using the function:

functions:
  hello:
    handler: src/handler.hello
    layers:
       - arn:aws:lambda:#{AWS::Region}:297975325230:layer:opentelemetry-lambda-extension:8
     environment:
       OPENTELEMETRY_COLLECTOR_CONFIG_FILE: /var/task/collector.yaml

Loading config file: /var/task/collector.yaml
2021-03-24 00:54:43.806 (+00:00)        INFO    service/service.go:396  Starting InProcess Collector... {"Version": "", "GitHash": "", "NumCPU": 2}
2021-03-24 00:54:43.806 (+00:00)        INFO    service/service.go:240  Setting up own telemetry...
2021-03-24 00:54:43.806 (+00:00)        INFO    service/service.go:277  Loading configuration...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xb756f6]

goroutine 82 [running]:
go.opentelemetry.io/collector/config.validateReceivers(...)
        /Users/alrex/dev/lightstep/go/pkg/mod/go.opentelemetry.io/[email protected]/config/config.go:625
go.opentelemetry.io/collector/config.ValidateConfig(0x0, 0xc00039d680, 0xc000379200, 0xc0003791d0)
        /Users/alrex/dev/lightstep/go/pkg/mod/go.opentelemetry.io/[email protected]/config/config.go:486 +0x26
go.opentelemetry.io/collector/service.(*Application).setupConfigurationComponents(0xc00016b2c0, 0x318dcc0, 0xc0000bc010, 0xc00017fbd0, 0x0, 0x3)
        /Users/alrex/dev/lightstep/go/pkg/mod/go.opentelemetry.io/[email protected]/service/service.go:282 +0x195
go.opentelemetry.io/collector/service.(*Application).execute(0xc00016b2c0, 0x318dcc0, 0xc0000bc010, 0xc00017fbd0, 0x0, 0x0)
        /Users/alrex/dev/lightstep/go/pkg/mod/go.opentelemetry.io/[email protected]/service/service.go:414 +0x4d7
go.opentelemetry.io/collector/service.New.func1(0xc000488b00, 0xc00017fc80, 0x0, 0x1, 0x0, 0x0)
        /Users/alrex/dev/lightstep/go/pkg/mod/go.opentelemetry.io/[email protected]/service/service.go:154 +0x9f
github.com/spf13/cobra.(*Command).execute(0xc000488b00, 0xc00017fc60, 0x1, 0x1, 0xc000488b00, 0xc00017fc60)
        /Users/alrex/dev/lightstep/go/pkg/mod/github.com/spf13/[email protected]/command.go:842 +0x453
github.com/spf13/cobra.(*Command).ExecuteC(0xc000488b00, 0x2cfade0, 0xc00006a7b8, 0xc07319)
        /Users/alrex/dev/lightstep/go/pkg/mod/github.com/spf13/[email protected]/command.go:950 +0x349
github.com/spf13/cobra.(*Command).Execute(...)
        /Users/alrex/dev/lightstep/go/pkg/mod/github.com/spf13/[email protected]/command.go:887
go.opentelemetry.io/collector/service.(*Application).Run(...)
        /Users/alrex/dev/lightstep/go/pkg/mod/go.opentelemetry.io/[email protected]/service/service.go:467
main.(*InProcessCollector).start.func2(0xc000467240, 0xc00017fbe0)
        /Users/alrex/dev/opentelemetry-lambda-extension/collector.go:91 +0x69
created by main.(*InProcessCollector).start
        /Users/alrex/dev/opentelemetry-lambda-extension/collector.go:89 +0x261
EXTENSION       Name: opentelemetry-lambda-extension    State: Started  Events: []

[wangzlei]

we have released initial version of AWS managed OTel layers for Python, JS and Java. You can try them again.
https://github.com/aws-observability/aws-otel-lambda#aws-managed-opentelemetry-lambda-layers

[stephenhong]

The OTel metrics support in Lambda hasn't been certified yet (We're still working on it). To help us to understand your problem, could you attach your lambda logs (w/ the collector logs) here?

Hello team, just wanted to check if this is still being worked on. I'm interested in testing OTEL metrics capabilities with lambda.

[npavlovatsynamedia]

Hi,

AWS announced the following:

Metric support now available in AWS Distro for OpenTelemetry

Does that mean that I can use same lambda layer to send traces (like here) and metrics?

[bryan-aguilar]

@npavlovatsynamedia ,

Metric support in lambda layers is a WIP, please stay tuned for future announcements.

[charrea6]

Hi, any update on when lambda support might be available for metrics?

[vasireddy99]

Hello, Thank you so much for
The lambda support for metrics is still WIP, however custom lambda layers in java and python for metrics can be build from here. Hope the below examples may help.

• Working example for Python
• Working example for java

Closing this issue since the Metrics is not GA yet for opentelemetry-lambda and status is dependant on upstream's language SDK's are not GA for metrics.
opentelemetry-JS project dashboard for reference.

Please feel free to reopen or create another issue for any questions/concerns.