aws-observability / aws-otel-php Goto Github PK

Flaky warning while calling /outgoing-http-call endpoint in integration test workflow for root scope not being detached correctly. This warning was newly implemented in this upstream PR.

OpenTelemetry PHP recommendation is to place $scope->end and $scope->detach calls in a finally block to ensure all spans are ended and their scopes are always detached correctly.

Hi,

First, let me say I'm super-excited about the potential of this project, and thanks to the contributors for their efforts to date. As a PHP developer running on AWS Lambda, I'm seeing this as a fantastic opportunity to fill an important gap.

I'm currently trying to get open-telemetry running on PHP on Lambda without success. I am hoping someone here can help me get to the bottom of it.

Environment

I'm running Bref on AWS Lambda with php 7.4 (yeah, I know) and open-telemetry 0.0.17, as follows:

    "open-telemetry/contrib-aws": "0.0.17",
    "open-telemetry/opentelemetry": "0.0.17",
    "open-telemetry/opentelemetry-php-contrib": "0.0.17",

Changes from the example

I notice that the latest 0.0.17 of open-telemetry has some changes since 0.0.14 that make the examples break, so I had to change a few things.

The line:

        $spanProcessor = new SimpleSpanProcessor(new OTLPExporter());

no longer works because OpenTelemetry\Contrib\OtlpGrpc\Exporter has been removed.

I replaced it with this, which I think this is equivalent.

        $transport = (new GrpcTransportFactory())->create(
            'http://127.0.0.1:4317' . OtlpUtil::method(Signals::TRACE)
        );
        $exporter = new OpenTelemetry\Contrib\Otlp\SpanExporter($transport);
        $spanProcessor = new SimpleSpanProcessor($exporter);

Collector layer

I've built a custom layer for the collector as described here.

My collector config is as follows:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  awsxray:
    region: ap-southeast-2
  awsemf:
    region: ap-southeast-2
  logging:
    loglevel: debug

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [logging, awsxray]
    metrics:
      receivers: [otlp]
      exporters: [awsemf]
  telemetry:
    logs:
      level: debug

Glueing OpenTelemetry + Xray together

I've also built a class in PHP to enclose all the openTelemetry+AWS code together for easy viewing here that puts all the OpenTelemetry code in one place.

IAM

And I added a IAM Policy to allow my Lambda execution environment access to write traces:

    Type: 'AWS::IAM::Policy'
    Properties:
      PolicyName: "${self:service}-${sls:stage}-open-telemetry"
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          Effect: Allow
          Action:
            - "logs:PutLogEvents"
            - "logs:CreateLogGroup"
            - "logs:CreateLogStream"
            - "logs:DescribeLogStreams"
            - "logs:DescribeLogGroups"
            - "xray:PutTraceSegments"
            - "xray:PutTelemetryRecords"
            - "xray:GetSamplingRules"
            - "xray:GetSamplingTargets"
            - "xray:GetSamplingStatisticSummaries"
            - "ssm:GetParameters"
          Resource:
            - "*"
      Roles:
        - !Ref IAMRoleLambda

Result

With all this in place, I get some logs but nothing shown visually in XRay.

I can see logs such as this, which I presume are generated from the collector.

2023-01-08T11:43:42 {"level":"info","ts":1673178222.7031276,"msg":"TracesExporter","kind":"exporter","data_type":"traces","name":"logging","#spans":1}
2023-01-08T11:43:42 {"level":"info","ts":1673178222.7031856,"msg":"ResourceSpans #0\nResource SchemaURL: \nResource attributes:\n     -> faas.name: Str(XXXXX)\n     -> faas.version: Str($LATEST)\n     -> cloud.region: Str(ap-southeast-2)\n     -> cloud.provider: Str(aws)\nScopeSpans #0\nScopeSpans SchemaURL: \nInstrumentationScope io.opentelemetry.contrib.php \nSpan #0\n    Trace ID       : 63baac6e38c7d4ad3a7b755834919a73\n    Parent ID      : 84bfa37214deb991\n    ID             : d810f21b345baa63\n    Name           : XXXX::YYYY::ZZZZ\n    Kind           : Internal\n    Start time     : 2023-01-08 11:43:42.607256547 +0000 UTC\n    End time       : 2023-01-08 11:43:42.695558894 +0000 UTC\n    Status code    : Unset\n    Status message : \nAttributes:","kind":"exporter","data_type":"traces","name":"logging"}

I can confirm the "Trace ID" in that payload matches the Lambda invocation trace ID (different format but same number). And if I misconfigure the open-telemetry transport then I get a connection error, so I know it's connecting :-)

When I look in AWS X-Ray's UI, I see my Lambda invocation but no subsequent PHP trace.

I'm not sure how to debug this further to work out whether it's open-telemetry or the collector or something-something AWS.

Thanks in advance, Scott

Hi again,

I've been reading that X-Ray has a dedicated set of annotation fields for tracing SQL:
https://docs.aws.amazon.com/xray/latest/devguide/xray-api-segmentdocuments.html#api-segmentdocuments-sql

I'm connecting a PHP app to an RDS Aurora (MySQL) database and I'd like to trace the SQL calls, and ideally see RDS as a node in the service map.

I currently have my SQL statements added as standard annotations, however ideallyI could use the x-ray sql segment fields, and ideally that would show RDS as a node in X-Ray.

I'm currently just doing this:

$span->setAttributes(['sql' => $query, 'params' => $params]);

It works, but I don't get any sort of visibility into DB usage, load, etc.

Not sure if this is a supported feature I've missed, or a feature request for future.

Thanks, Scott

Flaky warning while calling /outgoing-http-call endpoint in integration test workflow for root scope not being detached correctly. This warning was newly implemented in this upstream PR.

OpenTelemetry PHP recommendation is to place $scope->end and $scope->detach calls in a finally block to ensure all spans are ended and their scopes are always detached correctly.