aws-observability / aws-otel-js Goto Github PK
View Code? Open in Web Editor NEWAWS Distro for OpenTelemetry JavaScript SDK
Home Page: https://aws-otel.github.io/
License: Apache License 2.0
AWS Distro for OpenTelemetry JavaScript SDK
Home Page: https://aws-otel.github.io/
License: Apache License 2.0
Each package should have a package-level release notes folder, with a change log for each release.
Duplicate of: aws-observability/aws-otel-java-instrumentation#5
Hello, I am a little confused on what the intended usage pattern for the @opentelemetry/resource-detector-aws package.
npm shows this code snippet
import { detectResources } from '@opentelemetry/resources';
import { awsEc2Detector } from '@opentelemetry/resource-detector-aws'
const resource = await detectResources({
detectors: [awsEc2Detector],
})
const tracerProvider = new NodeTracerProvider({ resource });
However the await
functions means this cannot be at the top level in a js file and instead would need to be inside an async function but then we cannot guarantee this code executes before the servers require statements which is necessary for instrumentations to function
How is the module intended to be used?
So I'm trying to get instrumentation working and wondering why my console looks like
context(extract): {"_currentContext":{}}
carrier(extract): {"host":"localhost:8083","user-agent":"HTTPie/2.3.0","accept-encoding":"gzip, deflate","accept":"application/json, */*;q=0.5","connection":"keep-alive","content-type":"application/json","content-length":"33"}
traceHeader(extract): undefined
spanContext(extract): {"traceId":"0","spanId":"0","traceFlags":0}
context(inject): {"_currentContext":{}}
spanContext(inject): undefined
context(inject): {"_currentContext":{}}
spanContext(inject): undefined
context(inject): {"_currentContext":{}}
spanContext(inject): undefined
got /get
context(inject): {"_currentContext":{}}
spanContext(inject): undefined
context(inject): {"_currentContext":{}}
spanContext(inject): undefined
context(inject): {"_currentContext":{}}
spanContext(inject): undefined
context(inject): {"_currentContext":{}}
spanContext(inject): undefined
in spite of the fact that I've turned the log level for the NodeTracerProvider
all the way to ERROR
. Eventually after thinking I must be crazy for a while, I try a git grep
and
packages/opentelemetry-propagator-aws-xray/src/AWSXRayPropagator.ts:66: console.log('context(inject): ' + JSON.stringify(context));
packages/opentelemetry-propagator-aws-xray/src/AWSXRayPropagator.ts:68: console.log('spanContext(inject): ' + JSON.stringify(spanContext));
packages/opentelemetry-propagator-aws-xray/src/AWSXRayPropagator.ts:72: console.log('inject traceId: ' + otTraceId);
packages/opentelemetry-propagator-aws-xray/src/AWSXRayPropagator.ts:96: console.log('inject traceHeader: ' + traceHeader);
packages/opentelemetry-propagator-aws-xray/src/AWSXRayPropagator.ts:99: console.log('carrier(inject): ' + JSON.stringify(carrier));
packages/opentelemetry-propagator-aws-xray/src/AWSXRayPropagator.ts:103: console.log('context(extract): ' + JSON.stringify(context));
packages/opentelemetry-propagator-aws-xray/src/AWSXRayPropagator.ts:105: console.log('spanContext(extract): ' + JSON.stringify(spanContext));
packages/opentelemetry-propagator-aws-xray/src/AWSXRayPropagator.ts:118: console.log('carrier(extract): ' + JSON.stringify(carrier));
packages/opentelemetry-propagator-aws-xray/src/AWSXRayPropagator.ts:119: console.log('traceHeader(extract): ' + traceHeader);
packages/opentelemetry-propagator-aws-xray/src/AWSXRayPropagator.ts:153: console.log('parsedTraceId(extract): ' + parsedTraceId);
packages/opentelemetry-propagator-aws-xray/src/AWSXRayPropagator.ts:154: console.log('parsedSpanId(extract): ' + parsedSpanId);
packages/opentelemetry-propagator-aws-xray/src/AWSXRayPropagator.ts:155: console.log('parsedTraceFlags(extract): ' + parsedTraceFlags);
oh.
Ideally there should be some way for me to say I don't want to see these.
Hi there,
I was trying to follow the instructions listed in this readme and struggled for a while before I figured out that the instructions in the usage section were citing:
const { AWSXRayIdGenerator } = require('@aws/otel-aws-xray-id-generator');
When I did this in my own code, AWSXRayIdGenerator
was actually undefined
(therefore unusable), and I was only able to make it work if I switched the casing:
const { AwsXRayIdGenerator } = require('@aws/otel-aws-xray-id-generator');
Here's a repl.it to demonstrate.
The version of @aws/otel-aws-xray-id-generator
I'm using is 0.13.1
, although I tried switching to a couple older versions of this package with the same result.
Not sure if this is a recent change that needs to be updated in the documentation?
Thanks.
After the Soak Tests completed, a performance degradation was revealed for commit c72eb9f of the refs/heads/main
branch for the (http, manual) Sample App. Check out the Action Logs from the Soak Testing
workflow run on GitHub to view the threshold violation.
Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:
The threshold violation should also be noticeable on our graph of Soak Test average results per commit.
After the Soak Tests completed, a performance degradation was revealed for commit c72eb9f of the refs/heads/main
branch for the (http, manual) Sample App. Check out the Action Logs from the Soak Testing
workflow run on GitHub to view the threshold violation.
Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:
The threshold violation should also be noticeable on our graph of Soak Test average results per commit.
Just a reminder here, we need to test metric path for jssdk as well, it will be great we can add metric code in the sample app
After the Soak Tests completed, a performance degradation was revealed for commit 155c21f of the refs/heads/main
branch for the (http, manual) Sample App. Check out the Action Logs from the Soak Testing
workflow run on GitHub to view the threshold violation.
Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:
The threshold violation should also be noticeable on our graph of Soak Test average results per commit.
Currently, customer have to manually create a wire up an X-Ray tracer for use with OpenTelemetry.
https://github.com/open-telemetry/opentelemetry-js/blob/master/getting-started/README.md#initialize-a-global-tracer
Unfortunately, with X-Ray's config options, this turns out to be much more in depth (yes this whole thing):
https://github.com/open-o11y/aws-opentelemetry-js/blob/master/examples/basic-xray/tracer.js
We should consider vending a ready-to-go tracer.js file for customers to use.
Pretty self explanatory, is support for X-Ray remote sampling on the roadmap for anytime soon?
My package.json is the following:
{
"name": "aws-serverless-nodejs-opentelemetry",
"version": "1.0.0",
"description": "Reviewing and using open telemetry by using the AWS distro for [OpenTelemetry JS](https://github.com/aws-observability/aws-otel-js)",
"main": "handler.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"repository": {
"type": "git",
"url": "git+https://github.com/mfolivas/aws-serverless-nodejs-opentelemetry.git"
},
"author": "",
"license": "ISC",
"bugs": {
"url": "https://github.com/mfolivas/aws-serverless-nodejs-opentelemetry/issues"
},
"homepage": "https://github.com/mfolivas/aws-serverless-nodejs-opentelemetry#readme",
"dependencies": {
"@aws/otel-aws-xray-id-generator": "^0.13.1",
"@aws/otel-aws-xray-propagator": "^0.13.0",
"@opentelemetry/core": "^0.13.0",
"@opentelemetry/exporter-collector-grpc": "^0.13.0",
"@opentelemetry/metrics": "^0.13.0",
"@opentelemetry/node": "^0.13.0",
"opentelemetry-plugin-aws-sdk": "^0.2.1"
}
}
My serverless.yml is the following:
service: aws-serverless-nodejs-opentelemetry
frameworkVersion: '2'
provider:
name: aws
runtime: nodejs12.x
lambdaHashingVersion: 20201221
functions:
hello:
handler: handler.hello
Handler is trying to use the similar sample:
'use strict';
const tracer = require('./tracer')('aws-otel-integ-test');
const AWS = require('aws-sdk');
const meter = require('./metric-emitter');
module.exports.hello = async (event) => {
console.log('First request', handleRequest('/'))
console.log('Second request', handleRequest('/aws-sdk-call'))
console.log('Third request', handleRequest('/outgoing-http-call'))
return {
statusCode: 200,
body: JSON.stringify(
{
message: 'Go Serverless v1.0! Your function executed successfully!',
input: event,
},
null,
2
),
};
// Use this code if you don't use the http event with the LAMBDA-PROXY integration
// return { message: 'Go Serverless v1.0! Your function executed successfully!', event };
};
/** A function which handles requests and send response. */
function handleRequest(url) {
const requestStartTime = new Date().getMilliseconds();
// start recording a time for request
try {
if (url === '/') {
return 'healthcheck';
}
if (url === '/aws-sdk-call') {
const s3 = new AWS.S3();
s3.listBuckets(() => { });
const traceID = returnTraceIdJson();
return traceID;
}
if (url === '/outgoing-http-call') {
const traceID = returnTraceIdJson();
meter.emitsPayloadMetric(res._contentLength + mimicPayLoadSize(), '/outgoing-http-call', res.statusCode);
meter.emitReturnTimeMetric(new Date().getMilliseconds() - requestStartTime, '/outgoing-http-call', res.statusCode);
}
} catch (err) {
console.error(err);
}
}
//returns a traceId in X-Ray JSON format
function returnTraceIdJson() {
const traceId = tracer.getCurrentSpan().context().traceId;
const xrayTraceId = "1-" + traceId.substring(0, 8) + "-" + traceId.substring(8);
const traceIdJson = JSON.stringify({ "traceId": xrayTraceId });
return traceIdJson;
}
//returns random payload size
function mimicPayLoadSize() {
return Math.random() * 1000;
}
However, I am getting the following error:
ERROR Uncaught Exception {"errorType":"Error","errorMessage":"Failed to load gRPC binary module because it was not installed for the current system\nExpected directory: node-v64-linux-x64-glibc\nFound: [node-v79-darwin-x64-unknown]\nThis problem can often be fixed by running \"npm rebuild\" on the current system\nOriginal error: Cannot find module '/var/task/node_modules/grpc/src/node/extension_binary/node-v64-linux-x64-glibc/grpc_node.node'","code":"MODULE_NOT_FOUND","stack":["Error: Failed to load gRPC binary module because it was not installed for the current system","Expected directory: node-v64-linux-x64-glibc","Found: [node-v79-darwin-x64-unknown]","This problem can often be fixed by running \"npm rebuild\" on the current system","Original error: Cannot find module '/var/task/node_modules/grpc/src/node/extension_binary/node-v64-linux-x64-glibc/grpc_node.node'"," at Object.<anonymous> (/var/task/node_modules/grpc/src/grpc_extension.js:53:17)"," at Module._compile (internal/modules/cjs/loader.js:778:30)"," at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)"," at Module.load (internal/modules/cjs/loader.js:653:32)"," at tryModuleLoad (internal/modules/cjs/loader.js:593:12)"," at Function.Module._load (internal/modules/cjs/loader.js:585:3)"," at Module.require (internal/modules/cjs/loader.js:692:17)"," at Module.Hook._require.Module.require (/var/task/node_modules/require-in-the-middle/index.js:80:39)"," at require (internal/modules/cjs/helpers.js:25:18)"," at Object.<anonymous> (/var/task/node_modules/grpc/src/client_interceptors.js:144:12)"]}
The tracer is:
/*
* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS'" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*
*/
'use strict'
const { LogLevel } = require("@opentelemetry/core");
const { SimpleSpanProcessor, ConsoleSpanExporter } = require("@opentelemetry/tracing");
const { NodeTracerProvider } = require('@opentelemetry/node');
const { CollectorTraceExporter } = require('@opentelemetry/exporter-collector-grpc');
const { AWSXRayPropagator } = require('@aws/otel-aws-xray-propagator');
const { AwsXRayIdGenerator } = require('@aws/otel-aws-xray-id-generator');
const { propagation, trace } = require("@opentelemetry/api");
module.exports = (serviceName) => {
// set global propagator
propagation.setGlobalPropagator(new AWSXRayPropagator());
// create a provider for activating and tracking with AWS IdGenerator
const tracerConfig = {
logLevel: LogLevel.ERROR,
idGenerator: new AwsXRayIdGenerator(),
plugins: {
https: {
enabled: true,
// You may use a package name or absolute path to the file.
path: '@opentelemetry/plugin-https',
// https plugin options
},
"aws-sdk": {
enabled: true,
// You may use a package name or absolute path to the file.
path: "opentelemetry-plugin-aws-sdk",
},
},
};
const tracerProvider = new NodeTracerProvider(tracerConfig);
// add OTLP exporter
const otlpExporter = new CollectorTraceExporter({
serviceName: serviceName,
url: (process.env.OTEL_EXPORTER_OTLP_ENDPOINT) ? process.env.OTEL_EXPORTER_OTLP_ENDPOINT : "localhost:55680"
});
tracerProvider.addSpanProcessor(new SimpleSpanProcessor(otlpExporter));
tracerProvider.addSpanProcessor(new SimpleSpanProcessor(new ConsoleSpanExporter()));
// Register the tracer
tracerProvider.register();
// Return an tracer instance
return trace.getTracer("awsxray-tests");
}
Hello,
I have the following example that makes use of the aws-collector
and the open-telemetry instrumentation.
This is the repo and the exact commit: https://github.com/noliva/aws-observability/tree/f9e5abba1c1e240f90e56d1fc0f390c6f453874e
Basically, the problem is that when I do any requests to that service I don't see the DynamoDB instrumentation or the response code in the X-Ray console.
This is what a request looks like in the console:
This is what I would expect:
Not sure if I've done something wrong in the code or there is an issue with the SDK.
Thank you!
Hi,
Right now, the Xray Propagator (@aws/otel-aws-xray-propagator
) does not work with @opentelemetry/api
(API) other than 0.13.0. The reason for that is, it installs its own version of the API and therefore ignores any configuration set by the developer.
If you look at official Otel packages, those treat the API as a peerDependency, e.g.:
https://github.com/open-telemetry/opentelemetry-js/blob/b9c842613c4439435f1644c8c8e288d1fd0e384c/packages/opentelemetry-propagator-jaeger/package.json#L75
I have a node.js api (express) running on ecs fargate. I am running the aws otel collector (v.0.7.0) as sidecar container and have setup the javascript tracing as explained in the documentation. Everything works fine, but the http status code always seems to be 0 in x-ray, although it is included in the otel instrumentation. When I use the console exporter you can see that the http status code is there, so it gets lost somewhere when sending it to x-ray, or x-ray doesn't process it correctly, not sure where to look.
example of console export:
{ traceId: '6675317b0ad8205b6851b809395be4b7', parentId: undefined, name: 'HTTP GET', id: '9aed987b4122245c', kind: 1, timestamp: 1614809120176327, duration: 6635, attributes: { 'http.url': 'http://localhost:8080/woops', 'http.host': 'localhost:8080', 'net.host.name': 'localhost', 'http.method': 'GET', 'http.route': '', 'http.target': '/woops', 'http.user_agent': 'curl/7.64.1', 'http.flavor': '1.1', 'net.transport': 'IP.TCP', 'net.host.ip': '::ffff:127.0.0.1', 'net.host.port': 8080, 'net.peer.ip': '::ffff:127.0.0.1', 'net.peer.port': 56774, 'http.status_code': 401, 'http.status_text': 'UNAUTHORIZED' }, status: { code: 2 }, events: [] }
I am using the http, https, express auto instrumentation plugins.
There is a printscreen of a trace in the documentation which shows the same problem. The response code is also 0. https://aws-otel.github.io/docs/getting-started/js-sdk/trace-manual-instr#start-tracing
Excerpt of raw data of x-ray trace:
"http": { "request": { "url": "http://localhost:8080/woops", "method": "GET", "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36", "client_ip": "127.0.0.1", "x_forwarded_for": true }, "response": { "status": 0, "content_length": 0 } }, "aws": { "xray": { "auto_instrumentation": false, "sdk_version": "0.16.0", "sdk": "opentelemetry for nodejs" } }, "metadata": { "default": { "otel.resource.telemetry.sdk.name": "opentelemetry", "net.transport": "IP.TCP", "http.flavor": "1.1", "http.route": "/woops", "net.host.port": "", "http.status_text": "NOT MODIFIED", "otel.resource.service.name": "test", "otel.resource.telemetry.sdk.language": "nodejs", "net.host.ip": "::ffff:10.2.44.153", "otel.resource.telemetry.sdk.version": "0.16.0" } },
status (and also content) in the http response block are always 0. The metadata block contains http.status_text which is actually shown in xray, but http.status_code is not in there.
I am a newcomer to AWS. For the example of send-metric-app.js, I can't find the corresponding aws-otel-js-sample namespace on Cloudwatch Metrics, so I don't know where the data is uploaded
I have setted ~/.aws(config&credentials) in my locale and run it with node send-metric-app.js
Can someone tell me what I should do to report metric and Log data to CloudWatch,
code in below
'use strict';
// OTel JS - API
const { CollectorMetricExporter } = require('@opentelemetry/exporter-collector-grpc');
const { MeterProvider } = require('@opentelemetry/sdk-metrics-base');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions')
/** The OTLP Metrics Provider with OTLP gRPC Metric Exporter and Metrics collection Interval */
const meter = new MeterProvider({
resource: Resource.default().merge(new Resource({
[SemanticResourceAttributes.SERVICE_NAME]: "aws-otel-js-sample",
[SemanticResourceAttributes.CLOUD_REGION]: "ap-southeast-1",
[SemanticResourceAttributes.AWS_LOG_GROUP_NAMES]: "/aws/metric/test",
[SemanticResourceAttributes.AWS_LOG_STREAM_NAMES]: "open-telemetry"
})),
// Expects Collector at env variable `OTEL_EXPORTER_OTLP_ENDPOINT`, otherwise, http://localhost:4317
exporter: new CollectorMetricExporter(),
interval: 1000,
}).getMeter('aws-otel-js');
/** Counter Metrics */
const payloadMetric = meter.createCounter('payload', {
description: 'Metric for counting request payload size',
});
/** Up and Down Counter Metrics */
const activeReqMetric = meter.createUpDownCounter('activeRequest', {
description: 'Metric for record active requests',
});
/** Value Recorder Metrics with Histogram */
const requestLatency = meter.createValueRecorder('latency', {
description: 'Metric for record request latency',
});
/** Define Metrics Dimensions */
const labels = { pid: process.pid, env: 'beta' };
/** Send the defined metrics every seconds */
setInterval(() => {
payloadMetric.bind(labels).add(1);
activeReqMetric.bind(labels).add(Math.random() > 0.5 ? 1 : -1);
requestLatency.bind(labels).record(Math.random() * 1000)
}, 1000);
in origin code new Resource({[ResourceAttributes.SERVICE_NAME]: "aws-otel-js-sample"}), but ResourceAttributes is not enum, i don‘t know what's the name when i should set the namespace or log group, so i change it to SemanticResourceAttributes
After the Soak Tests completed, a performance degradation was revealed for commit c72eb9f of the refs/heads/main
branch for the (http, manual) Sample App. Check out the Action Logs from the Soak Testing
workflow run on GitHub to view the threshold violation.
Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:
The threshold violation should also be noticeable on our graph of Soak Test average results per commit.
As soon as I run
npm i @aws/otel-aws-xray-id-generator
I start seeing the following errors in my build
node_modules/@types/mocha/index.d.ts:2734:13 - error TS2403: Subsequent variable declarations must have the same type. Variable 'beforeEach' must be of type 'Lifecycle', but here has type 'HookFunction'.
2734 declare var beforeEach: Mocha.HookFunction;
~~~~~~~~~~
node_modules/@types/jest/index.d.ts:35:13
35 declare var beforeEach: jest.Lifecycle;
~~~~~~~~~~
'beforeEach' was also declared here.
and several other similar errors.
My project does have jest
& @types/jest
. It does not have mocha. Looking at the package file, I see
"dependencies": {
....
"@types/mocha": "7.0.2",
"@types/node": "12.12.47",
"@types/sinon": "9.0.4",
"@types/webpack-env": "1.15.2",
....
Were these supposed to be devDependencies
?
Hi,
I followed the sample app instructions however I don't see any metrics showing up in X-Ray. What is interesting is that the sample app in the Go repo works fine and I can see data showing up in X-ray.
Node app:
https://github.com/aws-observability/aws-otel-js/tree/main/sample-apps
Go app:
while checking the e2e test failure, i found the log in otcollector as below:
aws-ot-collector_1 | Span #0
aws-ot-collector_1 | Trace ID : e5ff1f6fc69c69a7dff5df74ebdf79dfaf3b7b87b8edee1f
aws-ot-collector_1 | Parent ID :
aws-ot-collector_1 | ID : 7b6efc69e75ff34f387dcd7b
aws-ot-collector_1 | Name : GET /
aws-ot-collector_1 | Kind : SPAN_KIND_SERVER
aws-ot-collector_1 | Start time : 2020-10-21 04:27:24.78015232 +0000 UTC
aws-ot-collector_1 | End time : 2020-10-21 04:27:26.786455808 +0000 UTC
aws-ot-collector_1 | Status code : STATUS_CODE_OK
aws-ot-collector_1 | Status message :
the trace id is not a valid xray trace id.
However, the sample app returned trace id as
➜ ~ curl http://127.0.0.1:4567/
{"traceId":"1-5f8fb8ac-aaff9d9069953687e4e47e4f"}%
I'm not sure which part is having issue at this moment, but looks like jssdk sends a wrong trace id?
I'm using the X-Ray Id generator in my tracing config, but the generated id are still not compatible with the X-Ray trace id spec, but instead they're just random strings as generated by open telemetry
This is my tracing.js file
const tracerConfig = {
idGenerator: new AWSXRayIdGenerator(),
instrumentations: [getNodeAutoInstrumentations()],
resources: Resource.default().merge(
new Resource({
[SemanticResourceAttributes.SERVICE_NAME]: 'backend',
}),
),
};
const provider = new NodeTracerProvider({ tracerConfig });
const exporter = new OTLPTraceExporter({
url: `http://localhost:4318/v1/traces`,
});
provider.addSpanProcessor(new SimpleSpanProcessor(exporter));
provider.register({
propagator: new AWSXRayPropagator(),
});
and this is the otel-config file for aws otel collector
extensions:
health_check:
pprof:
endpoint: 0.0.0.0:1777
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
exporters:
logging:
loglevel: debug
awsxray:
region: "us-east-1"
awsemf:
region: "us-east-1"
service:
pipelines:
traces:
receivers: [otlp]
exporters: [awsxray, logging]
metrics:
receivers: [otlp]
exporters: [awsemf]
extensions: [pprof]
telemetry:
logs:
level: debug
When i run the application, i get the following logs where trace id is just random string
{"level":"INFO","logger":"root","message":"my log message","service.name":"backend","span_id":"3d5882bfbb0b3c80","timestamp":"2022-06-17T05:45:11.638Z","trace_flags":"01","trace_id":"26bbc3411267031fb55ebb03fcbcc4b7"}
And the following error message form the otel collector
2022-06-17T05:48:53.201Z debug [email protected]/awsxray.go:99 Error translating span. {"kind": "exporter", "name": "awsxray", "error": "invalid xray traceid: 26bbc3411267031fb55ebb03fcbcc4b7"}
This one had me scratching my head for a while 😅
When using @aws/[email protected]
with @opentelemetry/[email protected]
and @opentelemetry/[email protected]
, doing this:
const headers = {
'X-Amzn-Trace-Id': 'Root=1-5759e988-bd862e3fe1be46a994272793;Parent=53995c3f42cd8ad8;Sampled=1'
};
const tracer = api.trace.getTracer(name);
const context = api.propagation.extract(headers);
const span = tracer.startSpan(spanName, {}, context);
Results in a span like this:
With a different traceId
and no parentSpanId
.
This despite the context correctly being extracted by @aws/otel-aws-xray-propagator
:
Note the EXTRACTED_SPAN_CONTEXT
Symbol. I couldn't find this anywhere in the opentelemetry-js source code, but it turns out this was removed in open-telemetry/opentelemetry-js#1589. Those changes were released as part of 0.12.0
.
It turns out this is happening because when @aws/otel-aws-xray-propagator
calls setExtractedSpanContext
, it's calling @opentelemetry/[email protected]
's version of that function, which still uses the EXTRACTED_SPAN_CONTEXT
Symbol
When tracer.startSpan
then tries to read the context, it's using our app's version of the core and api (0.12.0
) and goes through this flow:
getActiveSpan
tries to read the value using the ACTIVE_SPAN_KEY
key, but there's nothing there so it returns undefined
. This then bubbles back up the stack to become the return value of getParent
in startSpan
. startSpan
then creates the span as a new root span with a new trace ID and no parent.
@aws/otel-aws-xray-propagator
depends on "@opentelemetry/core": "^0.10.2"
. Because the version is specified using a caret (^
), it accepts newer versions that do not modify the left-most non-zero digit. Since opentelemetry-js is still in the 0.x.x
version range, this means that e.g. 0.10.3
would be accepted but 0.11.0
would not. In my case it results in @aws/otel-aws-xray-propagator
having its own version of @opentelemetry/[email protected]
in its node_modules
folder and it using that over the version we use in our app. Ditto for @opentelemetry/api
.
Fixing the issue is probably a matter of upgrading @aws/otel-aws-xray-propagator
's dependencies to 0.12.0
and adapting to any potential breaking changes in the propagator API.
When JavaScript SDK goes GA for metrics make sure that:
Note: this SDK is a dependency of our ADOT Lambda offering.
Demo should not be included.
Top level README should have clear directions the components included and their states.
Looking at this, the copyright header may be adding with a wrong indentation.
/cc @KKelvinLo
Hi,
I followed the docs for the sample app and I'm getting the following error whenever I curl or browse to any of the endpoints. Can you please help?
Using Node v14.17.3.
Node HTTP listening on 0.0.0.0:8080
/Users/xxxxx/code/aws-otel-js/sample-apps/node_modules/@opentelemetry/instrumentation-http/build/src/http.js:368
const currentSpan = api_1.getSpan(ctx);
^
TypeError: api_1.getSpan is not a function
at HttpInstrumentation._startHttpSpan (/Users/xxxxx/code/aws-otel-js/sample-apps/node_modules/@opentelemetry/instrumentation-http/build/src/http.js:368:35)
at Server.incomingRequest (/Users/xxxxx/code/aws-otel-js/sample-apps/node_modules/@opentelemetry/instrumentation-http/build/src/http.js:259:42)
at parserOnIncoming (_http_server.js:897:12)
at HTTPParser.parserOnHeadersComplete (_http_common.js:126:17)
Hi,
I found an issue where the parentId of a distributed span is not set when using the AWSXRayPropagator propagator. When a span is initialised via the AWS API Gateway, the header is properly passed to my Node.JS application however when I look in X-Ray the spans are not 'linked'. I can see both the API Gateway span as well as the Node.JS span when I search for the two trace id's but they're not linked.
I've enabled ConsoleSpanExporter() to see what's going on. When I use the 'out-of-the-box' propagator I can see this for example:
traceId: 'e5625bcfc8a54afc445a5783c9ce827a',
parentId: '420e23fec3fdd1a3',
name: 'HTTP POST',
id: '75f4d6fae0e11cc2',
kind: 1,
timestamp: 1614690886046893,
duration: 39607,
attributes: {
'http.url': 'http://localhost:3000/check',
'http.host': 'localhost:3000',
'net.host.name': 'localhost',
'http.method': 'POST',
'http.route': '/check',
'http.target': '/check',
'http.user_agent': 'PostmanRuntime/7.6.0',
'http.request_content_length_uncompressed': 94,
'http.flavor': '1.1',
'net.transport': 'IP.TCP',
'net.host.ip': '127.0.0.1',
'net.host.port': 3000,
'net.peer.ip': '127.0.0.1',
'net.peer.port': 55476,
'http.status_code': 200,
'http.status_text': 'OK'
},
status: { code: 1 },
events: []
}
However when I use AWSXRayPropagator, I get:
traceId: '603e3ac6cb28214c0ff4ce9a9b7c6ebf',
parentId: undefined,
name: 'HTTP POST',
id: 'e4c30d5d89963972',
kind: 1,
timestamp: 1614691014217177,
duration: 39954,
attributes: {
'http.url': 'http://localhost:3000/check',
'http.host': 'localhost:3000',
'net.host.name': 'localhost',
'http.method': 'POST',
'http.route': '/check',
'http.target': '/check',
'http.user_agent': 'PostmanRuntime/7.6.0',
'http.request_content_length_uncompressed': 94,
'http.flavor': '1.1',
'net.transport': 'IP.TCP',
'net.host.ip': '127.0.0.1',
'net.host.port': 3000,
'net.peer.ip': '127.0.0.1',
'net.peer.port': 55602,
'http.status_code': 200,
'http.status_text': 'OK'
},
status: { code: 1 },
events: []
}
After tracing the code I realised the following, the AWSXRayPropagator does not set the traceState in the context.
Comparing the following call in instrumentation-http: api_1.propagation.extract(api_1.ROOT_CONTEXT, headers)
I get this object with the out-of-the-box propagator:
Symbol(OpenTelemetry Context Key SPAN)
{
"traceId": "e5625bcfc8a54afc445a5783c9ce827a",
"spanId": "420e23fec3fdd1a3",
"traceFlags": 1,
"isRemote": "true",
"traceState": "TraceState"
}
where as with AWSXRayPropagator call I get:
Symbol(OpenTelemetry Context Key ACTIVE_SPAN)
{
"traceId": "60389b7527eb080c7eb3ca5c3e69ce27",
"spanId": "06a96d45a27c25e7",
"traceFlags": 1,
"isRemote": "true"
}
Now I'm not sure if it's the root cause of my problem but it's definitely causing the console span exporter to report no parent and I suspect this is also why X-Ray is not stitching them back together again. I can see all my spans going to my aws-otel-collector.
Any thoughts? Happy to provide more traces if required?
During Soak Tests execution, a performance degradation was revealed for commit 155c21f of the refs/heads/main
branch for the (http, manual) Sample App. Check out the Action Logs from the Soak Testing
workflow run on GitHub to view the threshold violation.
Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:
Problem:
Npm install can be used directly on github branches, forks and commits, no need to publish to NPM central.
Reproduction (on fresh machine):
npm install open-o11y/aws-opentelemetry-js
Command line output:
~/aws-xray-sdk-node-sample [ npm install open-o11y/aws-opentelemetry-js
> [email protected] postinstall /Users/mcmuls/aws-xray-sdk-node-sample/node_modules/aws-opentelemetry-js
> npm run bootstrap
> [email protected] bootstrap /Users/mcmuls/aws-xray-sdk-node-sample/node_modules/aws-opentelemetry-js
> lerna bootstrap
sh: lerna: command not found
npm ERR! file sh
npm ERR! code ELIFECYCLE
npm ERR! errno ENOENT
npm ERR! syscall spawn
npm ERR! [email protected] bootstrap: `lerna bootstrap`
npm ERR! spawn ENOENT
npm ERR!
npm ERR! Failed at the [email protected] bootstrap script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm WARN Local package.json exists, but node_modules missing, did you mean to install?
npm ERR! A complete log of this run can be found in:
npm ERR! /Users/mcmuls/.npm/_logs/2020-09-29T20_56_26_255Z-debug.log
npm WARN [email protected] No repository field.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] postinstall: `npm run bootstrap`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] postinstall script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! /Users/mcmuls/.npm/_logs/2020-09-29T20_56_26_324Z-debug.log
~/aws-xray-sdk-node-sample [ nano /Users/mcmuls/.npm/_logs/2020-09-29T20_56_26_324Z-debug.log
NPM logs
88 verbose stack Error: [email protected] postinstall: `npm run bootstrap`
88 verbose stack Exit status 1
88 verbose stack at EventEmitter.<anonymous> (/Users/mcmuls/.nvm/versions/node/v11.15.0/lib/node_modules/npm/node_modules/npm-life$
88 verbose stack at EventEmitter.emit (events.js:193:13)
88 verbose stack at ChildProcess.<anonymous> (/Users/mcmuls/.nvm/versions/node/v11.15.0/lib/node_modules/npm/node_modules/npm-life$
88 verbose stack at ChildProcess.emit (events.js:193:13)
88 verbose stack at maybeClose (internal/child_process.js:999:16)
88 verbose stack at Process.ChildProcess._handle.onexit (internal/child_process.js:266:5)
89 verbose pkgid [email protected]
90 verbose cwd /Users/mcmuls/aws-xray-sdk-node-sample
91 verbose Darwin 19.6.0
92 verbose argv "/Users/mcmuls/.nvm/versions/node/v11.15.0/bin/node" "/Users/mcmuls/.nvm/versions/node/v11.15.0/bin/npm" "install" "op$
93 verbose node v11.15.0
94 verbose npm v6.7.0
95 error code ELIFECYCLE
96 error errno 1
97 error [email protected] postinstall: `npm run bootstrap`
97 error Exit status 1
98 error Failed at the [email protected] postinstall script.
98 error This is probably not a problem with npm. There is likely additional logging output above.
99 verbose exit [ 1, true ]
We should modify sample app's /outgoing-http-call
to use https
scheme instead of http
. Currently, the sample app response throws status code 301
for request like http://aws.amazon.com
. We should change this request to something like this (https://aws.amazon.com
) which should return 200
response status code. However, I have tested the sample app code locally using https.get('https://aws.amazon.com')
but sample app does not instrument https://aws.amazon.com
outgoing call. However JS HTTP instrumentation package supports tracing http
and https
call both. We might need to spend some time to investigate on how can we turn instrumentation for both the scheme and if it's enabled by default for both the schemes then why sample app is not able to instrument https
scheme calls. Also, validate that if we have set up server which expects incoming http
requests then can we send out going https
requests while handling incoming requests?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.