datadog / dd-trace-js Goto Github PK

I have two services I am tracing through, a Node.js server on top of a Java backend. The IDs being sent to the Java backend by the Node client are periodically too large for dd-trace-java opentracing to parse into a Long.

Exception with no mapped status code
java.lang.NumberFormatException: For input string: "9772945757313177262"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[na:1.8.0_162]
	at java.lang.Long.parseLong(Long.java:592) ~[na:1.8.0_162]
	at java.lang.Long.parseLong(Long.java:631) ~[na:1.8.0_162]
	at datadog.opentracing.propagation.HTTPCodec.extract(HTTPCodec.java:60) ~[com.datadoghq.dd-trace-ot-0.8.0.jar:na]
	at datadog.opentracing.propagation.HTTPCodec.extract(HTTPCodec.java:15) ~[com.datadoghq.dd-trace-ot-0.8.0.jar:na]
	at datadog.opentracing.DDTracer.extract(DDTracer.java:249) ~[com.datadoghq.dd-trace-ot-0.8.0.jar:na]
	at io.opentracing.util.GlobalTracer.extract(GlobalTracer.java:143) ~[io.opentracing.opentracing-util-0.31.0.jar:na]

And here's the stack trace from my Node.js server, along with the ID's as they are in the DatadogSpan object.

TraceId: 1455231796184695300, SpanId: 9772945757313178000, ParentId: 15993920359388260000
{ error: 'java.lang.NumberFormatException',
  message: 'For input string: "9772945757313177262"' }
Error: {"error":"java.lang.NumberFormatException","message":"For input string: \"9772945757313177262\""}
    at /opt/app/lib/requestutils.js:85:13
    at propagateAslWrapper (/tmp/node_modules/async-listener/index.js:502:23)
    at /tmp/node_modules/async-listener/glue.js:188:31
    at proxyWrapper (/tmp/node_modules/async-listener/index.js:511:29)
    at /tmp/node_modules/async-listener/index.js:539:70
    at /tmp/node_modules/async-listener/glue.js:188:31
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)

Currently DatadogSpan doesn't implement the _log method and I don't find any way to pass in a custom _log method.

1. May I know what's the current recommended practice if we wants to be able to use span.log?
2. Are there any future plans to implement the _log method?

Thanks you :)

Hi,

I noticed that dd-trace v0.2.1's Opentracing API is incompatible with DataDog agent v5.12.3

When the following code is executed:

const tracer = require('dd-trace').init({ service: 'example' });

const span = tracer.startSpan('span_name');
span.setTag('resource.name', '/hello/:name');
span.setTag('span.type', 'web');

span.finish();

the following error is logged in trace-agent.log:

cannot decode v0.3 traces payload: msgp: attempted to decode type "int" with method for "uint"

The current workaround is to use dd-trace v0.1.7

Thank you in advance 🙏

👋,

I think it's because of a typo here "mutations" instead of "mutation":
https://github.com/DataDog/dd-trace-js/blob/master/src/plugins/graphql.js#L257

While debugging this I also discovered that this does not work as intended:
https://github.com/DataDog/dd-trace-js/blob/master/src/plugins/graphql.js#L15-L17

This might use the patched schema, because the object is never deeply copied.
I could reproduce this like so:

1. start graphql server
2. run mutation, works fine (does not wrap anything since there's a typo)
3. run query, schema is mutated
4. run mutation, throw errors: contextValue in undefined in #resolveField

The express and http plugins (possible more) do not set the service name tag in the same way, see:

This made my service show up as both *service-name* + http-client.

I set service name through:

require('dd-trace').init({
  service: process.env.SERVICE_NAME,
});

There is currently no option to pass a custom sampleRate to the Tracer. See mcuelenaere@b4b1b63 for a potential fix.

HI,

We have an koa app that uses firebase realtime database, but when we initialise the firebase application in the project, then we get the following trace in our logs;

MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added. Use emitter.setMaxListeners() to increase limit
    at _addListener (events.js:280:19)
    at TLSSocket.addListener (events.js:297:10)
    at TLSSocket.Readable.on (_stream_readable.js:772:35)
    at ClientRequest.req.on.socket (/pipeline/source/node_modules/dd-trace/src/plugins/http.js:50:16)
    at emitOne (events.js:121:20)
    at ClientRequest.emit (events.js:211:7)
    at tickOnSocket (_http_client.js:652:7)
    at onSocketNT (_http_client.js:668:5)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)

Please can you consider adding a Mongoose plugin.

It is built on top of MongoDB-core, but it doesn't work, it is probably an easy tweak.

I am currently wrapping the trace API in a promise to make it easier to work with:

type Trace = <U>({|
  childOf?: string,
  name: string,
  resolver: (span: any) => Promise<U>,
  resource: string,
  service?: string,
  tags?: { [string]: string },
  tracer: T,
  type: string
|}) => $await<U>;

export const trace: Trace = ({
  childOf,
  name,
  resolver,
  resource,
  service,
  tags = {},
  tracer,
  type
}) =>
  new Promise((resolve, reject) => {
    try {
      tracer.trace(
        name,
        { childOf, resource, service, tags, type },
        async span => {
          const result = await resolver(span);
          span.finish();

          resolve(result);
        }
      );
    } catch (error) {
      reject(error);
    }
  });

In the Tracing docs it says that support for koa web framework is "Coming soon". I would really love to bump this up in priority if that's possible :) Happy to test some stuff if needed. Thanks for your work!

Hi,

I noticed that for DataDog tracer to works, a span is required to have resource and type properties which can be set by settings resource.name and span.type respectively. But, this requirement is not documented in the documentations (e.g. this and this). Could this kind of requirements be formally documented?

Thank you in advance 🙏

Hiya,

It wasn't immediately apparent that dd-trace only works properly if you:

1. require() it
2. require() everything else it's supposed to instrument
3. make sure you require('dd-trace').init(); before you require() anything else instrumentable

It doesn't work if using module import statements. This had me pulling my hair out.

Better docs for dummies like me please :)

We installed the latest version (0.5.1) and deployed and all of our instances began throwing out-of-memory errors almost immediately. Rolling back the commit (removing dd-trace) fixed the issue. Other than a change to package.json the following is the entirety of our diff:

+const tracer = require('dd-trace');
+if (process.env.DD_TRACE_ENABLED === 'true') {
+  tracer.init({ plugins: false });
+
+  tracer.use('http');
+  tracer.use('express');
+  tracer.use('graphql');
+}
+

We've upgraded our PG module from 6.3 -> 7.4
We no longer see information for postgres in our services tab for our environment

{ Error: Cannot find module './https'
    at Function.Module._resolveFilename (module.js:536:15)
    at Function.Module._load (module.js:466:25)
    at Module.require (module.js:579:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/Users/lewis/dev/cirrus/node_modules/dd-trace/src/plugins/index.js:9:12)
    at Module._compile (module.js:635:30)
    at Object.Module._extensions..js (module.js:646:10)
    at Module.load (module.js:554:32)
    at tryModuleLoad (module.js:497:12)
    at Function.Module._load (module.js:489:3) code: 'MODULE_NOT_FOUND' }

It seems that src/plugins/index.js is attempting to require src/plugins/https.js which does not exist.

So I want to be able to write a log message containing a link to the corresponding APM-trace.
To do this I require having access to the span object/id somehow.

Is this possible with the current implementation or is that a wanted feature?

What parts are missing for this package to work in the browser? Is there a plan overall for traces from frontends?

Our datadog agents aren't exposed to the public internet, so we're thinking of having a small ingestion proxy that forwards spans. I think that means we'd want to plug in a custom write method or similar.

👋,

I'm new to DD, but it seems like most other DD libraries have some support for this.
Is adding this to the node library in the roadmap?

Thank you.

The plugin works great with express based nodejs apps

However we would like to trace request latency from client to server and back

When I use the module from a ReactJS app created using create-react-app I get this error:

./node_modules/require-in-the-middle/index.js
Module not found: Can't resolve 'module' in '...\node_modules\require-in-the-middle'

Hi guys,

I used dd-trace on my server for two weeks. I used it to get the stats from a parse server (it's an express app with extra libs) launched in cluster mode with pm2.
During this period, I noticed that my nodejs processes were using much processor's resources than before. It went to the roof, up to 100% per core for some requests.

After I launched every nodejs process with --prof to get the profiling, I get this result:
ticks parent name
113187 7.6% UNKNOWN
13640 12.1% LazyCompile: *write buffer.js:889:40
13639 100.0% LazyCompile: *string /home/ubuntu/parse-server/node_modules/msgpack-lite/lib/write-type.js:110:20
10863 79.6% LazyCompile: *<anonymous> /home/ubuntu/parse-server/node_modules/msgpack-lite/lib/write-type.js:237:26
10863 100.0% Builtin: ArrayForEach
10863 100.0% LazyCompile: *obj_to_map /home/ubuntu/parse-server/node_modules/msgpack-lite/lib/write-type.js:230:22
This is just a sample of the output, I got several line like this one ending up to msgpack-lite.
here is the report

For what I understood from this report it is the processor which is spending too many ticks in the msgpack-lite, I found from the package-lock.json that dd-trace-js was the only one using it. So I removed dd-trace-js from my code and everything went back to normal.

I don't know what happened but it will be great if there is a way for us to fix this issue in order to use datadog APM :)

Cheers,

What's the ETA on a KoaJS integration?
We're trying to manually wire up dd-trace but it's causing memory leaks.

The plateaus in the image is when we reverted the commit, after 4 attempts to figure out what was causing the ENOMEM errors.

this would be great to be able to drill down into problematic queries

How deep down is GraphQL supposed to instrument?

With a query like the following only these fields friends, friends.0.name, friends.0.pets, friends.1.name and friends.1.pets get instrumented. Should all the fields on the types pet and color, get instrumented too?

{
  friends {
    name
    pets {
      name
      colors {
        code
      }
    }
  }
}

Noop.bind should return a function to match how the real thing works.

Right now the instrumenter automatically loads plugins for the built-in Node integrations. This should not be the case if we are to support other platforms. The instrumenter should therefore only load plugins appropriate to the current platform.

Hey Folks - dd-trace isnt working for me.

1. Same dd-agent is sending traces from JVM perfectly fine.

2. connectivity to agent verified
   ➜ dd-trace-js git:(master) ✗ telnet localhost 8126 Trying ::1... Connected to localhost. Escape character is '^]'. ^C{}

3. installed using npm install

4. Executed test code and it never appears on datadog traces and
   ➜ dd-trace-js git:(master) ✗ curl http://localhost:8000/ Hello, World!%

trace-agent.logs

2018-05-07 17:38:30 INFO (service_mapper.go:59) - total number of tracked services: 0 2018-05-07 17:39:30 INFO (receiver.go:324) - no data received 2018-05-07 17:39:30 INFO (service_mapper.go:59) - total number of tracked services: 0 2018-05-07 17:40:30 INFO (receiver.go:324) - no data received 2018-05-07 17:40:30 INFO (service_mapper.go:59) - total number of tracked services: 0 2018-05-07 17:41:30 INFO (service_mapper.go:59) - total number of tracked services: 0 2018-05-07 17:41:40 INFO (receiver.go:324) - no data received 2018-05-07 17:42:30 INFO (service_mapper.go:59) - total number of tracked services: 0 2018-05-07 17:42:50 INFO (receiver.go:324) - no data received

Is this working for you ?

Typescript definitions (or using ts in the project 😄) would be helpful when interacting with this library.

Description

I would like to take advantage of the tags listed in the Integrations section of the documentation.

Issue

How to configure the tags is not explicitly addressed in the documentation, e.g. should a configuration object be passed that specifies both service and tags when initializing the span?

tracer
  .trace('web.request', {
    service: 'my_service',
    tags: { 
       // ...
    }
  })

I noticed in DataDog that none of my traces had database calls, except for the first query, which seemed to have them all belong to it.

After loads of debugging I worked out its down to the current span in the query being the span that creates the connection.

My solution for now was to stop using pool.query and use pool.getConnection, however I had to bind the callback to get the correct span for the query.

pool.getConnection(tracer.bind((err, conneciton) => {
    connection.query(sql, values, cb)
}))

The problem happens in pool.acquireConnection, when it enters onOperationComplete the span is changed to that original one.

Same reproduction steps as listed in Issue #77 and same output; however, the only difference is that I am using the call to .init to provide the configuration values (not ENV values). My modified version of the default, test code provided in the doc.s looks like:

const tracer = require('dd-trace').init({
  service: 'bobs-burges',
  hostname: 'localhost',
  port: 8126
});

Even with the above default values specified, no connectivity is established.

Hey there,

We are using the mysql2 module with require('mysql2/promise') which causes the patch to fail:

/home/max/git/my-api/node_modules/dd-trace/src/plugins/mysql2.js:81
  shimmer.wrap(mysql2.Pool.prototype, 'getConnection', createWrapGetConnection(tracer, config))
                          ^

TypeError: Cannot read property 'prototype' of undefined
    at Object.patch (/home/max/git/my-api/node_modules/dd-trace/src/plugins/mysql2.js:81:27)
    at Array.from.filter.filter.forEach.plugin (/home/max/git/my-api/node_modules/dd-trace/src/instrumenter.js:72:16)
    at Array.forEach (<anonymous>)
    at Instrumenter.hookModule (/home/max/git/my-api/node_modules/dd-trace/src/instrumenter.js:67:8)
    at Function.Module._load (/home/max/git/my-api/node_modules/require-in-the-middle/index.js:69:30)
    at Module.require (module.js:517:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/home/max/git/my-api/node_modules/mysql2/lib/pool.js:1:75)
    at Module._compile (module.js:573:30)
    at Object.Module._extensions..js (module.js:584:10)
    at Module.load (module.js:507:32)
    at tryModuleLoad (module.js:470:12)
    at Function.Module._load (module.js:462:3)
    at Function.Module._load (/home/max/git/my-api/node_modules/require-in-the-middle/index.js:25:24)
    at Module.require (module.js:517:17)
    at require (internal/module.js:11:18)

Node version: 8.4.0
dd-trace-js version: ^2.1.0
mysql2 version: ^1.5.2

I'm sure this has something todo with the Promise stuff from mysql2.

Is there any way I can use this module without rewriting my whole code?
Thanks in advance

I tried to migrate from jaeger to ddtracer

both instances are created from opentracing

 console.log(process.env.TRACER)

 const span = tracer.startSpan(host, { childOf: parentContext })

//this is where I try to debug
 console.log(span.context())

result when using jaeger

debug: jaeger
debug:  
{ _traceId: 
   { '0': 99,
     '1': 109,
     '2': 24,
     '3': 134,
     '4': 131,
     '5': 164,
     '6': 174,
     '7': 156 },
  _spanId: { '$ref': '$["_traceId"]' },
  _parentId: null,
  _traceIdStr: null,
  _spanIdStr: null,
  _parentIdStr: null,
  _flags: 0,
  _baggage: {},
  _debugId: '',
  _samplingFinalized: false }

result when using ddtracer

debug: datadog
debug:  
{ traceId: { buffer: [Object], offset: 0 },
  spanId: { '$ref': '$["traceId"]' },
  parentId: null,
  sampled: true,
  baggageItems: {},
  trace: { started: [Array], finished: [] } }

It seems like the contexts are entirely different. So, what do I probably miss?

Originally requested in #49

Looks like dd-trace will try to instrument local javascript files if they name-match supported modules.

For instance, we had a require('./mongodb'); in one of our projects, as it had unfortunately named its DAL with the same name as the mongodb module. dd-trace tried to instrument it and broke things pretty badly.

Might want to look at being more strict around which require()'d things dd-trace tries to instrument, if possible.

import {
  init
} from 'dd-trace';

init({
  hostname: process.env.DD_AGENT_SERVICE_HOST,
  port: process.env.DD_AGENT_SERVICE_PORT
});

Results in:

/srv/node_modules/dd-trace/src/proxy.js:49
    if (this._tracer === noop) {
             ^

TypeError: Cannot read property '_tracer' of undefined
    at init (/srv/node_modules/dd-trace/src/proxy.js:49:14)
    at Object.<anonymous> (/srv/dist/bin/index.js:12:21)
    at Module._compile (internal/modules/cjs/loader.js:678:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:689:10)
    at Module.load (internal/modules/cjs/loader.js:589:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:528:12)
    at Function.Module._load (internal/modules/cjs/loader.js:520:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:719:10)
    at startup (internal/bootstrap/node.js:228:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:576:3)

Solution:

Bind init method to the dd-trace object prior to exporting it.

Requested by @UnbrandedTech in #197

Hi there,

We'd like to take advantage of APM events and trace searching in Datadog, by adding custom tags to our ExpressJS and/or GraphQL spans, which are auto-instrumented.

Any examples on how to do this? I see there are some for Python/Ruby but none for Javascript.

From standard span tags on opentracing spec, error tag should be respected to "decide if and only if the application considers the operation represented by the Span to have failed" true if and only if the application considers the operation represented by the Span to have failed.

Currently we are relying on those three tags instead: ['error.type', 'error.msg', 'error.stack'].
reference

Hope that I can get some feedback. Will be happy to make a PR for it if necessary, thanks.

Please consider adding a dataloader plugin.

Use a logger to document what the agent is doing, which plugins are being enabled, what modules are being patched.

I humbly recommend https://github.com/gajus/roarr for all the reasons described in https://github.com/gajus/roarr#motivation.

Case in point: I have loaded dd-trace using:

import tracer from 'dd-trace';

// eslint-disable-next-line no-process-env
if (process.env.DD_AGENT_SERVICE_HOST && process.env.DD_AGENT_SERVICE_PORT) {
  tracer.init({
    // eslint-disable-next-line no-process-env
    hostname: process.env.DD_AGENT_SERVICE_HOST,
    // eslint-disable-next-line no-process-env
    port: process.env.DD_AGENT_SERVICE_PORT
  });
}

and I have no clue whats the reason "pg" or "express" data is not being logged; only "http" is being tracked.

Hi! After updating to 0.5.3, I'm now encountering the following error. I'm running on node 8.9.4 (in docker). 0.5.1 (and 0.5.2) are both behaving as expected.

No custom instrumentation has been used, with only the standard setup outlined here: https://docs.datadoghq.com/tracing/setup/nodejs/

TypeError: headers.authorization.startsWith is not a function at hasAmazonSignature (/home/app/ims-ui/server/node_modules/dd-trace/src/plugins/http.js:116:61) at requestTrace (/home/app/ims-ui/server/node_modules/dd-trace/src/plugins/http.js:39:12) at /home/app/ims-ui/server/node_modules/node-fetch/lib/index.js:1375:15 at new Promise (<anonymous>) at fetch (/home/app/ims-ui/server/node_modules/node-fetch/lib/index.js:1367:9) ...

https://github.com/graphql/express-graphql

Do you guys support this lib as well?

I ask because anyone running express under the hood is probably using express-graphql if they're using graphql. And you guys support express. :)

👋,

Unless i'm mistaking, the following should work:

yarn init
yarn add [email protected] [email protected]

// index.js
const tracer = require('dd-trace').init({ debug: true });
const graphql = require('graphql');

// Instrumented module "graphql/language/parser.js" was imported before calling tracer.init(). Please make sure to initialize the tracer before importing any instrumented module.
// Instrumented module "graphql/execution/execute.js" was imported before calling tracer.init(). Please make sure to initialize the tracer before importing any instrumented module.

node 8.9.4

edit: it actually hooked correctly despite the warnings, execution was failing because of #173

We're using redis' multi().<functions>.exec() in quite a few places - and they show up in somewhat weirdly in traces:

redisConnection.multi()
    .getset(sessionKey, newNonceValue)
    .expire(sessionKey, sessionExpiry)
    .exec(callback);

Given that the docs for multi / exec mentions the commands in between as a queue, I would expect it to be represented along these lines:

| multi / exec --------------|
  | getset ---| | expire --|

It looks like the tracer constructed in opentracing/tracer.js is ignoring the configured sample rate and using a hardcoded value of 1

this._sampler = new Sampler(1)

Unless I'm missing something this samples everything, regardless of config. Happy to put together a quick PR to patch -> it could probably use something as simple config.sampleRate || 1

These calls to cls-bluebird crash in its absence: https://github.com/DataDog/dd-trace-js/blob/master/src/platform/node/context/cls.js#L7

From #69 (comment)

On initialise tracer attempts to load a package.json file from the cwd. My application deployments don't include a package.json so the application fails to start.

/xxx/node_modules/dd-trace/src/platform/node/load.js:12
  this._service = pkg.name
                      ^
 TypeError: Cannot read property 'name' of undefined
    at Object.load (/xxx/node_modules/dd-trace/src/platform/node/load.js:12:23)
    at TracerProxy.init (/xxx/node_modules/dd-trace/src/proxy.js:19:16)
    at Object.<anonymous> (/xxx/lib/index.js:3:21)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
    at Function.Module.runMain (module.js:693:10)
    at startup (bootstrap_node.js:188:16)

The issue stems from this.service = coalesce(options.service, platform.env('DD_SERVICE_NAME'), platform.service()). Being a function coalesce is not lazy so all the arguments are evaluated before , the only work around is to add a package.json to make it happy.

Adding a default option or using lazy evaluation (e.g. by using the || operator) would solve the issue.

Please consider adding a memcached module plugin.