grafana / xk6-distributed-tracing

A k6 extension for distributed tracing.

⚠️ Archived ⚠️

This extension has been archived and is no longer maintained by the Grafana team.

At this time, the tracing feature is now available in the as part of the k6/experimental/tracing module in k6 OSS.

Furthermore, Distributed Tracing in Grafana Cloud k6 is now generally available in Grafana Cloud.

USE AT YOUR OWN RISK!

xk6-distributed-tracing

This extension adds distributed tracing support to k6!

That means that if you're testing an instrumented system, you can use this extension to start the traces on k6.

Currently, it supports HTTP requests and the following propagation formats: w3c, b3, and jaeger.

It is implemented using the xk6 extension system.

Build

To build a k6 binary with this extension, first ensure you have the prerequisites:

• Go toolchain
• Git

Then:

1. Download xk6:

$ go install go.k6.io/xk6/cmd/xk6@latest

2. Build the binary:

$ xk6 build --with github.com/grafana/xk6-distributed-tracing@latest

Example

import tracing, { Http } from 'k6/x/tracing';
import { sleep } from 'k6';

export let options = {
  vus: 1,
  iterations: 10,
};

export function setup() {
  console.log(`Running xk6-distributed-tracing v${tracing.version}`, tracing);
}

export default function() {
  const http = new Http({
    propagator: "w3c",
  });
  const r = http.get('https://test-api.k6.io');
  console.log(`trace_id=${r.trace_id}`);
  sleep(1);
}

Result output:

$ ./k6 run script.js

          /\      |‾‾| /‾‾/   /‾‾/
     /\  /  \     |  |/  /   /  /
    /  \/    \    |     (   /   ‾‾\
   /          \   |  |\  \ |  (‾)  |
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: script.js
     output: -

  scenarios: (100.00%) 1 scenario, 1 max VUs, 40s max duration (incl. graceful stop):
           * default: 1 looping VUs for 10s (gracefulStop: 30s)

INFO[0000] Running xk6-distributed-tracing v0.2.0  source=console
INFO[0000] trace_id=743fff0b96778539acb7139e72ea1e33
INFO[0001] trace_id=365f4637a52526db1de2d30a5568ca3a
INFO[0002] trace_id=c49e1df945049c5c3c8b59acc84d7d3b
INFO[0003] trace_id=53e1937d56aa172b46d2310e3380dfe9
INFO[0004] trace_id=d61e8757d35c9ca1780b88977ac56d72
INFO[0005] trace_id=358e794ed636d268a918dcd2f3f9db0a
INFO[0006] trace_id=992a959e09ee84f3905a215bec8b53a0
INFO[0007] trace_id=aee11c64de11744ab5b66d5dd8ed361b
INFO[0008] trace_id=c4dc45d857e99ede2bb902666457239d
INFO[0009] trace_id=7623d10293d9f03c15deb8055935664e

running (10.1s), 0/1 VUs, 10 complete and 0 interrupted iterations
default ✓ [======================================] 1 VUs  10s

     █ setup

     data_received..............: 1.6 kB 156 B/s
     data_sent..................: 1.7 kB 165 B/s
     http_req_blocked...........: avg=223.43µs min=146.53µs med=217.39µs max=314.54µs p(90)=276.68µs p(95)=295.61µs
     http_req_connecting........: avg=137.18µs min=87.22µs  med=130.17µs max=196.38µs p(90)=184.38µs p(95)=190.38µs
     http_req_duration..........: avg=6.58ms   min=5.07ms   med=6.45ms   max=7.91ms   p(90)=7.83ms   p(95)=7.87ms
     http_req_receiving.........: avg=187.27µs min=94.29µs  med=171.7µs  max=295.67µs p(90)=293.28µs p(95)=294.48µs
     http_req_sending...........: avg=128.07µs min=94.64µs  med=121.77µs max=175.65µs p(90)=160.41µs p(95)=168.03µs
     http_req_tls_handshaking...: avg=0s       min=0s       med=0s       max=0s       p(90)=0s       p(95)=0s
     http_req_waiting...........: avg=6.27ms   min=4.83ms   med=6.13ms   max=7.64ms   p(90)=7.56ms   p(95)=7.6ms
     http_reqs..................: 10     0.991797/s
     iteration_duration.........: avg=916.48ms min=65.67µs  med=1s       max=1s       p(90)=1s       p(95)=1s
     iterations.................: 10     0.991797/s
     vus........................: 1      min=1 max=1
     vus_max....................: 1      min=1 max=1

xk6-distributed-tracing's Issues

spanName isn't used

Is this intentional?

xk6-distributed-tracing/client/client.go

Line 83 in 72eb374

func (c *TracingClient) WithTrace(fn HttpFunc, spanName string, url goja.Value, args ...goja.Value) (*HTTPResponse, error) {

I'm having a problem where traces sent by version 72eb374947a2ef5291457d86d00eb689d9a23fa4 latest commit, aren't picked up by our services, instead a gateway service generates own trace_id which is carried over to other backends. I started reviewing code and notice the spanName isn't used, but passed with values like HTTP OPTIONS. Should users be able to configure it in Http constructor?

Gracefully shutdown the instrumentation

When k6 finishes, we don't shutdown the instrumentation! And that means... that sometimes we don't export the last spans.

Adjust xk6-distributed-tracing to use the new Sample.Metadata

http headers are canonicalized before sending

Despite the header being specified in lower case https://github.com/grafana/xk6-distributed-tracing/blob/master/client/traceid.go#L16

when it's put to a map golang changes the key to a canonicalized version which always starts with Uppercase letter this breaks compatibility of the b3 spec. Request aren't picked up by existing b3-compatible backends.

The expected header is b3, but remote backend receives B3, span and parent trace id aren't picked up:

Parent ID does not Satisfy W3C Propagation

The current W3C propagator in the client is using a 8 rune Hex, making it a 6 byte array. The W3C states the parent-id part of the traceparent has to be an 8 byte array, or a 16 rune hex. This causes for the traceparent to be ignored breaking the tracing when it reaches the service
This is found in the GenerateHeaderBasedOnPropagator() (./client/traceid.go#22)
hex8 := RandHexStringRunes(8)
The PR open makes the change to add a hex16 := RandHexStringRunes(16) variable to further use it in the w3c propagator header

[Research] Don't use the Global Provider pattern

It's a good practice to set the Trace Provider as Global!

But on k6, maybe the user wants to have multiple trace providers, with different configs for different types of requests.

This needs some research.

Add support for different exporters

The modules is `github.com/k6io/xk6-distributed-tracing` instead of `github.com/grafana/xk6-distributed-tracing`

Migrate to a k6/js.modules.Module (ModulesV2)

What?

In the k6 we deprecated, the usage of the old modules API and in the middle of the process of the migration to a new ModulesV2 API.

To do that, we need to clean up the usages of the deprecated methods:

grep --exclude-dir=vendor -Ri "common.Bind" .                                                          
./tracing.go:	return common.Bind(rt, tracingClient, ctx)

A good starting point can be our guide Create an extension.

Why?

The traditional approach of initializing JS modules involves calling common.Bind() on any objects that need to be exposed to JS. This method has a few technical issues we want to improve and isn't flexible enough to implement new features like giving extensions access to internal k6 objects.

Change root span name

Right now the name of the root span is "make-request".

The name of this span should be HTTP GET or something in line with that.

Return trace_id on the response object

That way we can use it from JavaScript and log it... or do other things.

Add support for different propagators

Add HTTP tags to the root span

http.status, http.code...

Right now the span doesn't have a lot of information. See this image:

For reference, here's a span created by the requests Python instrumentation:

Add a trace-id tag to the k6 HTTP metric

k6 stores various information as Tags for each metric data point.

For further investigation on the observability provider, it could be useful to add a tag with the trace-id value.

This feature could be included without the need for k6 changes. Mihail commented:

the http api takes Params as it's latest argument and you can add tags to it. But you need to be careful to add the additional tags to the last argument object as described in the js api docs instead of jsut to keep adding arguments aka
http.get(url, {headers: , tags: }) is correct http.get(url, {headers: }, {tags: }) doesn't work 🙂
Looking at the code this is likely a problem as well now as the current code just adds headers and if there is already something else provided as the last argument, the trace headers will just be added as another argument which will unfortunately not work 😉