grafana / xk6-output-kafka

k6 extension to output real-time test metrics to a Kafka topic.

xk6-output-kafka

This extension is extracted from the k6 kafka output so it can be used with xk6.

Build

To build a k6 binary with this extension, first ensure you have the prerequisites:

• Go toolchain
• Git

1. Build with xk6:

xk6 build --with github.com/grafana/xk6-output-kafka

This will result in a k6 binary in the current directory.

2. Run with the just built k6:

./k6 run -o xk6-kafka <script.js>

Instructions

You can configure the broker (or multiple ones), topic, and message format directly from the command line parameter like this:

./k6 run --out xk6-kafka=brokers=broker_host:9092,topic=k6

or if you want multiple brokers:

./k6 --out xk6-kafka=brokers={broker1,broker2},topic=k6,format=json

You can also specify the message format k6 will use. By default, it will be the same as the JSON output, but you can also use the InfluxDB line protocol for direct "consumption" by InfluxDB:

./k6 --out xk6-kafka=brokers=someBroker,topic=someTopic,format=influxdb

You can even modify some of the format settings such as tagsAsFields:

./k6 --out xk6-kafka=brokers=someBroker,topic=someTopic,format=influxdb,influxdb.tagsAsFields={url,myCustomTag}

Testing Locally

This repo includes a docker-compose.yml file that starts local Kafka environment with several dependencies and utilities baked-in. See lensesio/fast-data-dev for more information.

⚠️ Be sure that you've already compiled your custom k6 binary as described in the Build section!

We'll use this environment to run some examples.

1. Start the docker compose environment.

   docker compose up -d

   Once you see the following, you should be ready.

    [+] Running 2/2
    ⠿ Network xk6-output-kafka_default       Created
    ⠿ Container xk6-output-kafka-lensesio-1  Started

   Next, we'll use the k6 binary we compiled in the Build section above.

2. Open http://localhost:3030/ to display the landing page for your local Kafka Development environment provided by the docker image. From there, you can select the Kafka Topics UI.

3. Using our custom k6 binary, we can execute our example script outputting test metrics to kafka.

   ./k6 run --out xk6-kafka=brokers={localhost:9092},topic=k6-metrics examples/simple.js

   Refreshing your browser should now list the k6-metrics topic. Clicking into the topic will now show messages for real-time metrics published during the k6 test execution.

See also

• k6 Kafka Documentation
• Integrating k6 with Apache Kafka

xk6-output-kafka's Issues

k6 0.32 requires xk6-output-kafka that requires 0.31

go: go.k6.io/[email protected] requires
github.com/k6io/[email protected] requires
go.k6.io/[email protected] requires
github.com/k6io/[email protected]: invalid version: unknown revision a159d7c

Document config options

The part of the Readme related to the configuration is linking to the old Kafka documentation that it is not providing a detailed documentation about the available options.

The suggestion is to add a table listing all the dedicated options that this extension is adding.

Wrong case for json options

We are using camelCase for the json options in k6 core outputs, ideally, we should use the same in this project to be consistent.

The suggestion is to check the Config fields and fix the current options with the wrong case.

Remove dependency on k6's json and InfluxDB outputs

Both of the dependencies are so Kafka can format metrics in both of the formats.

While it's probably not great to copy-paste this code the alternative is that any changes to those might break this extension which especially as time goes by will not be an easy thing to avoid.

Even now the InfluxDB output will need to be refactored and moved from the old "collector" interface to the "Output" one, which will break this extension 😭.

Additionally, for InfluxDB we should probably copy 1-2 configurations such as tagsAsFields and precision.

We also intend on updating the influxdb library in k6 to support v2 which may or may not mean we will drop v1, but also means that the API will likely change and that

Given all of the above, I do think it's probably better to do this change now and drop any dependency on k6 that isn't the Output interface and its registration

Need CI with Github Actions

Need automated build and testing.

Update for k6 v0.41.0

Move to v2 of the influxdb api

it is only used to make the influxdb points so it should be 100% forward compatible given what I know of the problem, but maybe it will have slightly better performance and won't use a library that is deprecated at this point

SyncProducer performance

Hi 👋

I'm currently using this extension in the company I work and we're having some issues regarding the performance of sending the messages to our Kafka Broker. In order to speed up things, me and @knmsk are currently working on a full rewrite to match the new Output struct in k6 repository and also changing the SyncProducer to AsyncProducer.

The Idea is to improve throughput by using the AsyncProducer, batching messages and running in go routines. Also, we think that we should include an option to wait for acks from Kafka or not. This should be a choice for the users if they're willing to lose some messages to gain performance

Please let us know what you think 😄