leonyork / xk6-output-timestream Goto Github PK
View Code? Open in Web Editor NEWOutput metrics from K6 to AWS Timestream
License: GNU Affero General Public License v3.0
Output metrics from K6 to AWS Timestream
License: GNU Affero General Public License v3.0
Hello guys! I faced the issue during building xk6 with xk6-output-timestream and xk6-sql extensions.
Since the v0.8.29
version of the xk6-output-timestream extension, I discovered that the build process failed.
When I run the following command: xk6 build v0.47.0 --with github.com/grafana/xk6-sql --with github.com/leonyork/[email protected]
I'm getting an error.
2023/11/29 18:29:24 [INFO] Initializing Go module
2023/11/29 18:29:24 [INFO] exec (timeout=10s): /opt/homebrew/bin/go mod init k6
go: creating new go.mod: module k6
2023/11/29 18:29:24 [INFO] Pinning versions
2023/11/29 18:29:24 [INFO] exec (timeout=0s): /opt/homebrew/bin/go mod edit -require github.com/grafana/xk6-sql@latest
2023/11/29 18:29:24 [INFO] exec (timeout=0s): /opt/homebrew/bin/go mod tidy -compat=1.17
go: finding module for package github.com/nxadm/tail
go: found github.com/nxadm/tail in github.com/nxadm/tail v1.4.11
2023/11/29 18:29:24 [INFO] exec (timeout=0s): /opt/homebrew/bin/go mod edit -require github.com/leonyork/[email protected]
2023/11/29 18:29:25 [INFO] exec (timeout=0s): /opt/homebrew/bin/go mod tidy -compat=1.17
go: k6 imports
github.com/grafana/xk6-sql imports
go.k6.io/k6/js/modules imports
go.k6.io/k6/loader tested by
go.k6.io/k6/loader.test imports
go.k6.io/k6/lib/testutils/httpmultibin imports
google.golang.org/grpc/status imports
google.golang.org/genproto/googleapis/rpc/status: ambiguous import: found package google.golang.org/genproto/googleapis/rpc/status in multiple modules:
google.golang.org/genproto v0.0.0-20210226172003-ab064af71705 (/Users/dmytro/go/pkg/mod/google.golang.org/[email protected]/googleapis/rpc/status)
google.golang.org/genproto/googleapis/rpc v0.0.0-20230525234030-28d5490b6b19 (/Users/dmytro/go/pkg/mod/google.golang.org/genproto/googleapis/[email protected]/status)
2023/11/29 18:29:25 [INFO] Cleaning up temporary folder: /Users/dmytro/go/bin/buildenv_2023-11-29-1829.65010214
2023/11/29 18:29:25 [FATAL] exit status 1
For xk6-output-timestream extension of v0.8.28
it works fine, build is done.
Go version: go version go1.21.4 darwin/arm64
Could you please take a look at this issue so we can use the latest version of your extension? Thank you!
First, thanks for writing this extension and sharing it in office hours as well as in community forums. I have zero go skills today, otherwise I might have already dove in and submitted a PR on this.
It looks like when I run tests using k6 groups or derivatives, like the chaijs/expect library which I am using, I end up with records with a dimension called 'group' which are successfully written to the table, but it appears that the dimension can't be used in query syntax in a where clause for example. Timestream documentation lists 'group' as a reserved word, which is understandable I suppose. Another interesting reservation is on dimensions or such with names that start with ts_ which I learned the hard way.
One approach might be to prefix "group" with "k6_" but you may have some other thoughts or preferences. Cheers!
I tested this extension out yesterday and the dashboard does not seem to be matching up with what is being sent to TimeStream.
The dashboard is looking for a measure called "vus" but that is not being sent to TimeStream.
"targets": [
{
"alias": "Active VUs",
"database": "\"$Database\"",
"datasource": {
"uid": "${grafana-timestream-datasource}"
},
"dsType": "grafana-timestream-datasource",
"measure": "vus",
"policy": "default",
"rawQuery": "SELECT BIN(time, $__interval) AS time, AVG($__measure) AS $__measure\nFROM (\n SELECT time, SUM($__measure) AS $__measure\n FROM (\n SELECT BIN(time, 1s) AS time, AVG(measure_value::double) AS $__measure\n FROM $__database.$__table\n WHERE measure_name = '$__measure'\n AND $__timeFilter\n GROUP BY BIN(time, 1s)\n ORDER BY BIN(time, 1s) DESC\n )\n GROUP BY time\n)\nGROUP BY BIN(time, $__interval)",
"refId": "A",
"resultFormat": "time_series",
"table": "\"$Table\"",
"tags": [],
"waitForResult": true
}
],
"title": "Virtual Users",
Also the dashboard (grafana/dashboards/loadtest/loadtest.json) is looking for a column called instance_id
and that column is not being sent to TimeStream.
I tried to run this image by this command
docker run --network="host" -i ghcr.io/leonyork/k6:0.48.0-timestream0.8.66 run -o timestream \
-e K6_TIMESTREAM_DATABASE_NAME='ipay_timestream_db1' \
-e K6_TIMESTREAM_TABLE_NAME='load-test' \
-e K6_TIMESTREAM_REGION='us-west-2' \
-e AWS_REGION='us-west-2' - < ./scripts/max-vu.js
But it doesn't seem to pick up the database name and able name as it declared. The output was like
/\ |‾‾| /‾‾/ /‾‾/
/\ / \ | |/ / / /
/ \/ \ | ( / ‾‾\
/ \ | |\ \ | (‾) |
/ __________ \ |__| \__\ \_____/ .io
execution: local
script: -
output: Timestream (:)
I checked the source code which claim to handle environment vars but not doing so in code:
// GetConsolidatedConfig combines {default config values + JSON config +
// environment vars config values}, and returns the final result.
func GetConsolidatedConfig(
jsonRawConf json.RawMessage,
) (Config, error) {
result := NewConfig()
if jsonRawConf != nil {
jsonConf, err := parseJSON(jsonRawConf)
if err != nil {
return result, fmt.Errorf("unable to parse json config: %w", err)
}
result = result.apply(jsonConf)
}
envConfig := Config{}
if err := envconfig.Process("", &envConfig); err != nil {
return result, fmt.Errorf("unable to parse env config: %w", err)
}
result = result.apply(envConfig)
return result, nil
}
Whereas other output extensions's config consolidation logic does read env from a given parameter.
https://github.com/grafana/xk6-output-influxdb/blob/main/pkg/influxdb/config.go
Maybe I missed some magi code, but would love to know if there is a way to make the environment variable work.
When I run tests with the timestream output I am getting some good records ingested but I see some errors messages around records in the batch without dimensions. I am not sure if standard k6 output is generating these, or if it is a custom metric. I may try my hand at adding some debugging output to the extension but I have not done anything in go yet. Here is an example error:
ERRO[0024] failed to write component=timestream count=100 duration=72.525166ms error="unable to write records to timestream: operation error Timestream Write: WriteRecords, https response error StatusCode: 400, RequestID: 2M4VERUETG3GAX6EVVQR4LCI7M, ValidationException: At least one dimension is required for a record." records_address=0xc0122da2c8 t=20.122957185s
Cheers!
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
test/docker-compose.yml
nginx 1.27.0-alpine
.devcontainer/Dockerfile
docker/dockerfile 1
golang 1.22.5-bookworm
docker 27.0.3-cli
node 21.7.3-bookworm-slim
hadolint/hadolint v2.12.0-debian
golangci/golangci-lint v1.59.1
Dockerfile
docker/dockerfile 1
grafana/k6 0.52.0
grafana/Dockerfile
grafana/grafana-oss 11.0.0
test/Dockerfile
golang 1.20.5-buster
test/test.Dockerfile
.github/actions/build-builder/action.yaml
docker/setup-buildx-action v3
docker/build-push-action v6
.github/actions/integration-tests/action.yaml
aws-actions/configure-aws-credentials v4
.github/workflows/_build-grafana.yaml
actions/checkout v4
ubuntu 22.04
.github/workflows/_tag-release.yaml
actions/checkout v4
ubuntu 22.04
.github/workflows/build-and-test.yaml
actions/checkout v4
ubuntu 22.04
.github/workflows/build-test-tag.yaml
actions/checkout v4
actions/upload-artifact v4
docker/setup-buildx-action v3
actions/checkout v4
actions/checkout v4
gembaadvantage/uplift-action v2
ubuntu 22.04
ubuntu 22.04
ubuntu 22.04
ubuntu 22.04
.github/workflows/release-tagged.yaml
actions/checkout v4
gembaadvantage/uplift-action v2
docker/setup-buildx-action v3
dawidd6/action-download-artifact v6
dawidd6/action-download-artifact v6
ubuntu 22.04
.github/workflows/validate.yaml
actions/checkout v4
ubuntu 22.04
go.mod
go 1.20
github.com/aws/aws-sdk-go-v2 v1.30.3
github.com/aws/aws-sdk-go-v2/config v1.27.26
github.com/aws/aws-sdk-go-v2/service/timestreamwrite v1.27.3
github.com/mstoykov/envconfig v1.5.0
github.com/sirupsen/logrus v1.9.3
github.com/stretchr/testify v1.9.0
go.k6.io/k6 v0.52.0
test/go.mod
go 1.19
.devcontainer/Dockerfile
grafana/k6 0.52.0
go.k6.io/xk6 v0.12.1
debian_12/unzip 6.0
mvdan.cc/sh/v3 v3.8.0
debian_12/less 590
prettier 3.3.3
gembaadvantage/uplift v2.24.1
Dockerfile
grafana/k6 0.52.0
Makefile
grafana/k6 0.52.0
grafana/Makefile
grafana/grafana-oss 11.0.0
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.