leonyork / xk6-output-timestream Goto Github PK

Hello guys! I faced the issue during building xk6 with xk6-output-timestream and xk6-sql extensions.

Since the v0.8.29 version of the xk6-output-timestream extension, I discovered that the build process failed.

When I run the following command: xk6 build v0.47.0 --with github.com/grafana/xk6-sql --with github.com/leonyork/[email protected] I'm getting an error.

2023/11/29 18:29:24 [INFO] Initializing Go module
2023/11/29 18:29:24 [INFO] exec (timeout=10s): /opt/homebrew/bin/go mod init k6 
go: creating new go.mod: module k6
2023/11/29 18:29:24 [INFO] Pinning versions
2023/11/29 18:29:24 [INFO] exec (timeout=0s): /opt/homebrew/bin/go mod edit -require github.com/grafana/xk6-sql@latest 
2023/11/29 18:29:24 [INFO] exec (timeout=0s): /opt/homebrew/bin/go mod tidy -compat=1.17 
go: finding module for package github.com/nxadm/tail
go: found github.com/nxadm/tail in github.com/nxadm/tail v1.4.11
2023/11/29 18:29:24 [INFO] exec (timeout=0s): /opt/homebrew/bin/go mod edit -require github.com/leonyork/[email protected] 
2023/11/29 18:29:25 [INFO] exec (timeout=0s): /opt/homebrew/bin/go mod tidy -compat=1.17 
go: k6 imports
	github.com/grafana/xk6-sql imports
	go.k6.io/k6/js/modules imports
	go.k6.io/k6/loader tested by
	go.k6.io/k6/loader.test imports
	go.k6.io/k6/lib/testutils/httpmultibin imports
	google.golang.org/grpc/status imports
	google.golang.org/genproto/googleapis/rpc/status: ambiguous import: found package google.golang.org/genproto/googleapis/rpc/status in multiple modules:
	google.golang.org/genproto v0.0.0-20210226172003-ab064af71705 (/Users/dmytro/go/pkg/mod/google.golang.org/[email protected]/googleapis/rpc/status)
	google.golang.org/genproto/googleapis/rpc v0.0.0-20230525234030-28d5490b6b19 (/Users/dmytro/go/pkg/mod/google.golang.org/genproto/googleapis/[email protected]/status)
2023/11/29 18:29:25 [INFO] Cleaning up temporary folder: /Users/dmytro/go/bin/buildenv_2023-11-29-1829.65010214
2023/11/29 18:29:25 [FATAL] exit status 1

For xk6-output-timestream extension of v0.8.28 it works fine, build is done.
Go version: go version go1.21.4 darwin/arm64

Could you please take a look at this issue so we can use the latest version of your extension? Thank you!

First, thanks for writing this extension and sharing it in office hours as well as in community forums. I have zero go skills today, otherwise I might have already dove in and submitted a PR on this.

It looks like when I run tests using k6 groups or derivatives, like the chaijs/expect library which I am using, I end up with records with a dimension called 'group' which are successfully written to the table, but it appears that the dimension can't be used in query syntax in a where clause for example. Timestream documentation lists 'group' as a reserved word, which is understandable I suppose. Another interesting reservation is on dimensions or such with names that start with ts_ which I learned the hard way.

One approach might be to prefix "group" with "k6_" but you may have some other thoughts or preferences. Cheers!

I tested this extension out yesterday and the dashboard does not seem to be matching up with what is being sent to TimeStream.
The dashboard is looking for a measure called "vus" but that is not being sent to TimeStream.

      "targets": [
        {
          "alias": "Active VUs",
          "database": "\"$Database\"",
          "datasource": {
            "uid": "${grafana-timestream-datasource}"
          },
          "dsType": "grafana-timestream-datasource",
          "measure": "vus",
          "policy": "default",
          "rawQuery": "SELECT BIN(time, $__interval) AS time, AVG($__measure) AS $__measure\nFROM (\n  SELECT time, SUM($__measure) AS $__measure\n  FROM (\n    SELECT BIN(time, 1s) AS time, AVG(measure_value::double) AS $__measure\n    FROM $__database.$__table\n    WHERE measure_name = '$__measure'\n    AND $__timeFilter\n    GROUP BY BIN(time, 1s)\n    ORDER BY BIN(time, 1s) DESC\n  )\n  GROUP BY time\n)\nGROUP BY BIN(time, $__interval)",
          "refId": "A",
          "resultFormat": "time_series",
          "table": "\"$Table\"",
          "tags": [],
          "waitForResult": true
        }
      ],
      "title": "Virtual Users",

Also the dashboard (grafana/dashboards/loadtest/loadtest.json) is looking for a column called instance_id and that column is not being sent to TimeStream.

I tried to run this image by this command

 docker run --network="host" -i ghcr.io/leonyork/k6:0.48.0-timestream0.8.66 run -o timestream \
        -e K6_TIMESTREAM_DATABASE_NAME='ipay_timestream_db1' \
        -e K6_TIMESTREAM_TABLE_NAME='load-test' \
        -e K6_TIMESTREAM_REGION='us-west-2' \
        -e AWS_REGION='us-west-2'  - < ./scripts/max-vu.js

But it doesn't seem to pick up the database name and able name as it declared. The output was like

          /\      |‾‾| /‾‾/   /‾‾/   
     /\  /  \     |  |/  /   /  /    
    /  \/    \    |     (   /   ‾‾\  
   /          \   |  |\  \ |  (‾)  | 
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: -
     output: Timestream (:)

I checked the source code which claim to handle environment vars but not doing so in code:

// GetConsolidatedConfig combines {default config values + JSON config +
// environment vars config values}, and returns the final result.
func GetConsolidatedConfig(
	jsonRawConf json.RawMessage,
) (Config, error) {
	result := NewConfig()

	if jsonRawConf != nil {
		jsonConf, err := parseJSON(jsonRawConf)
		if err != nil {
			return result, fmt.Errorf("unable to parse json config: %w", err)
		}

		result = result.apply(jsonConf)
	}

	envConfig := Config{}
	if err := envconfig.Process("", &envConfig); err != nil {
		return result, fmt.Errorf("unable to parse env config: %w", err)
	}

	result = result.apply(envConfig)

	return result, nil
}

Whereas other output extensions's config consolidation logic does read env from a given parameter.

https://github.com/grafana/xk6-output-influxdb/blob/main/pkg/influxdb/config.go

Maybe I missed some magi code, but would love to know if there is a way to make the environment variable work.

When I run tests with the timestream output I am getting some good records ingested but I see some errors messages around records in the batch without dimensions. I am not sure if standard k6 output is generating these, or if it is a custom metric. I may try my hand at adding some debugging output to the extension but I have not done anything in go yet. Here is an example error:

ERRO[0024] failed to write component=timestream count=100 duration=72.525166ms error="unable to write records to timestream: operation error Timestream Write: WriteRecords, https response error StatusCode: 400, RequestID: 2M4VERUETG3GAX6EVVQR4LCI7M, ValidationException: At least one dimension is required for a record." records_address=0xc0122da2c8 t=20.122957185s

Cheers!

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update grafana/grafana-oss docker tag to v11.1.0

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository