influxdata / influx-stress

New tool for generating artificial load on InfluxDB

This repository has been archived and is no longer maintained. Please see the inch benchmarking tool instead.

Stress tool

Build Instructions

Building influx-stress requires the Golang toolchain. If you do not have the Golang toolchain installed please follow the instructions golang.org/doc/install

go get -v github.com/influxdata/influx-stress/cmd/...

Top Level Command

Create artificial load on an InfluxDB instance

Usage:
  influx-stress [command]

  Available Commands:
    insert      Insert data into InfluxDB

    Flags:
      -h, --help   help for influx-stress

      Use "influx-stress [command] --help" for more information about a command.

Insert Subcommand

Insert data into InfluxDB

Usage:
  influx-stress insert SERIES FIELDS [flags]

Flags:
  -b, --batch-size uint      number of points in a batch (default 10000)
  -c, --consistency string   Write consistency (only applicable to clusters) (default "one")
      --create string        Use a custom create database command
      --db string            Database that will be written to (default "stress")
      --dump string          Dump to given file instead of writing over HTTP
  -f, --fast                 Run as fast as possible
      --gzip int             If non-zero, gzip write bodies with given compression level. 1=best speed, 9=best compression, -1=gzip default.
      --host string          Address of InfluxDB instance (default "http://localhost:8086")
      --pass string          Password for user
  -n, --points uint          number of points that will be written (default 18446744073709551615)
      --pps uint             Points Per Second (default 200000)
  -p, --precision string     Resolution of data being written (default "n")
  -q, --quiet                Only print the write throughput
      --rp string            Retention Policy that will be written to
  -r, --runtime duration     Total time that the test will run (default 2562047h47m16.854775807s)
  -s, --series int           number of series that will be written (default 100000)
      --strict               Strict mode will exit as soon as an error or unexpected status is encountered
      --user string          User to write data as

Example Usage

Runs forever

$ influx-stress insert

Runs forever writing as fast as possible

$ influx-stress insert -f

Runs for 1 minute writing as fast as possible

$ influx-stress insert -r 1m -f

Writing an example series key

$ influx-stress insert cpu,host=server,location=us-west,id=myid

Writing an example series key with 20,000 series

$ influx-stress insert -s 20000 cpu,host=server,location=us-west,id=myid

Writing an example point

$ influx-stress insert cpu,host=server,location=us-west,id=myid busy=100,idle=10,random=5i

influx-stress's Issues

Switch to dep instead of gdm.

Also check in vendored dependencies.

Add query subcommand

The tool is suitable only write queries. (Maybe development team will add also read part of it.)

UDP support

Is sending via UDP supported?

General Feedback

• The tool is suitable only write queries. (Maybe development team will add also read part of it.)
• When we start test, while influx-stress is writing data to db, client (running test) cannot access to influxDb concurrently. And also, almost half of insert query get an error as "Error sending write: dialing to the given TCP address timed out". I think stress test try to use whole network and it leads to overload. (This is not a bug.)
• The documentation in github is really incomplete. I found some keys in code such as "username" is "user", "password" is "pass". I thought keywords would be the same with influxdb.
• When the test is finished, the total write data is prompt in console, if you compare the count with influxdb, they are not equal. (For example; stress test say 9.000.000 points is written, but when I query the db with count is almost 7 million.)

URL string returned is invalid

My influxdb and influx-stress are in different hosts.
Command is: ./influx-stress insert cpu,cluster=xxx,dc=xxx,host=xxx,cpu=xxx --batch-size 10000 --host 192.168.12.5 --points 1 --pps 10000 --series 10000 --stats-host 192.168.12.5 --kapacitor true
Test is failure, and error log is: no such host.

I modified writeURLFromConfig as following:
func writeURLFromConfig(cfg ClientConfig) string {
params := url.Values{}
params.Set("db", cfg.Database)
if cfg.User != "" {
params.Set("u", cfg.User)
}
if cfg.Pass != "" {
params.Set("p", cfg.Pass)
}
if cfg.RetentionPolicy != "" {
params.Set("rp", cfg.RetentionPolicy)
}
if cfg.Precision != "n" && cfg.Precision != "" {
params.Set("precision", cfg.Precision)
}
if cfg.Consistency != "one" && cfg.Consistency != "" {
params.Set("consistency", cfg.Consistency)
}

return "http://" + cfg.BaseURL + ":8086" + "/write?" + params.Encode()

}
With new code, influx-stress runs ok.

influx-stress/write/client.go

Line 221 in 5557223

return cfg.BaseURL + "/write?" + params.Encode()

Build Instructions.

Build instructions do not build on Ubuntu 16.04.

Needed to :
git clone https://github.com/influxdata/influx-stress.git

then :

cd influx-stress/cmd/main/
go build

Add build instructions to the readme

Change `user` and `pass` flags to `username` and `password`

The documentation in github is really incomplete. I found some keys in code such as "username" is "user", "password" is "pass". I thought keywords would be the same with influxdb.

Add Godeps file

Related to #6

Move Sink logic outside of insert.go

panic at program termination

Tested on Ubtuntu 16.04 and Windows 10 Pro, same behavior:

Using point template: ctr,some=tag n=0i <timestamp>
Using batch size of 10000 line(s)
Spreading writes across 100000 series
Output is unthrottled
Using 20 concurrent writer(s)
Running until ~18446744073709551615 points sent or until ~1m0s has elapsed
panic: send on closed channel

goroutine 49 [running]:
github.com/influxdata/influx-stress/cmd.(*multiSink).run(0xc04207c5a0)
        C:/Users/Andreone/go/src/github.com/influxdata/influx-stress/cmd/insert.go:306 +0x109
created by github.com/influxdata/influx-stress/cmd.(*multiSink).Open
        C:/Users/Andreone/go/src/github.com/influxdata/influx-stress/cmd/insert.go:298 +0x9b

Failed to create database: Bad status code during Create(CREATE DATABASE stress): 400, body: "missing parameter: db"

[root@53 bin]# ./influx-stress insert cpu,host=server,location=us-west,id=myid busy=100,idle=10,random=5i -r 30s --host http://127.0.0.1:9096/write?u=admin&p=admin
[1] 2695
[root@53 bin]# Using point template: cpu,host=server,location=us-west,id=myid busy=100,idle=10,random=5i
Using batch size of 10000 line(s)
Spreading writes across 100000 series
Throttling output to ~200000 points/sec
Using 20 concurrent writer(s)
Running until ~18446744073709551615 points sent or until ~30s has elapsed
Failed to create database: Bad status code during Create(CREATE DATABASE stress): 400, body: "missing parameter: db"
Aborting.

Flag to retry failed request

. If there were failed writes, this is partially by design, but can be changed. At the moment, we don't retry failed writes, I'll add a flag that retries until success.

influxdb-stress 连接influxdb 失败

[root@cdh6-pre2038 bin]# /root/go/bin/influx-stress insert -r 60s --strict --pps 800000 --host http://172.20.20.38:8086
Using point template: ctr,some=tag n=0i
Using batch size of 10000 line(s)
Spreading writes across 100000 series
Throttling output to ~800000 points/sec
Using 80 concurrent writer(s)
Running until ~18446744073709551615 points sent or until ~1m0s has elapsed
Failed to create database: Bad status code during Create(CREATE DATABASE stress): 401, body: {"code":"unauthorized","message":"unauthorized access"}
Aborting.

我的influxdb版本是2.0.8 然后说权限认证不通过 但是我怎么加上token呢
@chengshiwen

Fields values are ignored?

I'm running influx1.3.3. and the latest influx-stress binary.
influx-stress insert --pps 1 -n 10 cpu,host=server field1=100.0

select * from cpu
name: cpu
time                field1 host
----                ---- ----
1503086572364559306 0    server-0
1503086573367787961 0    server-1
1503086574366653212 0    server-2
1503086575368854918 0    server-3
1503086576369727488 0    server-4
1503086577365074040 0    server-5
1503086578368212529 0    server-6
1503086579369044237 0    server-7

Create a distribution of influx-stress

Build instructions incomplete/misleading

The build instructions don't explain how to get the code or put it in the right place in the GOPATH. An experienced Go developer will know what to do, but others have problems. I had to help someone at my company work through the gaps.

Also, the build command is more complicated than it needs to be. Go will install transitive dependencies automatically so the go install command only needs to target the main packages.

For anyone with a properly installed Go toolchain the following command will take care of both downloading the project code and building the binary. This is the command I suggested to my co-worker and it was much easier for them to use:

go get -v github.com/influxdata/influx-stress/cmd/...

Support TLS Connections

Our InfluxDB is TLS-only. Could we please update the stress test to allow for connections of that type?

influx-stress not generating random values instead giving only 0s.

As suggested, when I try to enter load using the influx-stress insert command:
influx-stress insert stress,data=test value=11.0,quality=2 -r 1s

It is inserting 0 in value and quantity column.
What I am expecting is a way to enter float or int random values to be inserted under the columns.

Configure influx-stress with environment variables

Some of the data associated influx-stress is sensitive. Being able to set these, and other, parameters through environment variables would be an nice experience.

No writes happen in case of series count less than 20

Issuing the command:

./influx-stress insert -r 1m -s 19 -f --host "http://<ip-addr>:8086"
the influx_stress tool doesn't write any data to influxd.

But for data of 20 and above, the writes happen fine.

Directions to resolve this would be appreciated.

超过500万条数据的插入会报错

[2019-06-30 22:15:09] Error sending write: dialing to the given TCP address timed out
[2019-06-30 22:15:09] Error sending write: dialing to the given TCP address timed out
[2019-06-30 22:15:09] Error sending write: dialing to the given TCP address timed out
[2019-06-30 22:15:09] Error sending write: dialing to the given TCP address timed out
[2019-06-30 22:15:09] Error sending write: dialing to the given TCP address timed out
[2019-06-30 22:15:09] Error sending write: dialing to the given TCP address timed out
[2019-06-30 22:15:09] Error sending write: dialing to the given TCP address timed out
[2019-06-30 22:15:09] Error sending write: dialing to the given TCP address timed out
[2019-06-30 22:15:09] Error sending write: dialing to the given TCP address timed out
[2019-06-30 22:15:09] Error sending write: dialing to the given TCP address timed out
panic: send on closed channel

goroutine 52 [running]:
github.com/influxdata/influx-stress/cmd.(*multiSink).run(0xc4200925d0)
/root/go/src/github.com/influxdata/influx-stress/cmd/insert.go:306 +0x102
created by github.com/influxdata/influx-stress/cmd.(*multiSink).Open
/root/go/src/github.com/influxdata/influx-stress/cmd/insert.go:298 +0x94