rogerwelin / cassowary

:rocket: Modern cross-platform HTTP load-testing tool written in Go

Hi 👋 My name is Roger Welin

DevOps, SRE & Developer

• 🌍  I'm based in Stockholm
• ✉️  You can contact me at [email protected]
• 🚀  I'm currently working on cfnctl
• 👉 You can check out my tech blog here
• 🧠  I'm learning Zig
• ⚡  Cloud Architect, DevOps, gopher, tool builder, developer, open source contributor, featured tech blogger

Skills

Socials

Badges

My GitHub Stats

cassowary's Issues

Enhancement: add plugins

Have you considered writing a plugin interface for cassowary? I make a lot of use of wrk because of that capability and it's really useful. (That could totally solve #15 without having to add any flags or anything like that, as long as there's a hook point in the right place). Would love to have that here because the feature set and metrics that you've got are far superior.

I wouldn't mind taking a shot at it, if that's the direction you want this project to go. Just say the word.

PATCH support is missing

Hi, I needed PATCH support similar to POST and I have modified the code in my local machine and it will be useful for others too. Could you please give me permission to contribute to this open source project with those changes by raising a pull request? thanks.

[Chore Feature Request]: Use cassowary_linux_x86_64.zip instead of cassowary_{VERSION}_Linux_x86_64.tar.gz.zip

Hi @rogerwelin and contributors.

Thanks to amazing tool.
I use cassowary and i want integrate downloads as binary, but i can't download LATEST version always, i need modify my ansible playbook to use.
I.e i currently can't download via https://github.com/rogerwelin/cassowary/releases/latest/download/cassowary_Linux_x86_64.tar.gz

It is possible change release build binary names to cassowary_Linux_x86_64.tar.gz or something static ?

Thanks again

Fails and exits with context deadline exceeded (Client.Timeout exceeded while awaiting headers)

When running cassowary run -u http://webapp.local -c 5 -f test_urls2.txt against a service I am gathering evidence on I am met with an abrupt exit, no stats and the following output:

2021/12/02 04:19:21 Get "http://webapp.local/page-44/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Is there a way to increase the timeout?
To my mind this should fall under "Failed Req" and the final stats output is the expected outcome.

Improve code coverage

Code coverage is rather low at ~22%. Should be raised to at least 50%

panic running the test

centos-7 x86_64 VM

$ cassowary/cassowary -v
cassowary - 學名 version 0.9.0
$ cassowary/cassowary run -u https://dev.test -c 500 -n 100000 --disable-keep-alive

Starting Load Test with 100000 requests using 500 concurrent users

  14% |█████                                   | [44s:4m36s]            panic: Get "https://dev.test": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

goroutine 262 [running]:
github.com/rogerwelin/cassowary/pkg/client.(*Cassowary).runLoadTest(0xc0002dc000, 0xc000300000, 0xc00009c060)
        /Users/roger/programmering/cassowary/pkg/client/load.go:93 +0x144a
github.com/rogerwelin/cassowary/pkg/client.(*Cassowary).Coordinate.func1(0xc0002dc000, 0xc000300000, 0xc00009c060, 0xc000022630)
        /Users/roger/programmering/cassowary/pkg/client/load.go:180 +0x3f
created by github.com/rogerwelin/cassowary/pkg/client.(*Cassowary).Coordinate
        /Users/roger/programmering/cassowary/pkg/client/load.go:179 +0x2b6````

passing different request Headers per each request from an external file

Is it possible to pass different (random) request Headers per each http request from an external file? if not, would you like to add this feature?

my use case is following, I would like to load test my application but for different user accounts in order to eliminate non-realistic cache hits. my app expects 'Authorization' request header to identify the end user.

POST/PUT data

cassowary should support POST/PUT data defined in a file

Question about timing the server processing

I can see in the line below that what you call ServerProcessing is the time between when the connection's been established and we get the response's first byte.

https://github.com/rogerwelin/cassowary/blob/master/pkg/client/load.go#L117

But according to Go's httptrace docs there's a hook called WroteRequest that seems to me more accurate to the start of the remote server process.

I remember digging a little deeper into the request's lifecycle when I wrote a similar feature but I'm no expert so it's just me being curious.

Why did you go with GotConn hook instead?

Thanks in advance.

Add proxy support

Add proxy support to enable cassowary to run in restrictive enterprise environments where all traffic must pass the proxy

is the Slurp mode gone?

I downloaded the latest release and the example for the slurp mode -f option is not working anymore, the option -f is not present anymore, is this intentional?

Cheers

Build issue

Could you please provide details of how to build the package, a standard Go way fails with many errors, e.g.

go version
go version go1.13.3 linux/amd64

go get github.com/rogerwelin/cassowary
# github.com/rogerwelin/cassowary
go/src/github.com/rogerwelin/cassowary/cli.go:140:19: cannot use cli.StringFlag literal (type cli.StringFlag) as type cli.Flag in array or slice literal:
        cli.StringFlag does not implement cli.Flag (Apply method has pointer receiver)
go/src/github.com/rogerwelin/cassowary/cli.go:145:16: cannot use cli.IntFlag literal (type cli.IntFlag) as type cli.Flag in array or slice literal:
        cli.IntFlag does not implement cli.Flag (Apply method has pointer receiver)
go/src/github.com/rogerwelin/cassowary/cli.go:150:19: cannot use cli.StringFlag literal (type cli.StringFlag) as type cli.Flag in array or slice literal:
        cli.StringFlag does not implement cli.Flag (Apply method has pointer receiver)
go/src/github.com/rogerwelin/cassowary/cli.go:155:19: cannot use cli.StringFlag literal (type cli.StringFlag) as type cli.Flag in array or slice literal:
        cli.StringFlag does not implement cli.Flag (Apply method has pointer receiver)
go/src/github.com/rogerwelin/cassowary/cli.go:159:19: cannot use cli.StringFlag literal (type cli.StringFlag) as type cli.Flag in array or slice literal:
        cli.StringFlag does not implement cli.Flag (Apply method has pointer receiver)
go/src/github.com/rogerwelin/cassowary/cli.go:163:17: cannot use cli.BoolFlag literal (type cli.BoolFlag) as type cli.Flag in array or slice literal:
        cli.BoolFlag does not implement cli.Flag (Apply method has pointer receiver)
go/src/github.com/rogerwelin/cassowary/cli.go:167:19: cannot use cli.StringFlag literal (type cli.StringFlag) as type cli.Flag in array or slice literal:
        cli.StringFlag does not implement cli.Flag (Apply method has pointer receiver)
go/src/github.com/rogerwelin/cassowary/cli.go:178:19: cannot use cli.StringFlag literal (type cli.StringFlag) as type cli.Flag in array or slice literal:
        cli.StringFlag does not implement cli.Flag (Apply method has pointer receiver)
go/src/github.com/rogerwelin/cassowary/cli.go:183:16: cannot use cli.IntFlag literal (type cli.IntFlag) as type cli.Flag in array or slice literal:
        cli.IntFlag does not implement cli.Flag (Apply method has pointer receiver)
go/src/github.com/rogerwelin/cassowary/cli.go:188:16: cannot use cli.IntFlag literal (type cli.IntFlag) as type cli.Flag in array or slice literal:
        cli.IntFlag does not implement cli.Flag (Apply method has pointer receiver)
go/src/github.com/rogerwelin/cassowary/cli.go:188:16: too many errors

[Package] Homebrew page?

Hi @rogerwelin and contributors.

Thanks for such a great work.
It is possible Homebrew package release?
Can you're publish to Homebrew?

Bug - use of Host header not working

When using a header of type "Host: www.xxxx.com", the request received by the server doesn't contain the host name.
In the pkg/client/load.go, the header processing block should look like something like this:

if len(c.RequestHeader) == 2 {
if c.RequestHeader[0] == "Host" {
request.Host = c.RequestHeader[1]
request.URL.Host = c.RequestHeader[1]
} else {
request.Header.Add(c.RequestHeader[0], c.RequestHeader[1])
}
}

Enhancement: add support for macports

It'd be great if this amazing tool was also available via macports since not everyone on the mac uses homebrew. Thanks!

feature request: more percentiles

I find it's often useful to see the p99/p100 behavior of a system. I think a spread of p90, p99, p100 is more informative than p95 only, though I understand it's nice to have a single metric.

I can see where in the code to start, but I am curious if it violates your design tenets. If so it's not worth doing a PR.

What do you think?

[FEATURE REQUEST] Add a `Throughput` in the Summary

Hello @rogerwelin,

First of all I have to say that this is a great project with clean source code, I was able to understand the code flow very quickly, good job!

I just want to add a Throughput label in the Summary section of the results output. I think it's quite useful for us, who develop web frameworks, to see that number right away. Also, you can move the CLI part package main on a "./cmd/cassowary" directory so we can use the github.com/rogewelin/cassowary package as a library too to take the results directly. I know that you have json output option too but it will require from us to download executable based on the host OS, then exec.Cmd, then read the json file and unmarshal it which it is quite slow operation and requires many steps that can fail.

• https://en.wikipedia.org/wiki/Measuring_network_throughput

Thanks again,
Gerasimos Maropoulos. Author of Iris web framework

add support for x509 authentication

All our services are behind x509 authentication, could you please add support for it. For instance I provided x509 support for hey tool here https://github.com/vkuznet/hey/tree/x509 via https://github.com/vkuznet/x509proxy

panic: runtime error: index out of range [-1]

Running a simple test:

$ cassowary run -u http://localhost:2015 -c 10 -n 100000

Starting Load Test with 100000 requests using 10 concurrent users

 100% |████████████████████████████████████████| [3s:0s]            3.000194405s
panic: runtime error: index out of range [-1]

goroutine 1 [running]:
main.calcMedian(0x0, 0x0, 0x0, 0x17800)
	/Users/rogerwelin/programming/cassowary/stats.go:29 +0xd8
main.(*cassowary).coordinate(0xc00001e700, 0xc00001e700, 0xc00014c120)
	/Users/rogerwelin/programming/cassowary/load.go:203 +0xe04
main.validateRun(0xc0000d4c60, 0x0, 0xc000176060)
	/Users/rogerwelin/programming/cassowary/cli.go:76 +0x35c
github.com/urfave/cli.HandleAction(0x1453980, 0x150e5c0, 0xc0000d4c60, 0xc0000d4c60, 0x0)
	/Users/rogerwelin/go/pkg/mod/github.com/urfave/[email protected]/app.go:523 +0xbe
github.com/urfave/cli.Command.Run(0x14e9f41, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0, 0x14f1292, 0xf, 0x0, ...)
	/Users/rogerwelin/go/pkg/mod/github.com/urfave/[email protected]/command.go:174 +0x51c
github.com/urfave/cli.(*App).Run(0xc000130380, 0xc00001e100, 0x8, 0x8, 0x0, 0x0)
	/Users/rogerwelin/go/pkg/mod/github.com/urfave/[email protected]/app.go:276 +0x718
main.runCLI(0xc00001e100, 0x8, 0x8)
	/Users/rogerwelin/programming/cassowary/cli.go:198 +0x946
main.main()
	/Users/rogerwelin/programming/cassowary/main.go:8 +0x45

I'm using this binary on macOS: https://github.com/rogerwelin/cassowary/releases/tag/v0.1.1

Feature request: export raw data

Thanks for this nice tool!

Is it possible to add an option to export the raw per-request data somehow? The histogram and boxplot are nice, but I'd really like to be able to do my own analysis and generate my own charts. One use case is to generate nice charts in company theme colors, or to perform downstream statistical analysis/modeling, outlier detection, etc.

If exporting the raw data isn't possible, then at least it would be nice to get enough data to generate my own histogram with a separate program (e.g. Gnuplot), maybe just a JSON array of bin locations and counts at each bin.

Scale out using nats

Docs says to use k8.

I personally think that nats would be much easier and provide the system to be reactive .

Some things NATS would make easy are:

file config is automatically updated on all nodes , so that each node has it

jobs allocated in real time so that nodes are able to reactively get work to do .

Raw csv output also synced back from all nodes in real time so you can see histogram etc in real time , but also batched .

Anyone interested ?

0.14.0 checksum changed

Hello. I am one of the maintainers of Homebrew, a package manager for macOS and Linux.

We noticed that the checksum of the 0.14.0 release has changed from:
385232478b8552d56429fbe2584950bfbe42e3b611919a31075366a143aae9a9
to:
04fbf1e563378bc37a5ad44e69a3b100647406ea2debae2d38d3c406399628fa

Did someone re-tag the release (which is the insane thing to do, see the git manual, or did someone malicious gain access to your repo?

Tags that change are seen as a major security issue on our side, and we need confirmation that this was only a retag.

In future situations were you need to retag, please create a new tag/release instead.

See Homebrew/homebrew-core#95741 for the issue on our side.

Build failure on WSL2

In WSL2 Ubuntu using: brew update && brew install cassowary

==> Downloading https://github.com/rogerwelin/cassowary/archive/v0.14.0.tar.gz
==> Downloading from https://codeload.github.com/rogerwelin/cassowary/tar.gz/v0.14.0
# #=O#- #
==> go build -ldflags -s -w -X main.version=0.14.0 ./cmd/cassowary
Last 15 lines from /home/azilber/.cache/Homebrew/Logs/cassowary/01.go:
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1
go: downloading github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d
go: downloading github.com/russross/blackfriday/v2 v2.0.1
go: downloading github.com/shurcooL/sanitized_anchor_name v1.0.0
go: downloading golang.org/x/image v0.0.0-20210216034530-4410531fe030
go: downloading github.com/go-fonts/liberation v0.1.1
go: downloading github.com/phpdave11/gofpdf v1.4.2
go: downloading github.com/go-latex/latex v0.0.0-20210118124228-b3d85cf34e07
go: downloading github.com/fogleman/gg v1.3.0
go: downloading github.com/ajstarks/svgo v0.0.0-20180226025133-644b8db467af
go: downloading github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0
go: downloading golang.org/x/text v0.3.5
go: downloading github.com/jmespath/go-jmespath v0.3.0
# runtime/cgo
as: unrecognized option '--gdwarf-5'

non-positive interval for NewTicker

Cassowary panics when combining the following flags and values:

cassowary run -u http://localhost:5000 -c 2 -n 50000 -d 30

Add histogram visualization

Add a option to export the load test as a histogram visualization