davecheney / httpstat Goto Github PK
View Code? Open in Web Editor NEWIt's like curl -v, with colours.
License: MIT License
It's like curl -v, with colours.
License: MIT License
I would like to use httpstat to analyze sporadic HTTP timeouts I'm experiencing in my application when calling a backend HTTP API. The problem is that when httpstat experiences timeouts the output looks like this:
$ httpstat http://localhost:8000/delay/999999
Connected to 127.0.0.1:8000
2018/07/30 08:52:39 failed to read response: Get http://localhost:8000/delay/999999: EOF
What I would like to see as output is the trace including DNS Lookup, TCP Connection, etc. up to the point where the timeout occured. Is that something that could be implemented in httpstat? If yes, I would be willing to help here.
I've found that when we return an error on file creation we are actually using the wrong variable (outputFile
). We should be using filename
instead:
Line 421 in fb43abf
I'm getting the following error trying to install this package
go get -u github.com/davecheney/httpstat
package github.com/davecheney/httpstat/vendor/github.com/fatih/color: /Users/wayne/go/src/github.com/davecheney/httpstat/vendor/github.com/fatih/color/.git exists but is not a directory
package github.com/davecheney/httpstat/vendor/golang.org/x/net/http2: /Users/wayne/go/src/github.com/davecheney/httpstat/vendor/golang.org/x/net/.git exists but is not a directory
Tried cleaning my $GOPATH and didn't make a difference.
sudo rm -r $GOPATH
mkdir $GOPATH
go version go1.8.1 darwin/amd64
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/wayne/go"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.8.1/libexec"
GOTOOLDIR="/usr/local/Cellar/go/1.8.1/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/8s/j9vt85xs4ps80919tlwsxdk00000gn/T/go-build266113172=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
GOROOT/bin/go version: go version go1.8.1 darwin/amd64
GOROOT/bin/go tool compile -V: compile version go1.8.1 X:framepointer
uname -v: Darwin Kernel Version 16.5.0: Fri Mar 3 16:52:33 PST 2017; root:xnu-3789.51.2~3/RELEASE_X86_64
ProductName: Mac OS X
ProductVersion: 10.12.4
BuildVersion: 16E195
lldb --version: lldb-370.0.42
Swift-3.1
Request send to remote host from following command:
httpstat -H 'Host: example.com' http:/example2.com
Contains:
Host: example2.com
header but not:
Host: example.com
I many times work with curl and pipes eg curl -v url | jq .
. Why not write the current output to stderr and the body to stdout?
I'm testing httpstat on a Windows machine and I get a huge TCP Connection time.
D:\>httpstat http://139.110.74.200/dsrv/status
Connected to 139.110.74.200:80
HTTP/1.1 200 OK
Server: Apache/2.4.6 (Red Hat Enterprise Linux) PHP/5.4.16
Content-Type: text/xml
Date: Thu, 11 Aug 2016 09:36:27 GMT
X-Powered-By: PHP/5.4.16
Body discarded
DNS Lookup TCP Connection Server Processing Content Transfer
[ 0ms | 9223372036854ms | 9ms | 1ms ]
| | | |
namelookup:0ms | | |
connect:9223372036854ms | |
starttransfer:9223372036854ms |
total:922337203
6854ms
I have tried changing the HTTP_PROXY
variable and when using a local cntlm
proxy it goes back to something reasonable:
D:\>set HTTP_PROXY=localhost:3128
D:\>set HTTPS_PROXY=localhost:3128
D:\>httpstat http://139.110.74.200/dsrv/status
Connected to 127.0.0.1:3128
HTTP/1.1 200 OK
Server: Apache/2.4.6 (Red Hat Enterprise Linux) PHP/5.4.16
Accept-Ranges: none
Content-Length: 415
Content-Type: text/xml
Date: Thu, 11 Aug 2016 09:43:50 GMT
Proxy-Connection: close
Via: 1.1 gproxy.postal.local:8080
X-Powered-By: PHP/5.4.16
Body discarded
DNS Lookup TCP Connection Server Processing Content Transfer
[ 0ms | 0ms | 29ms | 0ms ]
| | | |
namelookup:0ms | | |
connect:0ms | |
starttransfer:29ms |
total:29ms
I can provide additional information if needed.
(Thanks for this tool, it is awesome !)
I am trying to build httpstat with go get -v -u github.com/davecheney/httpstat
and it is failing. I am using go version go1.9 linux/amd64
:
$ go get -v -u github.com/davecheney/httpstat
github.com/davecheney/httpstat (download)
github.com/davecheney/httpstat/vendor/github.com/fatih/color (download)
package github.com/davecheney/httpstat/vendor/github.com/fatih/color: /home/aitva/go/src/github.com/davecheney/httpstat/vendor/github.com/fatih/color/.git exists but is not a directory
github.com/davecheney/httpstat/vendor/golang.org/x/net (download)
package github.com/davecheney/httpstat/vendor/golang.org/x/net/http2: /home/aitva/go/src/github.com/davecheney/httpstat/vendor/golang.org/x/net/.git exists but is not a directory
On the upside, if I run go install github.com/davecheney/httpstat
the build works properly! The vendored folders are cloned and go get
is right about .git
being a file:
~/go/src/github.com/davecheney/httpstat/vendor/github.com/fatih/color$ ls -la
total 72
drwxrwxr-x 3 aitva aitva 4096 sept. 20 11:00 .
drwxrwxr-x 3 aitva aitva 4096 sept. 20 11:00 ..
-rw-rw-r-- 1 aitva aitva 18220 sept. 20 11:00 color.go
-rw-rw-r-- 1 aitva aitva 8491 sept. 20 11:00 color_test.go
-rw-rw-r-- 1 aitva aitva 4397 sept. 20 11:00 doc.go
-rw-rw-r-- 1 aitva aitva 63 sept. 20 11:00 .git
-rw-rw-r-- 1 aitva aitva 1079 sept. 20 11:00 LICENSE.md
-rw-rw-r-- 1 aitva aitva 4691 sept. 20 11:00 README.md
-rw-rw-r-- 1 aitva aitva 35 sept. 20 11:00 .travis.yml
drwxrwxr-x 4 aitva aitva 4096 sept. 20 11:00 vendor
By coincidence I read through this code, and shortly after that read Idiomatic Go. There was one practice highlighted there that I had just seen in this code.
In flag.Usage os.Exit(2) is called, which flag.Parse already does. It would be nicer to do this within the if len(args) != 1
instead of within func usage()
.
But some values of fields may not be a real for latency if httpstat support proxy.
httpstat http://dave.cheney.net/about
works
httpstat dave.cheney.net/about
does not.
2016/09/24 13:54:20 unsupported url scheme ""
To make httpstat
easier to use, if no scheme is provided, stick a https://
on the front. Yes, https://
, if people want to be insecure, they can be explicit.
main.go:357: undefined: http.MethodHead
$ go version
go version go1.3.1 linux/amd64
It would be really good to have the SSL/TLS handshake latency.
would be interesting to have realtime sharing of stats to have more data points when troubleshooting a site. e.g. how did this test compare with others from your AS over the past day, or compared to other ASs, historical error rates. sort of how speedtest.net allows you to see how your download speeds compare to others. would be nice if there was an existing free service that it could just integrate like downdetector.com or netcraft.com, but they don't have free APIs.
First, this is awesome, thanks for making a Go version of this!
But it doesn't like when I need to specify a port:
httpstat "http://45.56.15.67:8680/"
2016/09/23 09:03:11 unable to resolve host: too many colons in address 45.56.15.67:8680:80
(Believe me, I wish this was just on port 80 like every other website. But I'm having connectivity issues to it which is why I thought I'd try httpstat, ha.)
When I try to use httpstat on service with HTTP/2 I got following error:
http2: server sent GOAWAY and closed the connection; LastStreamID=0, ErrCode=NO_ERROR, debug=""
I've made some quick changes to print the results in CSV format.
I need this to investigate an issue with increasing response time and I want to know which step causes this issue.
With these changes it is possible to run httpstat in a loop and write the output to a CSV file which can than be used for analyzing.
Example
$ for i in {1..20}; do httpstat -csv https://google.com >> out.csv && sleep 2; done;
This will print lines that look like this:
2,29,224,64,0,2,32,257,321,321
The columns are in the following order (HTTPS in this case):
I can prepare a pull request if you think this feature is useful.
The current solution is a bit quick and dirty, so I have to implement it in a proper way before.
% httpstat -O http://example.com/
Connected to 93.184.216.34:80
2016/09/25 21:01:06 unable to create file
If the URI ends in a /
we should either refuse to create the file, or substitute something else.
We should be able to revert 63dfb1c. A fix has been merged upstream.
See: fatih/color@0016e26
Maybe this isn't supposed to work. Just read the source.
In function schemify the strings.HasSuffix(uri, ":80")
does not work for host:80/path
.
I use tls client certificates for authentication to several web services I develop/debug. I am wondering if it would be helpful to add client certificate support to httpstat. This feature would be easy to implement. I would suggest we load a client certificate keypair and signing ca certificate from files that would be specified as command line flags into the http client's tls.Config. I have this implemented if this feature is of interest. Thanks!
With #7 merged only the final leg of a 30x redirect chain is reported. This is a regression.
zapf(~/src/github.com/davecheney/httpstat) % httpstat -L https://google.com/
Connected to 216.58.199.78:443
Connected to 150.101.161.207:443
HTTP/1.1 200 OK
Server: gws
Alt-Svc: quic=":443"; ma=2592000; v="36,35,34,33,32"
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
Date: Wed, 28 Sep 2016 01:59:28 GMT
Expires: -1
P3p: CP="This is not a P3P policy! See https://www.google.com/support/accounts/answer/151657?hl=en for more info."
Set-Cookie: NID=87=AW23QT-3Jr7sBhWZfUORexa_5nsDTXe9jp2iNKYKemqJlG7tM_LdEvqz1UZtUZ57QYm4PaoHrAlGpT_-S9KOC0Kg-xaozwQrdK3AoLwB7XWRQ6hrIrjndWgoQUiF695C; expires=Thu, 30-Mar-2017 01:59:28 GMT; path=/; domain=.google.com.au; HttpOnly
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block
Body discarded
DNS Lookup TCP Connection TLS Handshake Server Processing Content Transfer
[ 23ms | 21ms | 61ms | 201ms | 3ms ]
| | | | |
namelookup:23ms | | | |
connect:44ms | | |
pretransfer:106ms | |
starttransfer:307ms |
total:310ms
httpstat master $ curl -I https://www.google.com
HTTP/2.0 302
cache-control:private
content-type:text/html; charset=UTF-8
location:https://www.google.be/?gfe_rd=cr&ei=ycnqV5PxMKGk8wfZpZmoBg
content-length:259
date:Tue, 27 Sep 2016 19:34:33 GMT
alt-svc:quic=":443"; ma=2592000; v="36,35,34,33,32"
httpstat master $ ./httpstat https://www.google.com
Connected to 216.58.212.228:443
HTTP/1.1 302 Found
Alt-Svc: quic=":443"; ma=2592000; v="36,35,34,33,32"
Cache-Control: private
Content-Length: 259
Content-Type: text/html; charset=UTF-8
Date: Tue, 27 Sep 2016 19:34:39 GMT
Location: https://www.google.be/?gfe_rd=cr&ei=z8nqV9yNHqrH8AeTm6qQCQ
DNS Lookup TCP Connection TLS Handshake Server Processing Content Transfer
[ 5ms | 31ms | 93ms | 32ms | 0ms ]
| | | | |
namelookup:5ms | | | |
connect:36ms | | |
pretransfer:130ms | |
starttransfer:162ms |
total:162ms
It would be nice if the client allowed multiple requests with the same ssl session id so it would be possible to measure the impact of ssl tickets / ssl resume.
Hi,
It would be nice to have a -4 / -6 switch to force DNS resolution to ipv4 or ipv6.
Like this, we could compare response time between both protocol.
Regards,
See TODO in main.readResponseBody
Introduce the -max-directs flag from curl to limit the maximum number for 30x redirects to follow and set a sane default. cURL sets this to 50.
Feel free to close this issue right away. I was just thinking whether the Projects feature wouldn't be more suitable for keeping track of the TODO list (more than the README). I don't have experience with it though.
Needs to support multiple -H name=value
fields.
When I use installation command, I failed.
$ go get -u github.com/davecheney/httpstat [10:56:20]
# github.com/davecheney/httpstat
failed MSpanList_Insert 0x7925c0 0xeb13f9b6bae5 0x0 0x0
fatal error: MSpanList_Insert
runtime stack:
runtime.throw(0x2a3330, 0x10)
/usr/local/Cellar/go/1.6/libexec/src/runtime/panic.go:530 +0x90
runtime.(*mSpanList).insert(0x3a68a8, 0x7925c0)
/usr/local/Cellar/go/1.6/libexec/src/runtime/mheap.go:933 +0x293
runtime.(*mheap).freeSpanLocked(0x3a60a0, 0x7925c0, 0x100, 0x0)
/usr/local/Cellar/go/1.6/libexec/src/runtime/mheap.go:809 +0x4be
runtime.(*mheap).grow(0x3a60a0, 0x30, 0x0)
/usr/local/Cellar/go/1.6/libexec/src/runtime/mheap.go:675 +0x2a0
runtime.(*mheap).allocSpanLocked(0x3a60a0, 0x30, 0x4c2fb20)
/usr/local/Cellar/go/1.6/libexec/src/runtime/mheap.go:553 +0x4e3
runtime.(*mheap).alloc_m(0x3a60a0, 0x30, 0x100000000, 0x7924d0)
/usr/local/Cellar/go/1.6/libexec/src/runtime/mheap.go:437 +0x119
runtime.(*mheap).alloc.func1()
/usr/local/Cellar/go/1.6/libexec/src/runtime/mheap.go:502 +0x41
runtime.systemstack(0x7fff5fbfe910)
/usr/local/Cellar/go/1.6/libexec/src/runtime/asm_amd64.s:307 +0xab
runtime.(*mheap).alloc(0x3a60a0, 0x30, 0x100000000, 0xc820022000)
/usr/local/Cellar/go/1.6/libexec/src/runtime/mheap.go:503 +0x63
runtime.largeAlloc(0x60000, 0xc800000003, 0x4d2d0)
/usr/local/Cellar/go/1.6/libexec/src/runtime/malloc.go:766 +0xb3
runtime.mallocgc.func3()
/usr/local/Cellar/go/1.6/libexec/src/runtime/malloc.go:664 +0x33
runtime.systemstack(0x39f700)
/usr/local/Cellar/go/1.6/libexec/src/runtime/asm_amd64.s:291 +0x79
runtime.mstart()
/usr/local/Cellar/go/1.6/libexec/src/runtime/proc.go:1048
goroutine 1 [running]:
runtime.systemstack_switch()
/usr/local/Cellar/go/1.6/libexec/src/runtime/asm_amd64.s:245 fp=0xc824a3b508 sp=0xc824a3b500
runtime.mallocgc(0x60000, 0x0, 0xc800000003, 0x8)
/usr/local/Cellar/go/1.6/libexec/src/runtime/malloc.go:665 +0x9eb fp=0xc824a3b5e0 sp=0xc824a3b508
runtime.rawmem(0x60000, 0x2f)
/usr/local/Cellar/go/1.6/libexec/src/runtime/malloc.go:809 +0x32 fp=0xc824a3b608 sp=0xc824a3b5e0
runtime.growslice(0x1e80a0, 0xc824ccc000, 0x4c000, 0x4c000, 0x4c001, 0x0, 0x0, 0x0)
/usr/local/Cellar/go/1.6/libexec/src/runtime/slice.go:95 +0x233 fp=0xc824a3b678 sp=0xc824a3b608
cmd/link/internal/ld.Symgrow(0xc820082240, 0xc824a10fc0, 0x4c002)
/usr/local/Cellar/go/1.6/libexec/src/cmd/link/internal/ld/data.go:52 +0x26e fp=0xc824a3b730 sp=0xc824a3b678
cmd/link/internal/ld.addpctab(0xc824a10fc0, 0xc80004bfbc, 0xc820ecc030, 0x4bfbc)
/usr/local/Cellar/go/1.6/libexec/src/cmd/link/internal/ld/pcln.go:130 +0x6a fp=0xc824a3b7b0 sp=0xc824a3b730
cmd/link/internal/ld.pclntab()
/usr/local/Cellar/go/1.6/libexec/src/cmd/link/internal/ld/pcln.go:323 +0x8d1 fp=0xc824a3b9e0 sp=0xc824a3b7b0
cmd/link/internal/ld.Ldmain()
/usr/local/Cellar/go/1.6/libexec/src/cmd/link/internal/ld/pobj.go:241 +0x1f33 fp=0xc824a3be60 sp=0xc824a3b9e0
cmd/link/internal/amd64.Main()
/usr/local/Cellar/go/1.6/libexec/src/cmd/link/internal/amd64/obj.go:44 +0x19 fp=0xc824a3be68 sp=0xc824a3be60
main.main()
/usr/local/Cellar/go/1.6/libexec/src/cmd/link/main.go:27 +0x36f fp=0xc824a3bf40 sp=0xc824a3be68
runtime.main()
/usr/local/Cellar/go/1.6/libexec/src/runtime/proc.go:188 +0x2b0 fp=0xc824a3bf90 sp=0xc824a3bf40
runtime.goexit()
/usr/local/Cellar/go/1.6/libexec/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc824a3bf98 sp=0xc824a3bf90
I install go
using homebrew
$ go version [22:45:15]
go version go1.6 darwin/amd64
A PPA for debian / ubuntu would ease installation for linux users, and provide a way for linux users to stay up to date with the latest version easily.
My applications often have numerous IPs (CDNs, etc), so showing which IP is being connected to is helpful.
(Along the same lines, an option to connect to a particular IP could also be helpful; along the lines of the -x or --resolv options from curl).
Currently the https client chooses to allow invalid certs (I think). If this is the case, then this should be reversed and a flag added to enable this insecure behaviour.
Note to implementors: at the moment we're using to go flag library so I would prefer this flag to be a single character and that character be the same as one traditionally used by another tool like curl or wget.
When connecting to an IP address, the DNSDone trace hooks aren't fired.
curl -L
follows redirects.
Implementing this in httpstat should follow 30x redirects and print the timing output for each step in the redirection chain.
This will probably require refactoring the body of main.main
into functions.
Note: for implementors, please discuss your design here before slinging code.
From #16
I thought about this previously and have two suggestions.
Host:
firstI know this is more work, but I think this is only worth doing if it's done to this degree of precision.
Tested against a server that redirects, httpstat -O
wrote one file for each request using that request's URL to determine the output filename.
Possibly don't want to write multiple files if server redirects.
Possibly don't want filename to be controlled by response.
https://github.com/davecheney/httpstat/blob/master/main.go#L134 should set ServerName
in tls.Config
.
This would make it possible to use this tool with hosting situations where multiple sites share the same IP and there is no fallback.
Hi,
curl(1)
let you post data from files using -d @filename
. I use it all the time missed this feature when working with httpstat
.
If you think this is a relevant addition I can make a pull request.
The changes I made for this to work are here: https://github.com/lucindo/httpstat/commit/87c4d32f548248dcbe11d3501b4ed706656474fe
There are a bunch of small issues that vet and lint will pick up on related to printing and formatting. It would be good to have those fixed.
extra credit: add a go tool vet pass to .travis.yml so we don't have to reopen this issue once the current formatting errors are fixed.
note to implementors: I'm not interested in PR's renaming variables to whatever vet thinks is a standard, that is beyond my tolerance for bike shedding.
Notice this in my testing.
httpstat http://google.com
Connected to 172.217.3.206:80
HTTP/1.1 301 Moved Permanently
Server: gws
Cache-Control: public, max-age=2592000
Content-Length: 219
Content-Type: text/html; charset=UTF-8
Date: Mon, 26 Sep 2016 22:30:39 GMT
Expires: Wed, 26 Oct 2016 22:30:39 GMT
Location: http://www.google.com/
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block
DNS Lookup TCP Connection Server Processing Content Transfer
[ %s | %s | %s | %s ]
| | | |
namelookup:%s | | |
connect:%s | |
starttransfer:%s |
total:%s
110ms 23ms 34ms 0ms110ms 133ms 168ms 168ms %
If you query with https then the output looks right.
Connected to 172.217.3.206:443
HTTP/1.1 301 Moved Permanently
Server: gws
Alt-Svc: quic=":443"; ma=2592000; v="36,35,34,33,32"
Cache-Control: public, max-age=2592000
Content-Length: 220
Content-Type: text/html; charset=UTF-8
Date: Mon, 26 Sep 2016 22:31:06 GMT
Expires: Wed, 26 Oct 2016 22:31:06 GMT
Location: https://www.google.com/
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block
DNS Lookup TCP Connection TLS Handshake Server Processing Content Transfer
[ 0ms | 23ms | 137ms | 36ms | 0ms ]
| | | | |
namelookup:0ms | | | |
connect:23ms | | |
pretransfer:161ms | |
starttransfer:198ms |
total:198ms
size_download (The total amount of bytes that were downloaded.)
size_upload (The total amount of bytes that were uploaded.)
speed_download (The average download speed that curl measured for the complete download. Bytes per second.)
Can someone upload a binary for people that do not have/use Go?
I think, one should move all headers
type logic to separate file for convenience.
Currently the formatting of coloured output is handled before passing the string to fmt.Printf
. As an exercise I would like to see if it is possible to make the type of the argument passed to fmt.Printf
implement fmt.Formatter
and handle the colorisation of the output at the point at which it is printed.
note to implementors: this has no impact on the performance or memory usage of this program, in fact it'll probably make it infinitesimally slower and larger. I'm mainly interested in exploring this to improve the separation of concerns between the code that prepares the output, and the code that colourises it.
Hi! How about a flag, that will show only header of response like curl --head
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.