nitishm / engarde

Parse default envoy (and istio-proxy) access logs like a champ with engarde and jq

Engarde - Stay on guard with Envoy Access Logs

Parse default envoy access logs like a champ with engarde and jq

Motivation

Envoy access log messages are packed with a lot of useful information but in an unstructured log format. Without prior context, or even with context it can get cumbersome to visually inspect these log messages to extract useful information.

I was inspired by this tweet from Megan O'Keefe on twitter to create this tool for better readability of the envoy/istio-proxy access logs.

In addition, a special shout out to Richard Li from ambassador.io for this excellent article that provides more details on each of the subcomponents of the log message.

Installing

Homebrew

brew tap nitishm/homebrew-engarde
brew install engarde

Scoop

scoop bucket add engarde https://github.com/nitishm/scoop-bucket-engarde
scoop install engarde

Source

go get github.com/nitishm/engarde

This should install the compiled binary to your $GOBIN (or $GOPATH/bin).

Otherwise, clone the repository and build the binary manually using,

This package uses gomodules. Ensure that GO111MODULE=on is set if building outside $GOPATH in go version 1.11+

git clone https://github.com/nitishm/engarde.git
cd engarde/
go build -o engarde .
mv engarde /usr/local/bin/
export PATH=$PATH:/usr/local/bin/

Example (default format only)

Prerequisites

jq must be in your $PATH. If you do not have jq installed please download the binary here

Envoy

echo '[2016-04-15T20:17:00.310Z] "POST /api/v1/locations HTTP/2" 204 - 154 0 226 100 "10.0.35.28" "nsq2http" "cc21d9b0-cf5c-432b-8c7e-98aeb7988cd2" "locations" "tcp://10.0.2.1:80"' | engarde | jq

2019/09/03 22:31:33 Reading input from STDIN. Use the pipe "|" operator to redirect traffic to engarde
{
  "authority": "locations",
  "bytes_received": "154",
  "bytes_sent": "0",
  "duration": "226",
  "method": "POST",
  "protocol": "HTTP/2",
  "request_id": "cc21d9b0-cf5c-432b-8c7e-98aeb7988cd2",
  "response_flags": "-",
  "status_code": "204",
  "timestamp": "2016-04-15T20:17:00.310Z",
  "upstream_service": "tcp://10.0.2.1:80",
  "upstream_service_time": "100",
  "uri_path": "/api/v1/locations",
  "user_agent": "nsq2http",
  "original_message": "[2016-04-15T20:17:00.310Z] \"POST /api/v1/locations HTTP/2\" 204 - 154 0 226 100 \"10.0.35.28\" \"nsq2http\" \"cc21d9b0-cf5c-432b-8c7e-98aeb7988cd2\" \"locations\" \"tcp://10.0.2.1:80\""
}

Istio Proxy

kubectl logs -f foo-app-1 -c istio-proxy | engarde --use-istio | jq

{
  "authority": "hello-world",
  "bytes_received": "148",
  "bytes_sent": "171",
  "duration": "4",
  "method": "GET",
  "protocol": "HTTP/1.1",
  "request_id": "c0ce81db-4f5a-9134-8a5c-f8c076c91652",
  "response_flags": "-",
  "status_code": "200",
  "timestamp": "2019-09-03T05:37:41.341Z",
  "upstream_service": "192.168.89.50:9001",
  "upstream_service_time": "3",
  "upstream_cluster": "outbound|80||hello-world.default.svc.cluster.local",
  "upstream_local": "-",
  "downstream_local": "10.97.86.53:80",
  "downstream_remote": "192.168.167.113:39953",
  "uri_path": "/index",
  "user_agent": "-",
  "mixer_status": "-",
  "original_message": "[2019-09-03T05:37:41.341Z] \"GET /index HTTP/1.1\" 200 - \"-\" 148 171 4 3 \"-\" \"-\" \"c0ce81db-4f5a-9134-8a5c-f8c076c91652\" \"hello-world\" \"192.168.89.50:9001\" outbound|80||hello-world.default.svc.cluster.local - 10.97.86.53:80 192.168.167.113:39953 -"
}

Projects using Engarde

A UI powered by Engarde and Envoy docs for viewing envoy access logs on the browser : greghanson/engarde-viewer

engarde's Issues

`forwarded_for` field is not displayed

hyphen instead of underscore in parser.go:19

Add limit flag to truncate number of log lines parsed

Add limit flag to truncate number of log lines parsed

Add support for istio-proxy default log format

Istio uses a different default format as compared with the envoy default access log format

Add an option to use istio-proxy default format over envoy default format via command line flags.

"[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME%