Purpose of these repository is to aggregate prometheus-operator (using kube-prometheus jsonnet) with grafana stack with grafana Loki, grafana Tempo, grafana Agent and Grafana. Making it easy to demonstrate how grafana/prometheus observability works using examples like hotROD and TNS.
Then you can use this to create your own service monitors, prometheus rules and grafana dashboards.
Tested on: Docker for MAC v3.3.0 with Kubernetes enabled v1.19.7
We need an ingress and add new entries in /etc/hosts
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.45.0/deploy/static/provider/cloud/deploy.yaml
About ingress for Docker in Mac
Add it to /etc/hosts
127.0.0.1 grafana.example.local
127.0.0.1 alertmanager.example.local
127.0.0.1 prometheus.example.local
./apply.sh
If you want to check, please run kubectl
:
$ kubectl get pods --all-namespaces | egrep -v "kube-system|ingress"
NAMESPACE NAME READY STATUS RESTARTS AGE
default grafana-85bcb4f4cf-n5pqc 2/2 Running 0 8m14s
default grafana-agent-traces-57bgj 1/1 Running 0 8m17s
logging loki-0 1/1 Running 0 8m21s
logging promtail-9jtjf 1/1 Running 0 8m21s
monitoring alertmanager-main-0 2/2 Running 0 8m20s
monitoring kube-state-metrics-76f6cb7996-gdggq 3/3 Running 0 8m28s
monitoring node-exporter-42d6s 2/2 Running 0 8m27s
monitoring prometheus-adapter-59df95d9f5-rxcfc 1/1 Running 0 8m26s
monitoring prometheus-k8s-0 2/2 Running 0 8m19s
monitoring prometheus-operator-7775c66ccf-562ps 2/2 Running 0 8m35s
tracing tempo-0 2/2 Running 0 8m18s
Deploy it:
kubectl apply -f examples/hotrod-example.yaml
To remove:
kubectl delete -f examples/hotrod-example.yaml
Access hotrod application using port-forward:
kubectl port-forward -n default $(kubectl get pods -n default -l 'app.kubernetes.io/component=hotrod' |grep example |awk '{print $1}') 8080
Choose your ride!
Access grafana-local and using admin
and strongpassword
.
In Explore: {namespace="default",container="example-hotrod"}
and search for trace_id in one log line and click on it. Then click on "Tempo" button there.
Deploy TNS application
kubectl apply -f examples/tns-manifests.yaml
To remove:
kubectl delete -f examples/tns-manifests.yaml
- Go to the TNS Demo App Dashboard
- Zoom in on a section with failed requests if you are so inclined
- Panel Drop Down -> Explore
- Datasource Drop Down -> Loki
- Choose a log line with a traceID -> Tempo
- Go to Explore Tab Logs
- Choose Datasource Loki
- Run this query
{job="tns/app"} |json |line_format "{{.log}}" |logfmt | status>=500 and status <=599 and duration > 50ms
- Choose a log line with a traceID -> Tempo
Explore Tab Logs with line above
- Go to Explore Tab Metrics
- Choose Datasource thanos
- Run this query
histogram_quantile(.99, sum(rate(tns_request_duration_seconds_bucket{}[1m])) by (le))
- Click an exemplar
- Click the log icon on a span line
Explore Tab Metrics with histogram
./remove.sh
Required softwares:
- helm 3
- kubectl and kubectx
- jsonnet (go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb)
- gojsontoyaml (go get github.com/brancz/gojsontoyaml)
- docker-desktop with kubernetes enabled (can be minu-kube, not tested yet)
./update.sh
After upgrade all helm charts and jsonnet vendor directory, try to build it again.
./build.sh
./apply.sh
You should have these pods deployed:
$ kubectl get pods --all-namespaces | egrep -v "kube-system|ingress"
NAMESPACE NAME READY STATUS RESTARTS AGE
default grafana-85bcb4f4cf-n5pqc 2/2 Running 0 8m14s
default grafana-agent-traces-57bgj 1/1 Running 0 8m17s
logging loki-0 1/1 Running 0 8m21s
logging promtail-9jtjf 1/1 Running 0 8m21s
monitoring alertmanager-main-0 2/2 Running 0 8m20s
monitoring kube-state-metrics-76f6cb7996-gdggq 3/3 Running 0 8m28s
monitoring node-exporter-42d6s 2/2 Running 0 8m27s
monitoring prometheus-adapter-59df95d9f5-rxcfc 1/1 Running 0 8m26s
monitoring prometheus-k8s-0 2/2 Running 0 8m19s
monitoring prometheus-operator-7775c66ccf-562ps 2/2 Running 0 8m35s
tracing tempo-0 2/2 Running 0 8m18s
If everything is working fine, you can try to use some examples application to test it.
./remove.sh
https://github.com/grafana/tns
https://github.com/prometheus-operator/kube-prometheus
https://grafana.com/docs/loki/latest/logql/
https://github.com/grafana/agent
https://github.com/grafana/tempo
https://github.com/grafana/loki
https://github.com/grafana/helm-charts/tree/main/charts
https://github.com/jaegertracing/jaeger/tree/master/examples/hotrod