vegasbrianc / docker-traefik-prometheus

A Docker Swarm Stack for monitoring Traefik with Promethues and Grafana

Monitor Traefik with Prometheus

This Repo helps you get started with monitoring Traefik v2.0the amazing Reverse Proxy + so much more. Along with Traefik we will provision Prometheus for Time Series Data collection and Grafana for Visualization. Traefik will also act as a proxy in front of Promethues and Grafana while Prometheus monitors Traefik the other way. Cool, huh?

Before we can begin ensure you have Docker installed with Docker Swarm enabled. If you are using Docker for Desktop Mac or Windows you already have Swarm enabled. For all others please follow the Swarm setup guide.

The presentation and Video from this demo is also included in the Repo - 56k_Cloud_Traefik_Monitoring.pdf and Youtube Session

Goals of the Traefik Monitoring Repo:

• Provision a Traefik Stack with Prometheus metrics enabled
• Deploy Prometheus & Grafana
• Verify Traefik Metrics
• Configure Dashboards in Grafana for Traefik

Review the Traefik Monitoring Stack Deployment

In this section we will prepare and deploy our Traefik Reverse Proxy and our monitoring stack.

First, we need to clone this Repo to your Docker Swarm. Ensure you are performing this on your Manager node. If you only have one node then this is also the manager.

git clone https://github.com/vegasbrianc/docker-traefik-prometheus.git

Next, lets review what the stack is doing before we deploy it. With your favorite editor (vim of course). Open the docker-compose.yml file.

cd docker-traefik-prometheus
vi docker-compose.yml

The Traefik metrics are enabled by the command we pass to the Traefik container. The --metrics flag enables metrics and --metrics.prometheus.buckets=0.1,0.3,1.2,5.0 defines the Prometheus bucket value (typically in seconds). Prometheus monitors for all values and stores the metric in the appropriate bucket.

version: '3.7'

services:
traefik:
    image: traefik:v2.0
    command:
    - "--logLevel=DEBUG"
    - "--api"
    - "--metrics"
    - "--metrics.prometheus.buckets=0.1,0.3,1.2,5.0"
    - "--docker"
    - "--docker.swarmMode"
    - "--docker.domain=docker.localhost"
    - "--docker.watch"

Grafana and Prometheus are being deployed by Docker Swarm and the networking is managed by Traefik. We use labels for the services deployed to inform Traefik how to setup the frontend and backend for each service.

Grafana Deployment

deploy:
 labels:
  - "traefik.port=3000"
  - "traefik.docker.network=inbound"
  -  "traefik.frontend.rule=Host:grafana.localhost"

Prometheus Deployment

deploy:
  labels:
   - "traefik.frontend.rule=Host:prometheus.localhost"
   - "traefik.port=9090"
   - "traefik.docker.network=inbound"

Prometheus is also configured to monitor Traefik. This is configured in Prometheus.yml which enables Prometheus to auto-discover Traefik inside of Docker Swarm. Prometheus is watching for the Service Task tasks.traefik on port 8080. Once the service is online metrics will begin flowing to Prometheus.

Deploy Traefik, Prometheus, and Grafana

OK, we now know where everything is configured inside of the stack. The moment of truth DEPLOY

In the docker-traefik-prometheusdirectory run the following:

docker stack deploy -c docker-compose.yml traefik

Verify all the services have been provisioned. The Replica count for each service should be 1/1 Note this can take a couple minutes

docker service ls

Check the Metrics

Once all the services are up we can open the Traefik Dashboard. The dashboard should show us our frontend and backends configured for both Grafana and Prometheus.

http://docker.localhost:8080/

Take a look at the metrics which Traefik is now producing in Prometheus metrics format

http://localhost:8080/metrics

Login to Grafana and Visualize Metrics

Grafana is an Open Source visualization tool for the metrics collected with Prometheus. Next, open Grafana to view the Traefik Dashboards. Note: Firefox doesn't properly work with the below URLS please use Chrome

http://grafana.localhost

Username: admin Password: foobar

Open the Traefik Dashboard and select the different backends available

Note: Upper right-hand corner of Grafana switch the default 1 hour time range down to 5 minutes. Refresh a couple times and you should see data start flowing

Deploy a new webservice

Of course we couldn't do a demo without showing some Cat Gifs. This demo launches a random cat picture of the day served by three instances of the Cats Services.

docker stack deploy -c cats.yml cats

Let's have a look at our new service

http://cats.localhost

Refresh a few times and notice the Traefik Proxy is loadbalancing our requests to the 3 different Cat services.

Now, head back to Grafana. Refresh the Traefik dashboard in the upper right corner and set 5 minutes for our time range. Select, the Cats backend in the Dashboard.

docker-traefik-prometheus's Issues

Do example only work on log level Debug?

Do example only work on log level Debug?

Adding Docker swarm dashboard

I think it would be interesting to add docker swarm data into the dashboard. this could help notice if we need to add more resources to swarm.

more detail request

Hello, thanks a lot.

Now, I am testing this https://github.com/vegasbrianc/prometheus this worked perfectly, but i can't monitor traefik metrics.

I have started traefik with the docker-compose.yml file, then import https://grafana.com/dashboards/4475 but i can't see any data on this dashboard.

I have few questions

• Should I modify prometheus/prometheus.yml file?
• which dashboard should I import from grafana dashboards for traefik?

Traefik 2.0 upgrade after load balancer not working

Hi,
I've completed the set up correctly, but the cats project is working with 3 replicas. When requesting Cats application, only 1 container returns response. Other containers bad gateway.

You can try it (10x times refresh page) you will see:
http://cats.taigalab.com/

Alerting and some configs

Hi! Thanks for the great repository.

Could you explain me few things please

• how can I configure the alert manager with slack for prometheus?
• how can I add basic auth for those services (prometheus, traefik, alert manager) ?
• is it possible to serve Traefik in the same way as the other services ? example - traefik.localhost
• Can we remove close the port 8080?

Thanks in advance

Support grafana dashboard for Traefik v2

Hi,
I see you have updated the Traefik version but Grafana dashboard is still not compatible with Traefik v2. Do you have the updated version?
Thanks.

Cloning Repository and starting Traefik return ingress network error

Hi, i have cloned your repo and start with command ❯ docker stack deploy -c docker-compose.yml traefik

I receive from docker:

Creating network traefik
Creating service traefik_traefik
failed to create service traefik_traefik: Error response from daemon: rpc error: code = FailedPrecondition desc = service needs ingress network, but no ingress network is present

Dashboard not initially showing proper traffic at first + N/A data

Hi, thanks a lot for the very nice write-up and documentation here (and dashboard here: https://grafana.com/grafana/dashboards/2870 ).

There are a few things that still mystify me perhaps, and I'm not sure what. First, the entrypoints confused me a fair bit since I could barely find documentation on it. Here's what my docker-compose ended up looking like:

  traefik:
    restart: always
    image: "traefik:v2.3"
    container_name: "traefik"
    command:
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      # enable 80, 443, 27017, 8082
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.websecure-mongodb.address=:27017"
      - "--entrypoints.metrics.address=:8082"
      # redirect 80 to 443
      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      # automatic certificate generation for SSL
      - "--certificatesresolvers.le.acme.tlschallenge=true"
      - "[email protected]"
      - "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json"
      - "--log.level=DEBUG"
      # get dashboard/api
      - "--api=true"
      - "--api.dashboard=true"
      # enable metrics with prometheus
      - "--metrics=true"
      - "--metrics.prometheus=true"
      - "--metrics.prometheus.buckets=0.100000, 0.300000, 1.200000, 5.000000"
      - "--metrics.prometheus.entrypoint=metrics"
      - "--metrics.prometheus.addEntryPointsLabels=true"
      - "--metrics.prometheus.addServicesLabels=true"

which I don't think is so bad. You can see the metrics primarily towards the end, and I picked 8082 (just so I could understand what's different from default 8080 that traefik uses). My promtheus service configuration looked just about the same (you can judge for yourself):

  prometheus:
    restart: unless-stopped
    image: prom/prometheus
    container_name: "prometheus"
    volumes:
      - ./config/prometheus/:/etc/prometheus/
      - prometheus-storage:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.prometheus.rule=(Host(`itkpix-srv.ucsc.edu`) && PathPrefix(`/prometheus`))"
      - "traefik.http.routers.prometheus.entrypoints=websecure"
      - "traefik.http.routers.prometheus.tls.certresolver=le"
      - "traefik.http.services.prometheus.loadbalancer.server.port=9090"
      - "traefik.http.middlewares.strip-prometheus.stripprefix.prefixes=/prometheus"
      - "traefik.http.middlewares.strip-prometheus.stripprefix.forceSlash=false"
      - "traefik.http.routers.prometheus.middlewares=strip-prometheus@docker"
    networks:
      - internal

so far, so good. My "traefik" is "web" here. However, when I reach the prometheus yaml file, I found that the whole thing about "listening" for a docker swarm (which I wasn't using) wasn't working, so I changed this up to use the static config example just like the prometheus job you already defined:

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    static_configs:
        - targets: ['localhost:9090']

  - job_name: 'traefik'
    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    static_configs:
        - targets: ['traefik:8082']

Once I made these changes, the dashboard you provided started having some amount of data in it! However, I get a bit confused about how the "total number of services" matches the service drop down:

and whether or not there's some delay between one of them updating and the other updating? I'm also wondering how long it takes (or how much data one needs to collect) before the other parts of the dashboard are "N/A"

Here's where the metrics is (itkpix-srv.ucsc.edu:8082/metrics). Let me know if I should hide this behind an ip filtering as well or not (it's not clear to me if this ever needed to be exposed to the outside world, or if it was enough to only expose it to prometheus via a dependency on the traefik service).

Again thanks for all your work on this!