povilasv / coredns-mixin

A set of Prometheus alerts & grafana dashboards for CoreDNS.

Prometheus Monitoring Mixin for CoreDNS

A set of Grafana dashboards & Prometheus alerts for CoreDNS.

How to use

This mixin is designed to be vendored into the repo with your infrastructure config. To do this, use jsonnet-bundler:

Generate config files

You can manually generate the alerts, dashboards and rules files, but first you must install some tools:

$ go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb
$ brew install jsonnet

Then, grab the mixin and its dependencies:

$ git clone https://github.com/povilasv/coredns-mixin
$ cd coredns-mixin
$ jb install

Finally, build the mixin:

$ make prometheus_alerts.yaml
$ make dashboards_out

The prometheus_alerts.yaml file then need to passed to your Prometheus server, and the files in dashboards_out need to be imported into you Grafana server. The exact details will depending on how you deploy your monitoring stack.

Background

• For more information about monitoring mixins, see this design doc.
• CoreDNS Prometheus metrics plugin docs

coredns-mixin's Issues

Does not work without providing entire grafana configuration in _config

Hi, I am trying to get this to work with kube-prometheus customization but am unable to do so. To narrow the issue, I started with a fresh workspace and then modified their all-namespaces.jsonnet to include coredns mixin as follows:

local kp = (import 'kube-prometheus/kube-prometheus.libsonnet') +
           (import 'kube-prometheus/kube-prometheus-all-namespaces.libsonnet') +
           (import 'coredns-mixin/mixin.libsonnet') +{
  _config+:: {
    namespace: 'monitoring',

    prometheus+:: {
      namespaces: [],
    },
  },
};

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

I get the below error:

• jsonnet -J vendor -m manifests example1.jsonnet

• xargs '-I{}' sh -c 'cat {} | gojsontoyaml > {}.yaml' -- '{}'
  RUNTIME ERROR: Field does not exist: config
  vendor/grafana/grafana.libsonnet:47:20-44 thunk from <object >
  vendor/grafana/grafana.libsonnet:47:9-45 object
  example1.jsonnet:19:70-80 thunk from <$>
  :1293:24-25 thunk from <function >
  :1293:5-33 function
  example1.jsonnet:19:53-81 $

    During evaluation
  

I tried by adding config for Grafana as follows in the _config section

grafana+:: {
      config+: {
        sections+: {
          server+: {
            root_url: 'https://grafana.10-0-2-15.nip.io/',
          },
        },
        ldap: null,
      },
    },

This results in the below error:

• jsonnet -J vendor -m manifests example1.jsonnet
• xargs '-I{}' sh -c 'cat {} | gojsontoyaml > {}.yaml' -- '{}'
  RUNTIME ERROR: Field does not exist: ldap
  vendor/grafana/grafana.libsonnet:50:32-54 thunk from <object >
  vendor/grafana/grafana.libsonnet:51:36-49 thunk from <object >
  vendor/ksonnet/ksonnet.beta.4/k8s.libsonnet:9656:82-86
  vendor/ksonnet/ksonnet.beta.4/k8s.libsonnet:9658:41-45 object
  During manifestation

I ended up providing entire grafana configuration. Below is my final _config+::

 _config+:: {
    namespace: 'monitoring',
    grafana+:: {
      config+: {
        sections+: {
          server+: {
            root_url: 'https://grafana.10-0-2-15.nip.io/',
          },
        },
      },
      ldap: null,
      datasources: [{
      }],
      dashboards: {},
      folderDashboards: {},
      rawDashboards: {},
      container: {
        requests: { cpu: '100m', memory: '100Mi' },
        limits: { cpu: '200m', memory: '200Mi' },
      },
      containers: [],
      env: [],
      plugins: [],
      port: 3000,
    },

    prometheus+:: {
      namespaces: [],
    },
  },

Runbook is missing

All alerts has links to a runbook file that does not exists (e.g. https://github.com/povilasv/coredns-mixin/tree/master/runbook.md#alert-name-corednsforwarderrorshigh).

PrometheRule name is not compatble with kube-prometheus

Hi,

I've added your coredns-mixin to kube-prometheus in the JSONNET file, it builds correctly but the PrometheusRule cannot be applied to the Kubernetes cluster as is contains upper case characters in the name. When trying to apply the manifest the following error is raised:

The PrometheusRule "coreDNS" is invalid: metadata.name: Invalid value: "coreDNS": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')

Changing the start of the manifest as follows resolves the problem:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    prometheus: k8s
  name: coredns
  namespace: monitoring

multicluster variable is missing

Given the fact, all prometheus dashboards support multi cluster, can we include multicluster back here, please ?

add-runbook-links.libsonnet breaks runbook_url from other alerts

Hello!
I am trying to use this lib with kube-prometheus and when I import alerts like

local coreDNSAlerts = (import 'coredns-mixin/mixin.libsonnet').prometheusAlerts;
prometheusAlerts+:: jaegerAlerts + coreDNSAlerts

runbook_urls in all other alerts got rewritten

-        runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-nodefilesystemalmostoutoffiles
+        runbook_url: https://github.com/povilasv/coredns-mixin/tree/master/runbook.md#alert-name-nodefilesystemalmostoutoffiles