Coder Social home page Coder Social logo

grafana-dashboard-builder's Introduction

grafana-dashboard-builder

PyPI version Build Status Coverage Status

Introduction

grafana-dashboard-builder is an open-source tool for easier creation of Grafana dashboards. It is written in Python and uses YAML descriptors for dashboard templates.

This project has been inspired by Jenkins Job Builder that allows users to describe Jenkins jobs with human-readable format. grafana-dashboard-builder aims to provide similar simplicity to Grafana dashboard creation and to give users easy way how they can create dashboard templates filled with different configuration.

Installation

To install:

pip install grafana-dashboard-builder

or

python setup.py install

An alternative to installing it into the global site-packages is to add it to your user local directory (usually ~/.local).

pip install --user grafana-dashboard-builder

Usage

After installation you'll find grafana-dashboard-builder on your path. Help can be printed by --help command-line option.

usage: grafana-dashboard-builder [-h] -p PATH [PATH ...] [--project PROJECT] [-o OUT] [-c CONFIG]
                                 [--context CONTEXT] [--plugins PLUGINS [PLUGINS ...]]
                                 [--exporter EXPORTERS [EXPORTERS ...]]

optional arguments:
  -h, --help            show this help message and exit
  -p PATH [PATH ...], --path PATH [PATH ...]
                        List of path to YAML definition files
  --project PROJECT     (deprecated, use path) Location of the file containing
                        project definition.
  -o OUT, --out OUT     (deprecated, use config file and file exporter) Path
                        to output folder
  -c CONFIG, --config CONFIG
                        Configuration file containing fine-tuned setup of
                        builder's components.
  --context CONTEXT     YAML structure defining parameters for dashboard
                        definition. Effectively overrides any parameter
                        defined on project level.
  --plugins PLUGINS [PLUGINS ...]
                        List of external component plugins to load
  --exporter EXPORTERS [EXPORTERS ...]
                        List of dashboard exporters

To start you need to create project configuration that needs to be in one YAML document. And some examples with current can be found in sample project:

grafana-dashboard-builder -p ./samples/project.yaml --exporter file --config ./samples/config.yaml

Exporters

grafana-dashboard-builder provides several builtin exporters that can be enabled through --exporter option. Configuration for all of them is to be provided in configuration file given in --config option. Look at sample config.

File exporter

File exporter is used when you want to store dashboards as JSON files on your local disk.

file:
  output_folder: /some/directory/on/my/disk

To use file exporter run grafana-dashboard-builder with --exporter file option.

Grafana Elastic Search

grafana-dashboard-builder currently supports persisting dashboards to Elastic Search used by Grafana prior to version 2.0

To configure Elastic Search endpoint put following structure to your configuration file:

elastic-search:
  host: https://this-is-my-domain.com
  password: my_password
  username: my_username

With this configuration your dashboard will be uploaded to https://this-is-my-domain.com/es/grafana-dash/dashboard/dashboard_name

If you do not want to store your credentials in the configuration file you can use environment variables ES_PASSWORD and ES_USERNAME.

To use elastic search exporter run grafana-dashboard-builder with --exporter elastic-search option.

Grafana API

grafana-dashboard-builder currently supports Grafana version 2.0 API.

To configure Grafana endpoint put following structure to your configuration file:

grafana:
  host: https://this-is-my-domain.com
  password: my_password
  username: my_username

With this configuration your dashboard will be POSTed to https://this-is-my-domain.com/api/dashboards/db

If you do not want to store your credentials in the configuration file you can use environment variables GRAFANA_PASSWORD and GRAFANA_USERNAME.

You can use Organization API Key to access Grafana API:

Set token in your configuration file:

grafana:
  host: https://this-is-my-domain.com
  token: eyJrIjoiOGNTW...o2b2123kO==

Or in GRAFANA_TOKEN environment variable.

Read more about authentication in Grafana docs.

To use Grafana exporter run grafana-dashboard-builder with --exporter grafana option.

Supported data stores

At this moment grafana-dashboard-builder supports following data stores:

YAML definition format

Each component follows the same configuration format. Top level must contain 2 fields - name and component type. Under component type is wrapped definition of the component.

- name: some-name
  component-type:
    component-param1: param-value
    component-param2: other-value

Components can be defined in multiple source files that are passed through --path option. If a path is directory it is recursively walked and all files are processed.

Components

Components define basic building blocks such as rows, graphs and template queries. They can be defined in-place or be named and reused within other components and dashboards.

Components can define parameters that can be passed from parent component to its children.

- name: graph-name
  panels:
    - graph:
        target: target
        y_formats: [bytes, short]
        span: 4
- name: row-name
  rows:
    - row:
        title: Placeholder row
        panels:
            - graph-name
            - graph:
                target: target
                y_formats: [bytes, short]
                span: 4

Another component is template queries that allow you to define just one query string for hierarchical variables. Each query part that starts with $ sign will appear as one variable.

- name: template-name
  templates:
    - query:
        query: '{metric-prefix}.$component.$application'

Dashboard

Dashboard is top-level object composed of several components.

- name: overview
  dashboard:
    title: overview dashboard
    time_options: [1h]
    refresh_intervals: [5m]
    templates:
      - template-name:
            metric-prefix: '{metric-prefix}'
    time:
      from: now-12h
      to: now
    rows:
      - row-name

Project

Project is an entry point for builder and defines which dashboards will be generated and provides parameters to them.

- name: Example project
  project:
    dashboard-prefix: MyApp
    metric-prefix: metric.prefix
    dashboards:
        - overview

The biggest benefit of grafana-dashboard-builder is that you can generate several dashboards from one dashboard template just by defining multiple values for a parameter that is contained in dashboard name. Following project will generate 2 dashboards named prefix1-dashboard and prefix2-dashboard.

- name: Example project
  project:
    dashboard-prefix:
      - prefix1
      - prefix2
    dashboards:
      - '{dashboard-prefix}-dashboard'

External context definition

Thanks to project component you can use one dashboard template and configure it with different parameters. But what if you need to use different params based on the Grafana you are uploading dashboards to. That's why you can define configuration externally to your projects and dashboard templates.

You can reference configuration stored in YAML with -config option or even inline it to --context option. External configuration file can look like:

context:
  region: eu
  default-datacenter: cze

grafana-dashboard-builder's People

Contributors

berkgaut avatar danzika avatar dependabot-preview[bot] avatar gagallo7 avatar jaceksan avatar jakub-plichta avatar jakubplichta avatar lgtm-migrator avatar manueligno78 avatar ostumpf avatar pcerny avatar pracharm avatar t-8ch avatar trampfox avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

grafana-dashboard-builder's Issues

Placeholder {name} is not resolved in text/content

Below you can find example code.
In panel text, attribute content, if value contains variable defined in --config file, it is not resolved.

  • config file rack-na1.yaml (--config)
context:
  elastic-host: "myhost.com"
  • dashboard file vertica.yaml
- name: notice-documentation
  rows:
    - row:
        title: Documentation
        panels:
          - text:
              mode: markdown
              content: "Link to [Main Dashboard](https://{elastic-host}/xxxxx)"

Graph enhancements

Nice to have features for graphs:

  • y-axis label
  • super-nice - line colors

Component '['default-dashboard']' does not have 'name' attribute, skipping

Using the defaults, I'm getting the following error

$ grafana-dashboard-builder -p . --exporter grafana
2017-04-04 16:03:53,111 DEBUG Creating new connection with username=admin host=http://se-k8s-master.vivintsky.com/grafana/
2017-04-04 16:03:53,113 INFO Loading component type <class 'grafana_dashboards.components.projects.Project'>
2017-04-04 16:03:53,113 INFO Loading component type <class 'grafana_dashboards.components.links.LinksItemBase'>
2017-04-04 16:03:53,113 INFO Loading component type <class 'grafana_dashboards.components.panels.PanelsItemBase'>
2017-04-04 16:03:53,113 INFO Loading component type <class 'grafana_dashboards.components.rows.RowsItemBase'>
2017-04-04 16:03:53,113 INFO Loading component type <class 'grafana_dashboards.components.templates.TemplatesItemBase'>
2017-04-04 16:03:53,114 INFO Loading component type <class 'grafana_dashboards.components.dashboards.Dashboard'>
2017-04-04 16:03:53,114 INFO Loading component type <class 'grafana_dashboards.components.links.Links'>
2017-04-04 16:03:53,114 INFO Loading component type <class 'grafana_dashboards.components.panels.Panels'>
2017-04-04 16:03:53,114 INFO Loading component type <class 'grafana_dashboards.components.rows.Rows'>
2017-04-04 16:03:53,114 INFO Loading component type <class 'grafana_dashboards.components.templates.Templates'>
2017-04-04 16:03:53,114 INFO Loading component type <class 'grafana_dashboards.components.links.DashboardLink'>
2017-04-04 16:03:53,114 INFO Loading component type <class 'grafana_dashboards.components.links.AbsoluteLink'>
2017-04-04 16:03:53,114 INFO Loading component type <class 'grafana_dashboards.components.panels.Graph'>
2017-04-04 16:03:53,114 INFO Loading component type <class 'grafana_dashboards.components.panels.SingleStat'>
2017-04-04 16:03:53,115 INFO Loading component type <class 'grafana_dashboards.components.panels.Table'>
2017-04-04 16:03:53,115 INFO Loading component type <class 'grafana_dashboards.components.panels.Text'>
2017-04-04 16:03:53,115 INFO Loading component type <class 'grafana_dashboards.components.rows.Row'>
2017-04-04 16:03:53,115 INFO Loading component type <class 'grafana_dashboards.components.templates.Query'>
2017-04-04 16:03:53,115 INFO Loading component type <class 'grafana_dashboards.components.templates.EnumeratedTemplateBase'>
2017-04-04 16:03:53,115 INFO Loading component type <class 'grafana_dashboards.components.templates.CustomTemplate'>
2017-04-04 16:03:53,115 INFO Loading component type <class 'grafana_dashboards.components.templates.IntervalTemplate'>
2017-04-04 16:03:53,178 DEBUG Adding component 'project' with name 'Example project'
2017-04-04 16:03:53,179 INFO Component '['default-dashboard']' does not have 'name' attribute, skipping
2017-04-04 16:03:53,179 DEBUG Adding component 'dashboard' with name 'overview'
2017-04-04 16:03:53,179 DEBUG Adding component 'dashboard' with name '{component}-connections'
2017-04-04 16:03:53,179 DEBUG Adding component 'dashboard' with name '{component}-{env}-{nodes}-system-details'
2017-04-04 16:03:53,179 DEBUG Adding component 'rows' with name 'first-row'
2017-04-04 16:03:53,179 DEBUG Adding component 'rows' with name 'second-row'
2017-04-04 16:03:53,179 DEBUG Adding component 'rows' with name '{param}-row'
2017-04-04 16:03:53,180 DEBUG Adding component 'panels' with name 'connections'
2017-04-04 16:03:53,180 DEBUG Adding component 'panels' with name 'memory'
2017-04-04 16:03:53,180 DEBUG Adding component 'templates' with name 'template'
2017-04-04 16:03:53,180 DEBUG Adding component 'dashboard' with name 'some dashboard'
2017-04-04 16:03:53,180 DEBUG Adding component 'templates' with name 'another-template'
2017-04-04 16:03:53,180 DEBUG Adding component 'panels' with name 'singlestat-{title}'
Traceback (most recent call last):
  File "/usr/local/bin/grafana-dashboard-builder", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/grafana_dashboards/cli.py", line 98, in main
    projects = DefinitionParser().load_projects(paths)
  File "/usr/local/lib/python2.7/dist-packages/grafana_dashboards/parser.py", line 33, in load_projects
    registry.add(component)
  File "/usr/local/lib/python2.7/dist-packages/grafana_dashboards/components/base.py", line 72, in add
    'This contains %s attributes' % len(component.keys()))
AttributeError: 'str' object has no attribute 'keys'

Table panel

Hello,
can you add support for table panel?
We use it at work to show switch interface traffic.

Thanks

Add support for passing list of params to dashboard-link params

Singlestat template:

- name: singlestat-{title}
  panels:
    - single-stat:
        title: '{title}'
        error: false
        span: 4
        sparkline:
          show: false
          full: false
          lineColor: "rgb(31, 120, 193)"
          fillColor: "rgba(31, 118, 189, 0.18)"
        valueMaps:
          'null': '{title}'
        links:
          - dashboard-link:
              title: '{title}'
              dashboard: '{dashboard}'
              params: '{params}'

Dashboard definition:

      - row:
          title: 'Zendesk4'
          height: '250px'
          editable: true
          collapse: true
          panels:
            - singlestat-{title}:
                title:
                  - Zendesk4 connector overview:
                      dashboard: 'Zendesk4 Connector Overview'
                  - Zendesk4 connector resources:
                      dashboard: 'ATeam - connectors zendesk4 resources'
                      params:
                        - cluster: 51
                        - node: 'na1-connectors-www01'

Add support for panel type "dashlist"

There is new type of panel in "new" Grafana versions - "dashlist".
It is quite handy for crossroad use case.
In multiple rows / columns it displays list of dashboard names being links to these dashboards.
Dashboards are included using search or list of tags.
Also there is special type of dashlist displaying recently viewed dashboards.
Here is example JSON code:

  "rows": [
    {
      "panels": [
        {
          "headings": true,   # turn on headers on each 
          "limit": 20,             # do not display more than limit
          "span": 6,              # If more panels are in the same row
          "query": "",            # search dashboards using this query
          "recent": false,      # show recently viewed
          "search": true,       # turn on search
          "starred": false,     # display only starred
          "tags": [                 # search using tags
            "vertica_overview"
          ],
          "title": "Overview dashboards",
          "type": "dashlist"
       },

Support for an All option in template query

Grafana has an All option to add All value to query variable value.
graphite value: site.cluster.node.prefix-value-suffix.subvalue
Example setup for prefix-value-suffix:

  • variable name - var1
  • values query - $site.$cluster.$node.prefix-*
  • All option - has several flavors glob, wildcard, regex wildcard, regex values
"type": "query",
"datasource": null,
"refresh_on_load": false,
"name": "var1",
"options": [
  {
    "text": "All",
    "value": "\*"
  }
],
"includeAll": true,
"allFormat": "wildcard",
"query": "$site.$cluster.$node.prefix-*",

config and connection to grafana

I would like to see an example of how to connect to my grafana instance.

can you please add an example config.yaml to the project and readme.

Single-stat panels

Add support for other panel type beside graph - single stat panel. Would be nice if spark lines were supported :).

Create custom template from variable

It seems at the moment it is impossible to create a custom template variable in the dashboard from a variable defined in the builder yaml file.

Autorefresh setup

I'm not sure if this is possible, but it would be nice if dashboard autorefresh interval could be persisted in dashboard definition.

Support fine-tuning of template query creation

Currently template query use * for searching variable values. It would be nice to allow customization

  templates:
    - query:
        query: '{metric-prefix}.{component}.$application'
        application:
          metric: '*core*'
          current: first
          options:
            - first
            - last

Missing features

Could be useful to have the option to add the "transparent" parameter (under General tab) and also to hide the metrics entries (Metrics tab).

Support for regex in Template query

Grafana has an regex option to filter out values for given variable.
graphite value: site.cluster.node.prefix-value-suffix.subvalue
Example setup for prefix-value-suffix:

  • variable name - var1
  • values query - $site.$cluster.$node.prefix-*
  • values regex - /prefix-(\w+)-suffix/

json output:
"type": "query",
"datasource": null,
"refresh_on_load": false,
"name": "var1",
"query": "$site.$cluster.$node.prefix-*",
"regex": "/prefix-(\w+)-suffix/"

Grafana 4.5 - definition of X/Y axes is too complicated

  • Old way
y_formats: [s, short]
  • (Current) new way
yaxes:
  - yaxis:
      format: decmbytes
  - yaxis:
      format: decmbytes
xaxes:
  show: true
  format: time

I expect the following:

  • Second Y axis defaults to first one, if it is not needed
    show=true is mandatory + same format
  • X axis defaults to show=true and format=time

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.