grafana-toolbox / grafanimate

Animate timeseries data with Grafana.

grafanimate

Animate timeseries data with Grafana.

About

grafanimate captures screenshots while animating a Grafana dashboard by manipulating its time range control, i.e. navigating through time. The result can be rendered as a sequence of png images, an animated gif file, and as a video file.

Attention!

This program can put significant load on your Grafana instance and the underlying database machinery. Handle with care!

Setup

Prerequisites

This program uses the fine FFmpeg program for doing the heavy lifting within in its postprocessing subsystem.

grafanimate

pip install grafanimate

Note

We absolutely recommend installing this program into a Python virtualenv:

python3 -m venv .venv
source .venv/bin/activate
pip install grafanimate

Usage

Introduction

grafanimate works by operating on animations defined within description files, written in Python. In cinematography jargon, this is called "exposure sheet", or "dope sheet".

An exposure sheet (also known variously as "dope sheet", "camera instruction sheet", or "X-sheet") is a sheet of paper used primarily in traditional animation to mark out the timing of various actions and dialogue.

grafanimate offers convenient data types, AnimationScenario and AnimationSequence, for outlining an animation scenario made of multiple sequences.

Please have a look at the scenarios.py file for a full example containing multiple scenarios.

Synopsis

A scenario definition:

AnimationScenario(
    grafana_url="https://play.grafana.org/",
    dashboard_uid="000000012",
    sequences=[
        AnimationSequence(
            start="2021-11-15T02:12:05Z",
            stop="2021-11-15T02:37:36Z",
            every="5min",
            mode=SequencingMode.CUMULATIVE,
        ),
    ],
)

In order to run a built-in scenario, invoke:

grafanimate --scenario=playdemo --output=./animations

Details

grafanimate also supports relative timestamps, based on the fine pytimeparse2 library.

• Within every, you will express a duration.

Help

For getting a detailed and descriptive overview about all available command line options, please invoke:

grafanimate --help

Examples

Examples for scenario mode. Script your animations in file scenarios.py or any other Python module or file.

# Use freely accessible `play.grafana.org` for demo purposes.
grafanimate --scenario=playdemo --output=./animations

# Example for generating Luftdaten.info graph & map.
export GRAFANIMATE_OUTPUT=./animations
grafanimate --grafana-url=http://localhost:3000/ --dashboard-uid=1aOmc1sik --scenario=ldi_all

# Use more parameters to control the rendering process.
grafanimate --grafana-url=http://localhost:3000/ --dashboard-uid=acUXbj_mz --scenario=ir_sensor_svg_pixmap \
    --header-layout=studio --datetime-format=human-time --panel-id=6

Usage in Containers

You can use grafanimate with Docker and Podman. An OCI image is published to ghcr.io/panodata/grafanimate.

docker run --rm -it --volume=$(PWD)/animations:/animations ghcr.io/panodata/grafanimate \
    --header-layout=no-chrome \
    --video-fps=30 --video-framerate=30 \
    --scenario=playdemo --output=./animations

Gallery

Composition: The Hiveeyes Project. Developing a flexible beehive monitoring infrastructure. Clicking on an image will take you to the animated version.

luftdaten.info coverage

Coverage of luftdaten.info sensors starting October 2015 across Europe.

• Data source: luftdaten.info (LDI)
• Production: Luftdatenpumpe, LDI data plane v2, daily.
• Development: Erneuerung der Luftdatenpumpe. All contributions welcome.

Fine dust pollution on New Year's Eve

Animation of fine dust pollution on New Year's Eve 2018 across Europe.

• Data source: luftdaten.info (LDI)
• Production: Luftdatenpumpe, LDI data plane v2, historical.
• Development: Erneuerung der Luftdatenpumpe. All contributions welcome.

DWD CDC

Short weather film about temperature, sun and precipitation based on DWD/CDC data in March 2018.

• Data source: DWD Open Data (DWD)
• Production: DWD Climate Data Center (CDC), 10m-Werte: Aktuelle Lufttemperaturen, Sonnenscheindauer & Niederschlag
• Development: <work in progress>

IR-Sensor SVG-Pixmap

IR-Sensor SVG-Pixmap displaying temperature changes inside a beehive.

• Data source: Clemens Gruber (CG)
• Development: How to Visualize 2-Dimensional Temperature Data in Grafana

Background and details

Introduction

Animating things in Grafana across the time-axis in the spirit of the GeoLoop Panel Plugin hasn't been unlocked for Grafana in a more general way yet. Challenge accepted!

Time warp

At this programs' core is the code to set time range in Grafana:

timeSrv = angular.element('grafana-app').injector().get('timeSrv');
timeSrv.setTime({from: "2015-10-01", to: "2018-12-31"});

Rendering engine

Turtles all the way up, the main rendering work horse is a Firefox Browser automated through Marionette Python Client fame:

The Marionette Python client library allows you to remotely control a Gecko-based browser or device which is running a Marionette server.

Outlook

Neither Playlists nor Scripted Dashboards (now deprecated) offer these things to the user, but this program can be combined with both in order to implement more complex animations on top of Grafana.

---

Development

# Acquire sources.
git clone https://github.com/panodata/grafanimate
cd grafanimate

# Create and activate virtualenv.
python3 -m venv .venv
source .venv/bin/activate

# Install package in "editable" mode.
pip install --editable=.

# Run tests.
make test

Project information

The code lives on GitHub and the Python package is published to PyPI.

Contributing

We are always happy to receive code contributions, ideas, suggestions and problem reports from the community. Spend some time taking a look around, locate a bug, design issue or spelling mistake and then send us a pull request or create an issue. You can also discuss grafanimate on our forum, you are welcome to join.

Acknowledgements

Thanks to all the contributors who helped to co-create and conceive this program in one way or another. You know who you are.

Also thanks to all the people working on Python, Grafana, Firefox, FFmpeg, and the countless other software components this program is based upon.

License

grafanimate is licensed under the terms of the GNU AGPL v3 license.

grafanimate's Issues

Optimize size and rebuild behavior of OCI image

Optimize image size

Each command creates a layer in the docker image that contains the changes to the previous layer.
All files are still left in the previous layer and space is saved in the image.
To fix this, run apt install, pip install, and apt remove in a single RUN command so that files are not kept in hidden layers.

Originally posted by @intermittentnrg in #24 (comment)

Optimize rebuild behavior

Currently, pip install is copying all python files from /src to elsewhere so it now needs to run and will install dependencies when any .py/application file is changed. Changes in Python should not re-run pip etc., only if setup.py (or README.rst) were changed.

Originally posted by @intermittentnrg in #24 (comment) and #24 (comment)

Timing problem when rendering specific dashboard

When trying to render https://weather.hiveeyes.org/grafana/d/of6c9qlmk/uba-ldi-dwd-studio-dev, it feels like grafanimate has some problem taking the snapshot at exactly the time after data loading has finished, see screenshot:

Speed control

People asked us about how to control the recording speed of grafanimate:

Is it possible to adjust the "frameduration" with which a snapshot is present at the end of the video somewhere?

Grafana Graph data not showing

I'm having an issue with Grafanimate not displaying my data in Grafana. The program runs and produces the animation and video, however, it does not display any data (it displays the panel titles, but not the graphs themselves). When I check the log, I see these errors. Do you have ideas on why this might be happening?

Note: My dashboard is password protected, but I inputted the login credentials in my code.

Log Errors:

1663598594466   Marionette  DEBUG   Waiting for initial application window
console.warn: SearchSettings: "get: No settings file exists, new profile?" (new NotFoundError("Could not open the file at /var/folders/g1/pdxb7rws781f2fvd86cqvbx40000gp/T/tmpo6ihc0y7.mozrunner/search.json.mozlz4", (void 0)))
console.error: Region.jsm: "Error fetching region" (new TypeError("NetworkError when attempting to fetch resource.", ""))
console.error: Region.jsm: "Failed to fetch region" (new Error("NO_RESULT", "resource://gre/modules/Region.jsm", 420))
1663598595089   Marionette  DEBUG   1 <- [1,1,null,{"sessionId":"7d3ad611-aa7b-49bf-9d70-63b0e7330fe5","capabilities":{"browserName":"firefox","browserVersion":"104.0 ... ownTimeout":60000,"moz:useNonSpecCompliantPointerOrigin":false,"moz:webdriverClick":true,"moz:windowless":false,"proxy":{}}}]
1663598595091   Marionette  DEBUG   1 -> [0,2,"WebDriver:SetTimeouts",{"pageLoad":30000}]
1663598595091   Marionette  DEBUG   1 <- [1,2,null,{"value":null}]
1663598595092   Marionette  DEBUG   1 -> [0,3,"WebDriver:SetTimeouts",{"script":20000}]
1663598595092   Marionette  DEBUG   1 <- [1,3,null,{"value":null}]
1663598595096   Marionette  DEBUG   1 -> [0,4,"Marionette:GetContext",{}]
1663598595096   Marionette  DEBUG   1 <- [1,4,null,{"value":"content"}]
1663598595097   Marionette  DEBUG   1 -> [0,5,"Marionette:SetContext",{"value":"chrome"}]
1663598595097   Marionette  DEBUG   1 <- [1,5,null,{"value":null}]
1663598595104   Marionette  DEBUG   1 -> [0,6,"WebDriver:ExecuteScript",{"script":"Components.utils.import(\"resource://gre/modules/Preferences.jsm\");\n\n            ... Sandbox":true,"sandbox":"default","line":761,"filename":".venv/lib/python3.9/site-packages/marionette_driver/marionette.py"}]
1663598595111   Marionette  DEBUG   1 <- [1,6,null,{"value":null}]
1663598595113   Marionette  DEBUG   1 -> [0,7,"Marionette:SetContext",{"value":"content"}]
1663598595113   Marionette  DEBUG   1 <- [1,7,null,{"value":null}]
1663598595115   Marionette  DEBUG   1 -> [0,8,"WebDriver:Navigate",{"url”:”[“my_grafana_url}]
UNSUPPORTED (log once): POSSIBLE ISSUE: unit 1 GLD_TEXTURE_INDEX_2D is unloadable and bound to sampler type (Float) - using zero texture because texture unloadable
1663598595737   Marionette  DEBUG   1 <- [1,8,null,{"value":null}]
1663598595739   Marionette  DEBUG   1 -> [0,9,"WebDriver:FindElement",{"value":"grafana-app","using":"tag name"}]
1663598595744   Marionette  DEBUG   1 <- [1,9,null,{"value":{"element-6066-11e4-a52e-4f735466cecf":"f16c9061-b745-413b-8a82-9819d91acdcf"}}]
1663598596051   Marionette  DEBUG   1 -> [0,10,"WebDriver:ExecuteScript",{"script":"// http://stackoverflow.com/questions/5538972/console-log-apply-not-working-in-ie9 ... () == text.trim();\n    });\n};","args":[],"newSandbox":false,"sandbox":null,"line":153,"filename":"grafanimate/grafana.py"}]
1663598596052   Marionette  DEBUG   1 <- [1,10,null,{"value":null}]
1663598596054   Marionette  DEBUG   1 -> [0,11,"WebDriver:ExecuteScript",{"script":"// -*- coding: utf-8 -*-\n// (c) 2018-2021 Andreas Motl <[email protected] ... or().get(\"grafanaStudioSrv\");","args":[],"newSandbox":false,"sandbox":null,"line":153,"filename":"grafanimate/grafana.py"}]
1663598596057   Marionette  DEBUG   1 <- [1,11,null,{"value":null}]
1663598596058   Marionette  DEBUG   1 -> [0,12,"WebDriver:ExecuteScript",{"script":"return grafanaStudio.login([login_info]);”,”args":[],"newSandbox":false,"sandbox":null,"line":153,"filename":"grafanimate/grafana.py"}]
1663598596061   Marionette  DEBUG   1 <- [1,12,null,{"value":null}]
1663598596063   Marionette  DEBUG   1 -> [0,13,"WebDriver:ExecuteScript",{"script":"return console.info(\"Opening dashboard\");","args":[],"newSandbox":false,"sandbox":null,"line":153,"filename":"grafanimate/grafana.py"}]
1663598596065   Marionette  DEBUG   1 <- [1,13,null,{"value":null}]
1663598596065   Marionette  DEBUG   1 -> [0,14,"WebDriver:ExecuteScript",{"script":"return grafanaStudio.openDashboard(\”[dashboard_id]\”, {\”scenario\”: \”[scenario_name]\”, \" ...  \"use-panel-events\": false});","args":[],"newSandbox":false,"sandbox":null,"line":153,"filename":"grafanimate/grafana.py"}]
console.warn: LoginRecipes: "Falling back to a synchronous message for: [grafana_url]”

Ex of empty panels (screenshot from mp4 file):

What panel should look like:

Specify delta and duration from scenarios.py

Hello again! My use of this tool is intermittent (pun intended). It takes time for data to accumulate, so I only make new videos every few months.

I would like to provide custom delta/interval, i.e. duration that is skipped forward per frame.
I just want to provide my own value to have exact control.

The delta is calculated from the every parameter in scenarios.py in get_freq_delta.
https://github.com/panodata/grafanimate/blob/main/grafanimate/timeutil.py#L40

Maybe just skip the call to get_freq_delta is a delta is provided from scenarios.py? https://github.com/panodata/grafanimate/blob/main/grafanimate/model.py#L48

Event handling broken since Grafana 8 introduced React

Hi there,

since Grafana 8 introduced React, grafanimate is having a hard time to properly synchronize with the frontend in order to capture frames efficiently. More information can be found at ¹.

With kind regards,
Andreas.

Footnotes

1. https://community.grafana.com/t/how-to-add-event-handler-in-grafana-8-react-panel-plugins/58668/4. ↩

Debian Firefox ESR not used

On Debian, Firefox ESR is normally installed. In /usr/bin/ there is "firefox" and "firefox-esr". When find_firefox() sees "firefox", it tries to use it and fails. It needs to find and try "firefox-esr" first.

The follow change in marionette.py line 111 fixes the problem, and should still work for a regular firefox installation.

def find_firefox(cls):
    candidates = where.where("firefox-esr")
    candidates += where.where("firefox")
    candidates += [
        "/Applications/Firefox.app/Contents/MacOS/firefox-bin"
    ]

Looking forward to 2021

a little collection/notes on stuff trying to get things towards grafana 7:

six v1.13.0

Successfully built grafanimate
ERROR: mozrunner 8.2.1 has requirement six<2,>=1.13.0, but you'll have six 1.11.0 which is incompatible.
ERROR: mozlog 7.1.0 has requirement six>=1.13.0, but you'll have six 1.11.0 which is incompatible.

• seems like within ubuntu/debilian's pip-world, mozrunner and mozlog require six >=1.13.0. changing to that version in setup.py helped.

<grafana-app>

/seems/ to me that waiting for fails. (it is still there in grafana7).

list of panels as promises not aquired

when replacing the awaiting of with a little sleep, we get here:
2021-11-13 22:45:35,661 [grafanimate.grafana ] INFO : Waiting for "all-data-received" event
and go timeout.

all-data-received seems to wait for some promises in grafana-studio.js -- after uncommenting log('panel:', panel); it seems that the dashboard-list is already empty. turtles! turtles everywhere! :)

grafanimate --grafana-url=https://play.grafana.org/ --dashboard-uid=000000012 --scenario=wtf_demo
2021-11-13 23:20:13,854 [grafanimate.grafana           ] INFO   : Starting GrafanaWrapper on https://play.grafana.org/
2021-11-13 23:20:13,854 [grafanimate.marionette        ] INFO   : Starting Marionette Gecko wrapper
2021-11-13 23:20:13,855 [grafanimate.marionette        ] INFO   : Found "firefox" program at /usr/bin/firefox
2021-11-13 23:20:13,855 [grafanimate.marionette        ] INFO   : Check for running instance of Marionette/Firefox at localhost:2828
2021-11-13 23:20:13,859 [grafanimate.marionette        ] INFO   : Will reuse running Marionette/Firefox
2021-11-13 23:20:13,878 [grafanimate.grafana           ] INFO   : Starting Grafana at https://play.grafana.org/
2021-11-13 23:20:16,095 [grafanimate.grafana           ] INFO   : Waiting for Grafana to load
2021-11-13 23:20:19,644 [grafanimate.grafana           ] INFO   : Grafana loaded
2021-11-13 23:20:19,676 [grafanimate.animations        ] INFO   : Opening dashboard
2021-11-13 23:20:19,677 [grafanimate.grafana           ] INFO   : Running Javascript: return grafanaStudio.openDashboard("000000012", {"dashboard-view": null, "panel-id": null, "datetime-format": null, "header-layout": ["large-font"]});
2021-11-13 23:20:19,768 [grafanimate.grafana           ] INFO   : Waiting for "all-data-received" event
2021-11-13 23:20:39,794 [grafanimate.grafana           ] WARNING: Timed out waiting for data: Timed out after 20.0 seconds. Continuing anyway.
2021-11-13 23:20:39,796 [grafanimate.animations        ] INFO   : Dashboard ready
2021-11-13 23:20:39,796 [grafanimate.scenarios         ] INFO   : Running wtf demo
2021-11-13 23:20:39,799 [grafanimate.animations        ] INFO   : Creating rrule: dtstart=2021-11-12 09:00:00, until=2021-11-12 10:00:00, freq=5, interval=10
2021-11-13 23:20:39,802 [grafanimate.animations        ] INFO   : Datetime step: 2021-11-12 09:00:00
2021-11-13 23:20:39,803 [grafanimate.grafana           ] INFO   : Timewarp to 2021-11-12 09:00:00 -> 2021-11-12 09:09:59
2021-11-13 23:20:40,058 [grafanimate.grafana           ] INFO   : Waiting for "all-data-received" event

Cumulative time ranges

People asked for controlling grafanimate with cumulative time ranges/intervals, see also Kumulative Zeitintervalle mit grafanimate.