equinor / webviz-config

Make Dash applications from a user-friendly config file :book: :snake:

This package will be deprecated - we move instead all collaboration focus to the reusable React and Dash components in:

• https://github.com/equinor/webviz-subsurface-components
• https://github.com/equinor/webviz-core-components

---

Democratizing Python web applications

Writing a Dash web application gives a lot of flexibility, however, it also requires 🐍 Python knowledge from the person setting it up.

Webviz™ is a MIT-licensed configuration layer on top of Dash, which encourages making reusable components and dashboards, which can then be added/removed when creating an application using a short yaml configuration file.

This Python package, webviz-config, is the core plugin framework. For a real example repository using this plugin system, see e.g. webviz-subsurface.

These are the main user groups targeted by Webviz™:

• You do not know Python, and only want to add different predefined dashboards or visualizations/components in a certain order and/or on different pages in the application. Optionally with some text and mathematical equations (that you provide) inbetween the dashboards, explaining what the user is looking at.
• You know Python, and want to create generic or specialized dashboards you or other users can reuse by simply asking for it in the Webviz™ configuration file. This can be done without knowing JavaScript (see also Dash for more information).
• You know React, and want to create highly specialized visualization which Python or pure config-file users can reuse.

Webviz™ will create web applications with very 🔒 strict security headers and CSP settings, giving an rating of A+ on e.g. Mozilla observatory. It also facilitates a 🐳 Docker setup, where the Python code can be ran completely unpriviliged in a sandbox (both with respect to file system access and network communication).

Example configuration file and information about the standard plugins can be seen in the documentation.

The workflow can be summarized as this:

1. The user provides a 📖 configuration file following the yaml standard.
2. Webviz™ reads the configuration file and automatically writes the corresponding 🐍 Python code.
3. The created application can be viewed locally, or deployed using 🐳 Docker to a cloud provider. Both out of the box.

---

Installation

The recommended and simplest way of installing webviz-config is to run

pip install webviz-config

If you want to develop webviz-config and install the latest source code manually you can do something along the lines of:

git clone [email protected]:equinor/webviz-config.git
cd ./webviz-config
npm ci --ignore-scripts
# NOTE: If you are on Windows, make sure to first set `npm`'s default `script-shell` to `powershell` by running
# npm config set script-shell powershell
npm run postinstall
pip install -e .

After installation, there is a console script named webviz available. You can test the installation by using the provided example configuration file,

webviz build ./examples/basic_example.yaml

Without any additional arguments, this will

1. create a temporary folder
2. write the Python application code to that folder
3. start a localhost server

When stopping the server (press CTRL+C at any time), the temporary folder is deleted.

The optional arguments can be seen when running

webviz --help

Usage

See the introduction page for information on how you create a webviz configuration file and use it.

Creating new plugins

If you are interested in creating new plugins which can be configured through the configuration file, take a look at the contribution guide.

To quickly get started, we recommend using the corresponding cookiecutter template.

License

webviz-config is, with a few exceptions listed below, MIT licensed.

• The webviz-config logo is CC BY-NC-ND 4.0

webviz-config's Issues

Layout of plugins

A user wants to control how plugins are placed on a page.
Some testing is needed to find the best solution.
It is probably easiest to expose some classes that can be modified using css from a theme.

• Add a "plugin-layout" class to plugins
• Wrap plugins in a parent div with a "plugin-parent-layout" class

Update DataTable container to support latest DashTable

The Dash Table API has been changed in 4.0.0

https://dash.plot.ly/datatable/interactivity

E.g. the bool sorting is replaced by a string sort_action
filtering with filter_action

• Implement Dash Table API changes in the DataTable container

Interactions between plugins

This is mainly to start a discussion on how to have interaction between different plugins.

Motivation

The main use case for interactions is data "drilldown".
E.g. narrowing data selection by having a global filter or by selecting data in one visualization which updates data in another visualization.

Here is an example of potential interaction between two webviz-subsurface plugins.
A callback has been added to the SummaryStats plugin which stores the realization of the currently clicked line. A callback in the Intersect plugin picks up the stored realization and updates the plot. This simple interactivity makes it much easier to navigate and understand the underlying data.

This type of interactivity is fairly easy to do within a single plugin and is described well in the Dash documentation, but it is currently not possible to do (in a robust way) between different Webviz plugins.

Solutions

The easiest solution is just to have it as it is today. Interactivity can be added for individual plugins without allowing any interactions between different plugins.

This will probably give rise to alot of duplicate code as the same type of visualizations would need to be combined in different plugins

We could mitigate this by introducing e.g. "nested plugins" which can be built by combining multiple plugins with some additional callbacks to share data

We could also make plugins more re-usable by separating out visualizations from control components (e.g. dropdowns) and callbacks making it easier to build new plugins using existing code.

Alternatively we can investigate having a shared app state with key, value pairs which plugins can read and write to.

PdfViewer

A user wants to view PDF files in webviz

This could be easily done by using the built in dash html components.
E.g. html.Embed(pdf_file)
But this will require changes to the CSP.

• Investigate most secure way of adding PDF files
• Add a container to view PDF files

Support for sub pages

A user wants to have sub pages to organize a lot of content.

• Add support for tabs in the config file

• Add tabs in the jinja templating

• Add theming of tabs

Extend Markdown file with math support

...perhaps using either MathJax or KaTeX.

Depends on #27.

Implement Webviz.Graph class

Should we implement a webviz.Graph class that inherits dcc.Graph for use in our containers?
This would allow us to set e.g. config options such as responsive and modeBarButtonsToRemove for all graphs as default.

Portable app has hot reload

The portable standalone generated app uses hot reload when running on localhost. This adds unnecessary runtime since flask then loads twice initially (hot reload is only useful during interactive making of the webviz configuration file, i.e. when running in non-portable mode - not when viewing a finished app).

Make Markdown container portable

• Add Markdown file to Webviz store

Custom themes modify default theme settings

Expose plotly layout

A user want to style Plotly graphs using a theme with e.g. colors
This could be done by defining a dictionary that is passed into each container.
The dictionary could then be added to a plotly graph in a callback.
E.g. my_plot['layout'].update(self.plotly_layout)

• Add a plotly_layout dict to the default theme
• Add plotly_layoutas a SPECIAL_ARGS

Extend scope of hot reload

Currently hot reload has mostly been tested with simple yaml configuration files, using simple plugins. It also only checks the yaml configuration file for changes - not files linked to within the configuration file.

• Extend file watch to also include input files to different plugins? Markdown files, ++
• Does caching work also with hot reload?

Users want browser to open when Dash is ready

When running webviz build some_config.yaml, users want the webviz application to open automatically in default browser.

Consider doing the same when running webviz build some_config.yaml --portable ./some/outputfolder.

Parallelize generation of `webviz_storage`?

Use Dash 1.0.0

• Bump Dash version to 1.0.0

• Rewrite tests to use new Dash pytest fixtures

Add tests of complete build

Both non-portable and portable mode of webviz build.

Change cache type

Currently we are using simple flask-caching. This is not 100% multithread compatible according to docs.

Change to e.g. redis?

Collect ANSI escape sequences in e.g. class or module as constants

Currently there are ANSI sequences like '\033[92m' in different output statements. These are hard to read in the code. Collect as constants to a single class/module, and instead use those definitions.

Show error messages and traceback

With debug=False it looks like Python error messages are not shown in the terminal. Investigate how to combine (if possible):

1. debug=False
2. print error messages to terminal

There is some information here and here.

Webviz store not working after introducing webviz container placeholder

When running webviz build with the --portable option, no files are generated in webviz_store.

Add container parent class

A user that is developing a container should be able to inherit some default arguments.
(e.g. plotly_layout)

• Add parent class for containers
• Rewrite existing containers to use parent class

Setup readthedocs

Hot reload not working

If debug in the flask server is not set to True, it looks like hot reloading of the dash application is not working. According to dash documentation, there is a flag dev_tools_hot_reload which can be set to True to enable hot reloading (without enabling the whole suite of debug tools).

Here is the line in the template setting debug/dev_tools_hot_reload and here is the dash documentation on the dev tools.

Custom containers are not registerable

Before the WebvizContainer class was introduced custom containers could be added e.g like so:

1. Create a python script with a Container class. E.g.:

import dash_html_components as html


class OurCustomContainer():

    def __init__(self, title: str):
        self.title = title

    @property
    def layout(self):
        return html.Div([
                         html.H1(self.title),
                         'This is just some ordinary text'
                        ])

2. Make the scripts location available in PYTHONPATH
3. Include the Container in the config file as:
   - container: <scriptname>.OurCustomContainer

With the addition of the WebvizContainerparent class the above script should be:

import dash_html_components as html
from webviz_config.containers import WebvizContainer


class OurCustomContainer(WebvizContainer):

    def __init__(self, title: str):
        self.title = title

    @property
    def layout(self):
        return html.Div([
                         html.H1(self.title),
                         'This is just some ordinary text'
                        ])

and added to the config file in the same way.

This does not work and raises an AttributeError:

AttributeError: module 'webviz_config.containers' has no attribute 'OurCustomContainer'

Metadata shown through parent container

Based on an idea from @perolavsvendsen. The base class for WebvizContainers could perhaps facilitate showing metainformation like:

• What is the data source?
• Who to contact for more detailed information?
• A guided tour in the more complex containers (using more text than what is practical to always show/include in e.g. a plot).

Table Plotter: Plot options that are not in use for a given plot should be hidden

• Hide unused plot options

• Add tests

Restructure webviz-config code

The webviz-config code base has gotten new features since first draft was commited. The code is still quite modular, however, there are room for improvement and restructuring.

Increase CACHE.timeout

Increase the cache timeout from 60 seconds to 1 hour.

Markdown from files

A user wants to render a markdown text file
Currently markdown can be added as individual lines in the config file. This gets messy for large sections of text.

• Add container that reads markdown files
• Add support for images (if possible)

Add link to hosted plugin documentation on ParserError

When plugins are missing a required argument, or it is of the wrong type, the user is currently provided by a decent error message. Example:

Traceback (most recent call last):
[...]
webviz_config._config_parser.ParserError: The plugin `DataTable` requires the argument `csv_file` in your configuration file.

This could perhaps be further improved by also giving a URL to hosted plugin documentation (if available from the setup.py install script and project_urls["Documentation"]).

Container dependencies

Support for adding container specific assets.

Add hosting guidelines to docs for Azure

Data download

Depends on #37.

There are needs of getting the data used in a container/visualization. This could e.g. for using the values further in Excel sheets for specific calculations, for use in reports etc.

Add a "Download to csv"-function to the future ABC-container modebar, and let each container author provide a function which e.g. provides a dataframe structure which should be transferred to csv.

Improved status print when creating portable instance

When creating a portable instance, the different functions called can be time consuming and many.
The user can sometimes be unsure if it is still running and/or overall progress

E.g.:

• Add a dynamic "terminal spinner" to show the script is still live.
• Add overall progress (e.g. finisihed with x% of the functions to be called).

Automatic deploy to pypi

The package should be automatically built and uploaded to pypi on tag releases.

List all installed plugins

When setting up a configuration file, it is currently difficult for the average user to know from the command line which plugins are installed.

• Add a subcommand e.g. webviz plugins which lists all installed plugins, a short explanation of what it does, together with link to hosted documentation (if it exists in setup.py, see also #100).

Support list and dict in webviz store

Currently list and dict can't be used as arguments in functions given to webviz_store.

Simplify docker building process

Simplification and standardization can be done by changing the app dockerfile to use an official base image. In addition to end-user simplification, we facilitate pushing out security updates if necessary in the dependency tree.

• Verify that current deployment setup works with recent changes.
• Check if manually created webviz base image can be replaced with official base image built from https://github.com/equinor/webviz-docker.
• Do changes in https://github.com/equinor/webviz-docker if necessary.
• Simplify the setup in https://github.com/equinor/webviz-config/tree/master/webviz_config/static by going from

  ARG WEBVIZ_BASE_IMAGE
  FROM $WEBVIZ_BASE_IMAGE
  

  to FROM webviz/base_image.

Figure width

I don't seem to be able to choose the width (and height) of a figure within the Markdown container.

When creating a figure with ![Alt text](./example_banner.png "Some caption") adding =100x200 at the end as in https://stackoverflow.com/questions/14675913/changing-image-size-in-markdown doesn't seem to work.

I also tried using pure html <img src="./example_banner.png" width="200"/> with no success.

Use Markdown in Syntaxhighlighter

Update Chrome driver in Travis

With Travis you now recently get Chrome version 74 when asking for stable Chrome.

This results in Travis CI failing due to incompatible driver. Update to e.g. this line to a more recent driver like https://chromedriver.storage.googleapis.com/74.0.3729.6/chromedriver_linux64.zip

Move `WebvizContainer` to separate module

Move the abstract base class WebvizContainer, which all actual containers inherit, to separate module in order to remove risk of circular import (this can happen if a container author imports their own container in a non-standard way in e.g. test scripts).

Certificate docstrings and scope

• Add more docstrings to the _certificate.py module.
• Extend lifetime, and simultaneously even further restrict DNS_NAME scope (to e.g. localhost only).

From «basic authentication» to token based

Currently webviz-config uses basic HTTPS authentication from dash-auth when running on localhost . This degrades the user experience due to:

• The user has to provide some previously configured webviz username+password pair each time opening a browser —> less productive.
• On a single-user (e.g. private) computer/machine it is an unnecessary barrier (the server is only accessible from localhost, and there is only one user with access locally).

An alternative approach could be to use a token 🔑 based approach instead when running on localhost. The user will be familiar with this approach from Jupyter notebooks.

• https://jupyter-notebook.readthedocs.io/en/stable/security.html
• https://neilmadden.blog/2019/01/16/can-you-ever-safely-include-credentials-in-a-url/

Add CI service

E.g. Travis.

Remove html Title argument from containers

Some containers has a title argument as input which renders a dcc.H1 element.
This should instead be handled by markdown outside the container to allow more flexibility in container placement and styling.

Add metadata info and 'do not edit' suggestion in generated app file

Improve default theme

A user expects a default theme when starting webviz.

• Improve fonts in default theme

• Make default logo

Certificate install instructions

Currently install instructions are printed to the terminal, and only for Chromium.

Change e.g. to

• automatically open guide in default browser (this will ensure that the user also becomes aware of what is the default browser).
• move guide from terminal to github page.
• add for other popular browsers as well (Firefox etc).

Contact person phone number

When typed as an integer without e.g. "+47", yaml treats it as an integer, which then fails when bleach html sanitize it (expects a string).