openclimatefix / uk-analysis-dashboard

This is a Streamlit app for the OCF team that reports database statistics

uk analysis dashboard

Internal dashboard for OCF to track UK forecast statistics.

The analysis dashboard is a tool that was developed for OCF’s internal use and continues to evolve.

Built with Streamlit, a Python-based framework made specifically for creating data apps, the dashboard tracks and displays Quartz Solar and other data model statistics, such as mean absolute error (MAE) on both the national and GSP level. The database provides the error statistic using Sheffield Solar's PVLive day-after updated values as the baseline --the larger the error, the less accurate the forecast.

Thanks to the analysis dashboard, OCF has a valuable feedback tool for understanding the accuracy of both the Quartz Solar forecast and other models in production.

Installation

In the main project folder, install requirements: pip install -r requirements.txt or pip3 install -r requirements.txt.

Run streamlit hello to check that Streamlit installed. A "Welcome to Streamlit!" page should open in the browser.

Create a login secret: echo "password = example" > src/.streamlit/secrets.toml.

Database connection

To run the app locally, you'll need to connect it to the forecast development database

OCF team members can connect to the forecast development database using these Notion instructions. Add DB_URL= (db_url from notion documents) to a secrets.toml file. Follow the instructions in the Notion document to connect to the database v.

Run app: cd src && streamlit run main.py.

files

main.py

main.py contains functions for the home page of the app, which focuses on MAE for the OCF Quartz Solar forecast.

forecast.py

forecast.py contains functions for the forecast page. The forecast page looks at how well each of OCF's forecast models is performing compared to PVLive updated truth values.

status.py

status.py contains functionality for the status pagwe and allows the OCF team to update the forecast status in the database. This is one of the advantages of using an interface like Streamlit, facilitating status updates in a database.

auth.py

auth.py contains code for the basic authenticaion that's been put in place.

pvsite_forecast.py

TODO

site_toolbox.py

TODO

plots/make_pinball_and_exceedance_plots.py

Function to make pinball and exceedance plots. This shows how good the probabilistic forecasts are doing.

plots/ramp_rate.py

Function to make ramp rate plots.

🛠️ infrastructure

.github/workflows contains some CI actions.

1. docker-pipeline.yml: Creates and publishes a docker image.

With any push to main, in order to deploy changes, the Terraform Cloud variable is updated with the commit reference and deployed to AWS Elastic Beanstalk.

Environmental Variables

• DB_URL: The database url which will be queried for forecasts
• password: The password for accessing the code
• SHOW_PVNET_GSP_SUM: Option to show pvnet_gsp_sum model or not. This defaults to zero

Contributors

The following folks have contributed to this repo.

uk-analysis-dashboard's Issues

analysis dashboard pv site forecast page suggestions / feedback

Detailed Description

This issue is to collect feedback and ideas on the PV Site Forecast page recently deployed on the Development Analysis Dashboard.

The page currently looks like this:

For the moment, it's fairly basic.
The original issue, #41 , was to plot forecast and generation (if data exists) values for a single pv site.

A user can select a site by site_uuid (the unique id used in the database to differentiate sites) and choose a start date for the forecast and generation data.
PV Site Forecast data currently only has 3-4 days of history, so the date selection doesn't go back any further than that for the moment. Some sites have generation data (displayed in orange), but many don't.

What other information about the individual site page could be helpful?

Feel free to comment below what features or parameters could be helpful to add to this page. Ex. Lat / long coordinates, site generation capacity, etc.

Add 90 and 150 forecast horizons

add to https://github.com/openclimatefix/uk-analysis-dashboard/blob/main/src/main.py#L132

dont enter gsp or dno when entereing a site

similar to openclimatefix/pv-site-api#108

attach a `site` to a `sitegroup`

attached all sites to ocf `sitegroup` see code here - https://github.com/openclimatefix/pv-site-datamodel/blob/main/scripts/add_all_sites_to_user.py

Unassigned variable in probabilistic GSP

On the Forecast page, if you select GSP ID 1-317 (i.e. not national) and select the pvnet_v2 model the following error message is displayed:

UnboundLocalError: local variable 'show_prob' referenced before assignment
Traceback:
File "/usr/local/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 541, in _run_script
    exec(code, module.__dict__)
File "/app/main.py", line 271, in <module>
    page_names_to_funcs[demo_name]()
File "/app/forecast.py", line 202, in forecast_page
    if model != "cnn" and len(forecast) > 0 and show_prob:

This error doesn't occur with the cnn model

Auth0 not working

see chris-garrett/garrett-streamlit-auth0#22

plot ulite forecast horizons

Detailed Description

Would be great to be able select mulitple forecast horizons

move get_data.py functions to pv-site-datamodel

Detailed Description

The get_data.py file contains functions that might be more useful in the pv-site-datamodel.

Context

See the following issue:
openclimatefix/pv-site-datamodel#86
These functions were written as the dashboard was being developed when the datamodel didn't offer a way of getting the needed data.

Possible Implementation

• Move functions to the pv-site-datamodel.
• Once the datamodel is updated and deployed, update the ocf-dashboard to import the functions.

Show probablistic levels on forecast plot

Show probabilistic levels on forecast plot, would be great to add the shading on this plot too. Then we can see it for different models.

If plevels is empty then dont show anything

Add metric sum of all gsp

Detailed Description

Add metric on metric page that displays "Daily Latest MAE All GSPs".

Context

• This gives an metrix of how good the forecast is predicting for all the GSPs

Possible Implementation

• load another metric using name="Daily Latest MAE All GSPs" and plot on graph. There will be one data point for each day

Site user page

Add a site user page that does the following

• #53
• #54
• #55
• #56
• #57
• #58
• #59

This is linked with this openclimatefix/pv-site-datamodel#73

Problem loading function in new release

Might need to look at Docker file

Would be useful to run some sort of test on docker file, to see if imports work

OCF Dashboard Sites Toolbox on Production

Detailed Description

Please leave your comments here concerning OCF Dashboard Sites Toolbox deployment.

wide plots

st.set_page_config(layout="wide")

add this text (once) before plots to make all plots wide

Add PVLive sum of GSPs

It would be really useful to add PVLive sum of GSPs as an alternative option to the PVLive national estimate

Context

• @JackKelly had mentioned that PVLive sum of GSPs is likely a better estimate of the true output than the PVLive national estimate. However, the national estimate is what everyone else uses, so I don't think a straight replacement would be the best thing to do. Maybe have both.
• PVNet 1 and 2 produce a national estimate by predicting the GSP values and then summing them. To help monitor the performance of those models, it would be good to have the PVLive sum of GSPs - the PVNet sum is likely to be more similar to this. This could help us asses whether the model is getting the forecast wrong, or if the potential errors in the PVNet national forecast is due to the difference between the PVLive national and PVLive sum of GSPs.

Possible Implementation

• As a user, I'd love to be able to see the PVLive sum of GSPs when looking at national, and maybe see both PVLive options and our predictions on the same graph on the forecast tab. Maybe it could be done in the same kind of way that we select the models ncluded, which defaults just to the cnn?
• If possible, it would be great to optionally show the the MAE metrics computed with respect to the PVLive sum of GSPs. Maybe just a toggle which to compare to?

Add graphs with forecast horizons on x-axis

Detailed Description

Based on feedback in the UI/UX meeting, it would be good to have a couple graphs that manipulate the MAE by forecast horizon data.

One graph should plot the forecast horizons on the x-axis and showing one line per day in the date range in the chart window.

Another graph show show a new “average line”, where you average the forecast horizon MAE across the date range. For example, the first point would be the average MAE for the 60-minute forecast horizon values for the date range selected.

Context

This will give an idea of the MAE by forecast horizon across a range of days.

Possible Implementation

Use the plotly library to make this chart.

Update analysis dashboard readme

Detailed Description

The analysis dashboard is evolving, it'd be good to update at least the name on the readme and explain what the dashboard does.

• Update title from OCF internal ui to UK Analysis Dashboard
• Add a description of some of the statistics provided by the dashboard
• Add a link to the notion documents

Context

Good to keep documentation up to date.

Possible Implementation

Make changes to the Readme document

adding security scopes to dashboard

Detailed Description

This issue is the beginning of a discussion of how to add authorization to the dashboard.

• Discuss how to add auth0 scopes to the app
• Add warning to if we are adding a non ocf user to the ocf group.
• Spike - find popup component to use in app to double check user agrees to an action (ie. update, delete, etc.)
• Currently users who might add users or add sites are Peter, Brad, Sol, Zak, and Rachel

Context

The main users of the ocf dashboard will be internal, but it's good to think about who will have permission to do what. If we're deleting a site, it would be good that only certain people have that authorization.

Add write status

Detailed Description

Add an method that adds a status to the database

Context

• currently we write them in manually

Possible Implementation

• make a new status table
• show current status
• add a input text box that adds a status to the end point. It should should create a 'status' object, which has a status ('ok', 'warning' e.t.c) and a message

create a `user`, `site` and `site group`

Detailed Description

Add three functions to

• create user
• create site
• create site group

Context

• easy way to make users, sites and site group

possible implentation

• show example json data
• let user change the text
• press submit

Add export button

TODO

Cnn / National_xg

Detailed Description

Would be great to be able to load cnn and/or National_xg

Context

• good to compare two national models

Possible Implementation

• ....

Show MAE for the nowcasting app National forecast

refactor sites toolbox file

Detailed Description

The sitestoolbox.py file is rather long.

A separate file should be created for this feature with a couple files for the functions currently at the top of the sitestoolbox.py file.

Turn on/off option

• Option to turn Probablistic on and off. With a tick box
• Option to turn off Sum of GSPs

Add API user page

Detailed Description

Add a page where we can select

• a user
• start and end date

Then see a timeline of what routes they hit in the API

Context

Great to have clarity on what the users are doing

• see request table here and here

Possible Implementation

• load the user api requests from the tables,
• plot a time line of the different requests
• colour the request by the different route used

Bug: not distinct when using `get_gsp_yield_sum`

Need to load distinct on location as other we get some double counting, if there are multiple values in the database

See first part of plot

Need to adapt get_gsp_yield_sum and distinct on GSPYieldSQL.datetime_utc and location.gsp_id. This might be a bit tricky and involve a subquery first

refactor, use nowcasting_datamodel

Detailed Description

use pvsite_datamodel read function. This will reduce the code, and allow other components to use the various read functions

bug: Showing PVLive Sum twice

should only show the orange lines once

Pvnet by default

Detailed Description

Could we show PVnet with the adjuster on by default

Context

• CNN (or PVnet 1) is not being used any more

Possible Implementation

• change it here and here

Forecast page "Creation time" and "Forecast horizon" bug

The streamlit app running on dev is erroring out on the forecast page when the selected forecast type is "Creation time" or "Forecast horizon". Error message is:

File "/usr/local/lib/python3.9/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 565, in _run_script
    exec(code, module.__dict__)
File "/app/main.py", line 455, in <module>
    page_names_to_funcs[demo_name]()
File "/app/forecast.py", line 245, in forecast_page
    raise e
File "/app/forecast.py", line 215, in forecast_page
    properties_0 = forecast[0]._properties

It was working previously, and it still works on production. I'm not sure if it could be to do with PVNet2 being updated to be probabilistic on dev?

second column show be rmse

delete a `sitegroup`

attach a `sitegroup` to a user

Load both PVnet and National_xg

TODO

[meta] issue Data to display with Internal UI

This is a meta list of the data that should be included in the team Internal UI Streamlit app.

• #2

@dantravers, feel free to add a description of the data you'd like the Internal UI to display. I've converted the task above to an issue, but feel free to just add bullet points here with a description, and I can convert each point to an issue later.

Build docker container and EB app to deploy internal ui

Detailed Description

I would like to deploy the internal ui using docker and elastic beanstalk.

This article might be helpful with that.

And this article goes into using docker and EB to deploy a Streamlit app.

I've never used docker or created an EB app before, so will need to figure out the following to begin with:

1. a little bit more about what building a docker container entails
2. how to create the Dockerfile itself and build the docker image/ container.
3. identify if I need two containers for this project and possibly two beanstalk apps, one for the data deployed from the database and one for the streamlit code

Here is an example Dockerfile for a Streamlit app that Peter gave me to start.

Context

The internal ui runs locally but needs to be deployed.

Possible Implementation

Set up the appropriate docker and EB infrastructure to deploy the app so the team can access it.

Rename `cnn` to `PVnet 1.0`

Need to rename

• cnn to PVnet 1.0
• pvnet_v2 to PVnet 2.0

This will help make it clearer

select a `user` and show details, like what `sitegroup` they are in

Add password

Detailed Description

Good to add a password to the streamlit app

Context

• keep things safe, slightly sensitive info

Possible Implementation

• make password a github secret
• write password in Onepassword
• https://discuss.streamlit.io/t/how-to-set-login-password-for-my-streamlit-app/13150/5
• Future could use Auth0

select a `sitegroup` and show what `users` are attached, and what `sites` they have attached

UI suggestions for sites toolbox

Detailed Description

These are ideas for tidying up the sites toolbox UI

• In a user's site list, show the client_site_id before site_uuid
• Change label for date on site information. This refers to created_utc, which is in the datamodel => date_added_to_ocf_db would be better
• Grouping of functions => organize functions so they're still modular but perhaps grouped together by color or style
• Add some documentation of users and site groups, how multiple relationships work, link to github

Context

Sites Toolbox UI on the OCF dashboard could be improved by these changes to look a bit tidier and be easier to use.

Possible Implementation

Make changes to the UI in the scr/sites_toolbox.py file.

select a `site` by uuid, and see the details

Add tests

Detailed Description

Add tests that check data can be ready and put into plots

• unittests for each plotting function
• unittest to check app starts and runs

Context

• good to write tests

Possible Implementation

• mock database connection, see example here - TODO
• write tests to check plots can be made (might need to refactor so testing is easier)

site forecast page

Detailed Description

Add a page that can see the site forecasts

Context

• good to be able to select one or mulitle sites and see the forecast
• similar issue to forecast page for gsp and national

Possible Implementation

• drop down menu to select a site
• options on start and end date
• see truths (if there are any)
• see forecasts

Pvnet gsp sum: colour

Describe the bug

Cant load forecast for pvnet_gsp_sum as there is no colour assicoated it with

To Reproduce

Steps to reproduce the behavior:

1. Go to forecast page
2. load pvnet_gsp_sum

Expected behavior

For the forecast to show up

Additional context

add model to colour dict

Metrics page is getting a bit busy

Detailed Description

Metrics page is getting a bit busy, so can we tidy it up
this could be done by having some options to turn on/off graphs

Context

• nice to keep UI tidy
• recent added #73

Possible Implementation

• add check box for each plot in side panel, than can turn them on and off

Add PVlive intra day vs day-after

Detailed Description

Would be good to get the metrics for PVlive intraday

Context

Good to compare PVlive intra day and PVlive day -after

Possible Implementation

Could go on the 'Quartz Solar MAE with RMSE for Comparision`

MAE only until 15th MAY

Metrics only show up to 16th of May,

need to look what happened, of what has changed.

Useful things to do

• check metrics task is not falling over
• check what happen round 16th May, was there an upgrade? went from 1.0.4 to 1.0.5 which added the adjuster metrics
• Get SQL query that the page users