openclimatefix / uk-pv-forecast-blend

Service to blend forecast together

UK PV Forecast Blend

This app is for blending forecast together

The main application (forecast_blend/app.py)

1. Loads different forecasts
2. Blends them together using weights
3. Saves them ready to be used by the api
4. This is done for all GSPs

This was previously done in the API

We always update the ForecastValueLatest table, but we only update the ForecastValue table every 30 minutes

Details

• Note we only blend forecasts if they are made within 2 hours. If all forecasts are older than this, then all forecasts are used.
• The probabilistic forecasts are now blended using the same method as the expected value

  graph TD;
      subgraph App
      S(Start) --> LF
      LF(Load All Forecasts);
      LF --> Filter(Filter Forecasts);
      Filter -->  Blend[Blend \n Expected values];
      Blend --> N4(Blend Probabilistic \n- only for National);
      N4 --> S(Save Forecast)
      S --> F(Finish)
      end
      
      subgraph Blending
    A(All Forecasts);
    W2(Weights);
    A --> SUT(Split unique 'target times' \n and not);
    SUT --> |Unique Target times| B
    SUT --> |Duplicated Target times| C(Loop over each \n target time)
    C --> BW
    W2 --> BW
    BW(Blend using weights) --> SumCheck
    SumCheck{At least one \n forecast available} --> |yes| B
    SumCheck --> |no| Blend2(Blend forecast with \n next set of weights)
    Blend2 --> SumCheck2
    SumCheck2{At least one \n forecast available} --> |yes| B
    SumCheck2 --> |no| BB(Break)
    B(Blended \n Forecast)
    end

Tests

Tests are in the tests folder and can be run using pytest

Might need to

export PYTHONPATH=${PYTHONPATH}:./forecast_blend

uk-pv-forecast-blend's Issues

Diagram

Detailed Description

It will be useful to create daigram how foreasts get blended together

Context

• good for visibility

Possible Implementation

mermaid diagram, focusing on https://github.com/openclimatefix/uk-pv-forecast-blend/blob/main/forecast_blend/utils.py#L139

Save blended metadata

Detailed Description

Would be great to save what models and weights are used for each forecast

Context

• good to know where the values come from

Possible Implementation

1. add to 'plevel' column in nowcasting_datamodel
2. Get forecasts to save to both plevel and properties.
3. Update blend service to use plevel column, but save to both
4. Get forecasts to only save to plevel
5. Update api to read plevel column
6. Update blend service to save the weights used for each model in the properties column

Odd looking forecasts - 2023-09-05

Odd loading of weights

Looks like it struggles to load pure PVnet_v2 weights, so then loads a mixture of pvnet_v2 and national_xg

This related to #8

odd results 2023-09-04

Maybe becasue of EUMESAT data outage

bug: null values in propertoes

Detailed Description

Work out and try to fix why the properties blend ended up being

Context

• airflow was out of memory so the forecast may have not run
• target_time='2023-10-24 23:30:00+00'
• This error actually happened in the API, but was made by the blend service

Possible Implementation

if Null, make properties an empty dict

probablistic use pvnet from 0-8 hours

Detailed Description

from probablistic forecast use 0-8 hours

Context

• pvnet forecast probablsitic are better. than national_xg

Possible Implementation

• look at function https://github.com/openclimatefix/uk-pv-forecast-blend/blob/main/forecast_blend/blend.py#L124

Extract Blend into seperate component

Currently the API blends the 3 forecasts together.

It would be great if we have a separate component that doesnt this blending and saves the results.
The API can then just load this results.

This is good as it increase API speed, and also creates a history of the forecasts presented to clients

00.55 failure

Describe the bug

This bug seems to happen every day a 00.55

Expected behavior

not to happen

Additional context

Just set up ariflow-slack integration to make it clear when something fails