About timesteps about lstm-load-forecasting

[dafrie]

Hi Yishiheng,
Interesting question! Upfront notice: Since a few years have passed since, take everything with a grain of salt...

You are right that with the timesteps=1 and otherwise the default configuration, the network does not have knowledge about previous data and the whole point of RNN/LSTM would be moot.
However, since the stateful=True option is always set in the models here, the state of the LSTM cell is propagated between batches. As I understood it, this should theoretically allow the network to learn (strong) patterns very far apart without having to explicitly specify the timesteps. References: http://philipperemy.github.io/keras-stateful-lstm/

Please correct me if I am wrong and also would be very interesting to see, whether a different specification with timesteps>1 would improve the predictions!

[Yisiheng]

@dafrie emmmmm,
I think that if stateful=True, the operation does cause the propagation of state of the LSTM cell from the batch to next batch. But ,usually, when we use new batch to train the model. we will reset the state of the LSTM cell. I think that the reason why we reset the state is because the parameters that are trained don't contain the state of LSTM cell. when we use the model to forecast the load, initial state should be zero state (or other default initial state) rather than legacy state in training period.
Excuse me,I don't know if my expression is clear or not.

[dafrie]

Hm I understand your point, but let me quickly summarise how I saw it (and coded it...):

• Generate a multivariate dataset for the whole period

• Train the network with stateful=true and a batch size of 1 (e.g. one point in time)
  --> Do not call reset_state() between each batches, since otherwise the whole LSTM approach here wouldn't make much sense. Reason why not a different input shape (with lagged data for example): I did not want to explicitly model this and I understood that the LSTM cell state could help achieve this long term memory effect I wanted for modelling long-term patterns.

• Run training for multiple epochs, before each epoch actually calling reset_state()

• For testing/predictions, actually call reset_state() before prediction and feed in the multivariate data (except the actual load) and let it:

  • Create a forecast for the whole test period at once
  • Create a rolling forecast by retraining the model after each day (24 hours, simulating a real world scenario) but not resetting state each time.

Does that make sense? Since this was only a "pet" project for me and I unfortunately did not have further contact with LSTM, I am quite sure that the approach/architecture of this repository could be done much smarter... Generally I would be really interested to see a comparison with, for example a timesteps>1 and with stateful=False...