spallas / time-attention

Implementation of RNN for Time Series prediction from the paper https://arxiv.org/abs/1704.02971

Hi there

• I’m currently working as a Software Engineer and SRE for Italy's Central Bank
• Previously worked on MLOps, time series forecasting and optimization algorithms for EnelX Noth America
• Contacts: X

time-attention's Issues

ERROR: Incompatible shapes: [32,17] vs. [31,17]

Hi，when I run train.py , there is some error. Can you help me.

(tensorflow-gpu) root@fudan:/anaconda3/envs/tensorflow-gpu/time-attention# (tensorflow-gpu) root@fudan:/anaconda3/envs/tensorflow-gpu/time-attention# python train.py
2019-09-10 20:00:16.350976: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-09-10 20:00:16.519047: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-09-10 20:00:16.519848: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212] Found device 0 with properties:
name: Tesla M40 major: 5 minor: 2 memoryClockRate(GHz): 1.112
pciBusID: 0000:00:08.0
totalMemory: 11.92GiB freeMemory: 11.81GiB
2019-09-10 20:00:16.519891: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1312] Adding visible gpu devices: 0
2019-09-10 20:00:16.837772: I tensorflow/core/common_runtime/gpu/gpu_device.cc:993] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11434 MB memory) -> physical GPU (device: 0, name: Tesla M40, pci bus id: 0000:00:08.0, compute capability: 5.2)
====================================== EPOCH 0 ======================================
Traceback (most recent call last):
File "/root/anaconda3/envs/tensorflow-gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1361, in _do_call
return fn(*args)
File "/root/anaconda3/envs/tensorflow-gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1340, in _run_fn
target_list, status, run_metadata)
File "/root/anaconda3/envs/tensorflow-gpu/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 516, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [32,17] vs. [31,17]

How to get attention weights

Thank you very much for your answer. I now know how to run your code. But I have an another question: how do I get input attention weights as the authors did in the original paper ( A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction), I tried but failed. Looking forward to your help.

How to run

hi, I have troubling running the code. How to run the code of the control experiments, such as how to produce result.csv and attn.csv, what commands should be used to implement them?
l am looking forward from you.

Decoder Inputs

Hi,

Great work, thank you. I wanted to ask you about decoder inputs because when I checked your data loader and decoder layer I saw:

def window(
    df: pd.DataFrame,
    size: int,
    driving_series: List[str],
    target_series: List[str],
):
    X = df[driving_series].values
    y = df[target_series].values
    X_T = []
    y_T = []
    for i in range(len(X) - size + 1):
        X_T.append(X[i : i + size])
        y_T.append(y[i : i + size])

    return np.array(X_T), np.array(y_T)

and

        with tf.variable_scope("DecoderRNN"):
            # TODO: check order of hidden states and cell states
            cell = LSTMCell(self.config.p, state_is_tuple=True)
            initial_state = cell.zero_state(self.config.batch_size, tf.float32)
            c_t = tf.get_variable("c_t", [self.config.batch_size, self.config.m])
            state = initial_state
            s_, d = state

            for t in range(self.config.T):
                # if t > 0: tf.get_variable_scope().reuse_variables()
                if self.config.temporal_att_enabled:
                    beta = self._attention(d, s_, encoder_outputs)

                    c_t = tf.reduce_sum(beta * encoder_outputs, axis=1)
                else:
                    c_t = encoder_outputs[:, t, :]

                if t < self.config.T - 1:
                    y_c = tf.concat([tf.expand_dims(past_history[:, t], -1), c_t], axis=1)
                    y_tilde = tf.layers.dense(y_c, 1)
                    (cell_output, state) = cell(y_tilde, state)
                    s_, d = state

            d_c = tf.concat([d, c_t], axis=1)
            y_T = tf.layers.dense(tf.layers.dense(d_c, self.config.p), 1)
            y_T = tf.squeeze(y_T)

        loss = tf.losses.mean_squared_error(y_T, past_history[:, - 1])
        return y_T, loss

Correct me if I am mistaken, but I feel like in these 2 functions you are using the same array for both target variable and decoder inputs then you are only feeding T-1 steps to decoder, shouldn't it be if t < self.config.T : instead of if t < self.config.T - 1: because I feel like when it is like that you are only feeding past 8 steps to decoder instead of past 9 steps as suggested in the paper?

Thank you

Why the first data point can be predicted?

I got the plot after running test.py with sml2010 dataset. It's wired that in training part, validation part and test part, the first several data can be predicted: if T=10, the first 10 values will be the input to predict the 11th datapoint, the prediction line should start from 11th data point.

Matplotlib warning too many images open

Currently the images created through train.py:plot during training are not handled properly producing the following warning:

RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory.