benjikcf / neural-net-with-financial-time-series-data

This solution presents an accessible, non-trivial example of machine learning (Deep learning) with financial time series using TensorFlow

Neural-Net-with-Financial-Time-Series-Data is an open source software project using endogenous factors to predict daily log return of financial asset. The project includes serveral technical indicators (ie. Stochastics, Moving Average Convergence/Divergence oscillator) to train a LSTM neural network by stochastic gradient descent with warm restart(SGDR) and cosine annealing. This flexible architecture enables you to deploy with Nvidia CuDNN computation without rewriting code by yourself. Hyperparameters are fine-tuned by Bayesian search.

Latest Result:

The current LSTM model result for predicting daily log return.

Old model Result

This old model uses LSTM to predict stock price.

LSTM cell

This is the LSTM cell we used in the model.

It is faster than normal LSTM cell because of the implementation of CuDNN LSTM and batch normalization in the model.

Update:

26/03/2017 First update

1. Recurrent neural network with LSTM is added to the code.
2. Keras with tensorflow is also implemented.
3. Tensorboard for neural network visualization is also added to the code.

14/04/2017 Second update

1. Normalized adjusted close price.
2. A new data downloader has been implemented for simplicity
3. Added more variable to predict the adjusted close price
4. More accurate result, significantly less mean square error
5. Extra visualization for close price
6. Denormalization will be fixed soon
7. Twitter sentiment analysis is currently on testing stage

16/04/2017 Third update

1. Updated denormalization
2. More test results available

18/04/2017 Fourth update

1. Updated fundamental data from Kaggle for NYSE

19/04/2017 Fifth update

1. Supporting Python 3.5 on Windows 10
2. Significant improvement in accuracy

29/04/2017 Sixth update

1. ^GSPC Data since 1970 has been added, more training data, higher accuracy
2. 7 years of test data
3. Object oriented programming
4. Hyperparameters for dropout has been tested

08/05/2017 Seventh update

1. All Hyperparameters have been tested and results have been uploaded.
2. Fixed comment for the data loader
3. More technical analysis like volume, moving average and other indexes will be added

28/05/2017 Eighth update

1. Using Quandl instead of Pandas datareader
2. Correlation heatmap has been addded
3. Using Adjusted OHLCV for the network
4. All functions can be loaded from lstmstock.py
5. A Quandl api key is provided temporarily for those who do not own a quandl account
6. Moving averages have been added

02/10/2017 Nineth update

1. Event driven analysis
2. Switched to Tensorflow LSTM model

25/06/2018 Tenth update

1. Complete rewrite of News downloader, removed Newsapi in order to get full access to NYTImes data for free
2. Moving Average Convergence/Divergence oscillator (MACD), Stochastic Oscillator, Average True Range are added to train the model.
3. log return is now used as target variable.
4. Keras on top of Tensorflow is used.
5. Randomized Search from SKLearn is used for optimization.

11/10/2018 Eleventh update Serveral state of the art techniques are applied

1. CuDNN LSTM is used to accelerate training
2. Stochastic gradient descent with warm restart
3. Cosine annealing
4. Use Bayesian search to optmize hyperparameters.
5. New splitting method
6. Dataset is provided
7. HDF files are used to accelerate reading time

How to use Quandl

If you want to train your neural network on more data, feel free to load the data directly from Quandl. You should be able to get the historic price data of a particular stock after login. Use Export > Python > api key and insert the api key to your model. https://www.quandl.com/databases/WIKIP

References:

Bernal, A., Fok, S., & Pidaparthi, R. (2012). Financial Market Time Series Prediction with Recurrent Neural Networks.

Box, G. E., Jenkins, G. M., Reinsel, G. C., & Ljung, G. M. (2015). Time series analysis: forecasting and control. John Wiley & Sons.

Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., ... & Cai, J. (2015). Recent advances in convolutional neural networks. arXiv preprint arXiv:1512.07108.

Hutto, C.J. & Gilbert, E.E. (2014). VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text. Eighth International Conference on Weblogs and Social Media (ICWSM-14). Ann Arbor, MI, June 2014.

Jaeger, H. (2001). The “echo state” approach to analysing and training recurrent neural networks-with an erratum note. Bonn, Germany: German National Research Center for Information Technology GMD Technical Report, 148(34), 13.

Jaeger, H. (2002). Tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the" echo state network" approach (Vol. 5). GMD-Forschungszentrum Informationstechnik.

Maass, W., Natschläger, T., & Markram, H. (2002). Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural computation, 14(11), 2531-2560.

neural-net-with-financial-time-series-data's Issues

Use return in finance modeling, not price!

I think your model is not better than just use last day price to predict.
Use return in finance modeling, not price!

import datetime
import pandas as pd
import pandas_datareader.data as web
%matplotlib inline

stock_name = '^GSPC'

start = datetime.datetime(1950, 1, 1)
end = datetime.date.today()
df = web.DataReader(stock_name, "yahoo", start, end)

y = df['Adj Close']
prediction = df['Adj Close'].shift(-1)
pd.DataFrame({'y': y, 'prediction': prediction})[15476:].plot() 

What is the meaning of the data?

Hello, I am very glad that you provided the data, but I don't understand what the meaning of the data is. Can you add the documentation for the data?

Accelerated Learning, very low testing scores

Though not an issue I thought i would Share this with the community

by changing NeroNet Paramaters to:
seq_len = 2
d = 100000
shape = [7, seq_len, 1]
neurons = [168, 168, 64, 1]
epochs = 20
BS =512
Vs =0.01
(d = dropout Rate)
(shape is different as my dataset includes Open, High, Low, Volume,Volume_Change(from previous Day),High/Low Change, Open/Close Change, Close)

(neurons = changed to increase accuracy with minimal performance hinderence)
(epochs = changed from 90 to 20 due to platu effect at epoch 18 for a majority of stocks within NSYE,and NASDAQ)
(BS = Batch_Size. I chose 512 as it has the quickest response time, and itterations through the batches, same as POSTED Coding. No Need to change)
(VS = Validation_Split = chose 0.01 over .1 because of MSE Drop Rate)

a
Train Score: 0.000134 MSE (0.012 RMSE)
Test Score: 0.000056 MSE (0.007 RMSE)
8.75045371055603 Seconds
Please enter Tiker: goog
goog
Train Score: 0.008935 MSE (0.095 RMSE)
Test Score: 0.027300 MSE (0.165 RMSE)
4.698930501937866 Seconds
Please enter Tiker: aa
aa
Train Score: 0.000075 MSE (0.009 RMSE)
Test Score: 0.000060 MSE (0.008 RMSE)
17.19175362586975 Seconds
Please enter Tiker: ibm
ibm
Train Score: 0.000030 MSE (0.005 RMSE)
Test Score: 0.000098 MSE (0.010 RMSE)
20.26670789718628 Seconds

As you can see the time showed is the time it took to retrieve Data, build the model, Test the model, and score the model.

Also During plotting of prediction to actual price, roll newy_test to align the charts. I found rolling values forward one day, aligned them pretty well. Thus meaning to predict the next days, you must have sequence length set to 2.

# IMPORTANT #
#newy_test is roll forward, because values are close to correct values, to align Gains and Drops.
#Without this prediction would be one Day behind
newy_test = np.roll(newy_test,1,0)

This way your Rise, Drop point will mainly line up. Yes that means the last value of newy_test is relevant and could be dropped all together. However My Buy/Sell Flag indicator, notifies the day in advance of Drop or Rise. Skipping the last entry all together.

y_train = train[:, -1][:,-1]

Hi Benji!

Your code is amazing - well done with all your research up til now!

Please could you answer one question for me?

The line below returns an error: IndexError: too many indices for array:

y_train = train[:, -1][:,-1] # day m + 1 adjusted close price

I'm confused because the first [:, -1] creates a 1D array so the error is caused when the second [:, -1] tries to make another 1D array of the original 1D array which doesn't work. Is this line correct?

Your reply is much appreciated!

Mark

Recurrent Neural Network.ipynb is not working properly

When I run your Recurrent Neural Network.ipynb, I get the following error:

InvalidArgumentError                      Traceback (most recent call last)
~/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
   1321     try:
-> 1322       return fn(*args)
   1323     except errors.OpError as e:

~/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tensorflow/python/client/session.py in _run_fn(feed_dict, fetch_list, target_list, options, run_metadata)
   1304       # Ensure any changes to the graph are reflected in the runtime.
-> 1305       self._extend_graph()
   1306       return self._call_tf_sessionrun(

~/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tensorflow/python/client/session.py in _extend_graph(self)
   1339       with self._graph._lock:  # pylint: disable=protected-access
-> 1340         tf_session.ExtendSession(self._session)
   1341     else:

InvalidArgumentError: No OpKernel was registered to support Op 'CudnnRNN' with these attrs.  Registered devices: [CPU], Registered kernels:
  <no registered kernels>

	 [[Node: cu_dnnlstm/CudnnRNN = CudnnRNN[T=DT_FLOAT, direction="unidirectional", dropout=0, input_mode="linear_input", is_training=true, rnn_mode="lstm", seed=0, seed2=0](cu_dnnlstm/transpose, cu_dnnlstm/ExpandDims_1, cu_dnnlstm/ExpandDims_2, cu_dnnlstm/concat)]]

During handling of the above exception, another exception occurred:

InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-14-29326f3cbcc3> in <module>
      8                      steps_per_epoch=np.ceil(epoch_size/batch_size),
      9                      epochs=3)
---> 10 model.fit(X_train, y_train, callbacks=[lr_finder])
     11 
     12 lr_finder.plot_loss()

~/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tensorflow/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, **kwargs)
   1346           initial_epoch=initial_epoch,
   1347           steps_per_epoch=steps_per_epoch,
-> 1348           validation_steps=validation_steps)
   1349 
   1350   def evaluate(self,

~/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tensorflow/python/keras/engine/training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, callback_metrics, initial_epoch, steps_per_epoch, validation_steps)
    166       'metrics': callback_metrics or [],
    167   })
--> 168   callbacks.on_train_begin()
    169   callback_model.stop_training = False
    170   for cbk in callbacks:

~/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tensorflow/python/keras/callbacks.py in on_train_begin(self, logs)
    145     logs = logs or {}
    146     for callback in self.callbacks:
--> 147       callback.on_train_begin(logs)
    148 
    149   def on_train_end(self, logs=None):

<ipython-input-10-98122f67f6ac> in on_train_begin(self, logs)
     43         '''Initialize the learning rate to the minimum value at the start of training.'''
     44         logs = logs or {}
---> 45         K.set_value(self.model.optimizer.lr, self.min_lr)
     46 
     47     def on_batch_end(self, epoch, logs=None):

~/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tensorflow/python/keras/backend.py in set_value(x, value)
   2682       x._assign_placeholder = assign_placeholder
   2683       x._assign_op = assign_op
-> 2684     get_session().run(assign_op, feed_dict={assign_placeholder: value})
   2685 
   2686 

~/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tensorflow/python/keras/backend.py in get_session()
    441   if not _MANUAL_VAR_INIT:
    442     with session.graph.as_default():
--> 443       _initialize_variables(session)
    444   return session
    445 

~/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tensorflow/python/keras/backend.py in _initialize_variables(session)
    665     # marked as initialized.
    666     is_initialized = session.run(
--> 667         [variables_module.is_variable_initialized(v) for v in candidate_vars])
    668     uninitialized_vars = []
    669     for flag, v in zip(is_initialized, candidate_vars):

~/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
    898     try:
    899       result = self._run(None, fetches, feed_dict, options_ptr,
--> 900                          run_metadata_ptr)
    901       if run_metadata:
    902         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

~/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
   1133     if final_fetches or final_targets or (handle and feed_dict_tensor):
   1134       results = self._do_run(handle, final_targets, final_fetches,
-> 1135                              feed_dict_tensor, options, run_metadata)
   1136     else:
   1137       results = []

~/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1314     if handle is None:
   1315       return self._do_call(_run_fn, feeds, fetches, targets, options,
-> 1316                            run_metadata)
   1317     else:
   1318       return self._do_call(_prun_fn, handle, feeds, fetches)

~/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
   1333         except KeyError:
   1334           pass
-> 1335       raise type(e)(node_def, op, message)
   1336 
   1337   def _extend_graph(self):

InvalidArgumentError: No OpKernel was registered to support Op 'CudnnRNN' with these attrs.  Registered devices: [CPU], Registered kernels:
  <no registered kernels>

	 [[Node: cu_dnnlstm/CudnnRNN = CudnnRNN[T=DT_FLOAT, direction="unidirectional", dropout=0, input_mode="linear_input", is_training=true, rnn_mode="lstm", seed=0, seed2=0](cu_dnnlstm/transpose, cu_dnnlstm/ExpandDims_1, cu_dnnlstm/ExpandDims_2, cu_dnnlstm/concat)]]

Caused by op 'cu_dnnlstm/CudnnRNN', defined at:
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/ipykernel_launcher.py", line 16, in <module>
    app.launch_new_instance()
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/ipykernel/kernelapp.py", line 505, in start
    self.io_loop.start()
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tornado/platform/asyncio.py", line 132, in start
    self.asyncio_loop.run_forever()
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/asyncio/base_events.py", line 301, in run_forever
    self._run_once()
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/asyncio/base_events.py", line 1198, in _run_once
    handle._run()
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/asyncio/events.py", line 125, in _run
    self._callback(*self._args)
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tornado/ioloop.py", line 758, in _run_callback
    ret = callback()
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tornado/stack_context.py", line 300, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tornado/gen.py", line 1233, in inner
    self.run()
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tornado/gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/ipykernel/kernelbase.py", line 357, in process_one
    yield gen.maybe_future(dispatch(*args))
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tornado/gen.py", line 326, in wrapper
    yielded = next(result)
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/ipykernel/kernelbase.py", line 267, in dispatch_shell
    yield gen.maybe_future(handler(stream, idents, msg))
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tornado/gen.py", line 326, in wrapper
    yielded = next(result)
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/ipykernel/kernelbase.py", line 534, in execute_request
    user_expressions, allow_stdin,
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tornado/gen.py", line 326, in wrapper
    yielded = next(result)
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/ipykernel/ipkernel.py", line 294, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/ipykernel/zmqshell.py", line 536, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 2819, in run_cell
    raw_cell, store_history, silent, shell_futures)
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 2845, in _run_cell
    return runner(coro)
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/IPython/core/async_helpers.py", line 67, in _pseudo_sync_runner
    coro.send(None)
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 3020, in run_cell_async
    interactivity=interactivity, compiler=compiler, result=result)
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 3185, in run_ast_nodes
    if (yield from self.run_code(code, result)):
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 3267, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-12-671884cecb64>", line 1, in <module>
    model = build_model()
  File "<ipython-input-11-a59f75026d23>", line 3, in build_model
    model.add(CuDNNLSTM(256, input_shape=(X_train.shape[1:]), return_sequences=True))
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tensorflow/python/keras/engine/sequential.py", line 163, in add
    layer(x)
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tensorflow/python/keras/layers/recurrent.py", line 527, in __call__
    return super(RNN, self).__call__(inputs, **kwargs)
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tensorflow/python/keras/engine/base_layer.py", line 703, in __call__
    outputs = self.call(inputs, *args, **kwargs)
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tensorflow/python/keras/layers/cudnn_recurrent.py", line 109, in call
    output, states = self._process_batch(inputs, initial_state)
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tensorflow/python/keras/layers/cudnn_recurrent.py", line 493, in _process_batch
    is_training=True)
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tensorflow/python/ops/gen_cudnn_rnn_ops.py", line 115, in cudnn_rnn
    is_training=is_training, name=name)
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3414, in create_op
    op_def=op_def)
  File "/home/zh/anaconda3/envs/AdaNet/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1740, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): No OpKernel was registered to support Op 'CudnnRNN' with these attrs.  Registered devices: [CPU], Registered kernels:
  <no registered kernels>

	 [[Node: cu_dnnlstm/CudnnRNN = CudnnRNN[T=DT_FLOAT, direction="unidirectional", dropout=0, input_mode="linear_input", is_training=true, rnn_mode="lstm", seed=0, seed2=0](cu_dnnlstm/transpose, cu_dnnlstm/ExpandDims_1, cu_dnnlstm/ExpandDims_2, cu_dnnlstm/concat)]]

My environment is:
Tensorflow==1.9
Keras==2.2.4

@BenjiKCF
What is the cause of this? What should I do? There is no environmental requirement in your description.

multiple steps forecast

Hi Benji!

your code is very inspiring and i tried to develop a model that foresees two time-steps forward

In this regard I modified the data_load function so that it takes n steps backward in the X_train/test series and set it against a y_train/test 2-array.

I modified the neurons list to output in Dense a 2-vector object.

And last I wrote a predict function and a plot function for the 2-step-forecast.

I do not normalized features lables and forecasts I will do in the future.

After a bit of hyperfine tuning it returns a good score for the mse and rmse:
Train Score: 0.00000 MSE (0.00 RMSE)
Test Score: 0.00153 MSE (0.04 RMSE)

It can find quite well the trend, but it returns all forecasts with negative directions...
Do you (or anyone) have any suggestions?

import seaborn as sns
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.pyplot as plt2
import pandas as pd
from pandas import datetime
import math, time
import itertools
from sklearn import preprocessing
import datetime
from sklearn.metrics import mean_squared_error
from math import sqrt
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation
from keras.layers.recurrent import LSTM
from keras.models import load_model
import keras
from numpy import newaxis
pd.core.common.is_list_like = pd.api.types.is_list_like
import pandas_datareader.data as web
import h5py
from keras import backend as K
import quandl

quandl.ApiConfig.api_key = 'rthmY7Sfx7XW31YRrcpm'
seq_len = 2
shape = [seq_len, 9, 2]
neurons = [256, 256, 64, 2]
dropout = 0.2
decay = 0.5
epochs = 100
stock_name = 'AAPL'
global_start_time = time.time()

def get_stock_data(stock_name, normalize=True, ma=[]):
    """
    Return a dataframe of that stock and normalize all the values. 
    (Optional: create moving average)
    """
    df = quandl.get_table('WIKI/PRICES', ticker = stock_name)
    df.drop(['ticker', 'open', 'high', 'low', 'close', 'ex-dividend', 'volume', 'split_ratio'], 1, inplace=True)
    df.set_index('date', inplace=True)
    
    # Renaming all the columns so that we can use the old version code
    df.rename(columns={'adj_open': 'Open', 'adj_high': 'High', 'adj_low': 'Low', 'adj_volume': 'Volume', 'adj_close': 'Adj Close'}, inplace=True)
    
    # Percentage change
    df['Pct'] = df['Adj Close'].pct_change()
    df.dropna(inplace=True)
    
    # Moving Average    
    if ma != []:
        for moving in ma:
            df['{}ma'.format(moving)] = df['Adj Close'].rolling(window=moving).mean()
    df.dropna(inplace=True)
    
    if normalize:
        min_max_scaler = preprocessing.MinMaxScaler()
        df['Open'] = min_max_scaler.fit_transform(df.Open.values.reshape(-1,1))
        df['High'] = min_max_scaler.fit_transform(df.High.values.reshape(-1,1))
        df['Low'] = min_max_scaler.fit_transform(df.Low.values.reshape(-1,1))
        df['Volume'] = min_max_scaler.fit_transform(df.Volume.values.reshape(-1,1))
        df['Adj Close'] = min_max_scaler.fit_transform(df['Adj Close'].values.reshape(-1,1))
        df['Pct'] = min_max_scaler.fit_transform(df['Pct'].values.reshape(-1,1))
        if ma != []:
            for moving in ma:
                df['{}ma'.format(moving)] = min_max_scaler.fit_transform(df['{}ma'.format(moving)].values.reshape(-1,1))  
                
    # Move Adj Close to the rightmost for the ease of training
    adj_close = df['Adj Close']
    df.drop(labels=['Adj Close'], axis=1, inplace=True)
    df = pd.concat([df, adj_close], axis=1)

    #df.to_csv('aap.csv')  
    return df

df = get_stock_data(stock_name, ma=[50, 100, 200])

def plot_stock(df):
    print(df.head())
    plt.subplot(211)
    plt.plot(df['Adj Close'], color='red', label='Adj Close')
    plt.legend(loc='best')
    plt.subplot(212)
    plt.plot(df['Pct'], color='blue', label='Percentage change')
    plt.legend(loc='best')
    plt.show()

#plot_stock(df)

def load_data(stock, seq_len):
    amount_of_features = len(stock.columns)
    data = stock.values
    sequence_length = seq_len + 2 # index starting from 0
    result = []

    for index in range(len(data) - sequence_length): # maxmimum date = lastest date - sequence length
        result.append(data[index: index + sequence_length]) # index : index + 22days

    result = np.array(result)
    row = round(0.8 * result.shape[0]) # 80% split

    train = result[:int(row), :,:] # 80% date
    X_train = train[:, :-2,:] # all data until day m
    y_train = train[:, -2:,:][:,:,-1] # day m + 1 adjusted close price

    X_test = result[int(row):, :-2,:]
    y_test = result[int(row):, -2:,:][:,:,-1]

    X_train = np.reshape(X_train, (X_train.shape[0], X_train.shape[1], amount_of_features))
    X_test = np.reshape(X_test, (X_test.shape[0], X_test.shape[1], amount_of_features))
    
    print ("________________________________________________________________")
    print ("X_train shape = {}".format(X_train.shape))
    print ("y_train shape = {}".format(y_train.shape))
    print ("")
    print ("X_test shape = {}".format(X_test.shape))
    print ("y_test shape = {}".format(y_test.shape))
    print ("________________________________________________________________")

    return [X_train, y_train, X_test, y_test]

X_train, y_train, X_test, y_test = load_data(df, seq_len)

def build_model(shape, neurons, dropout, decay):
    model = Sequential()

    model.add(LSTM(neurons[0], input_shape=(shape[0], shape[1]), return_sequences=True))
    model.add(Dropout(dropout))

    model.add(LSTM(neurons[1], input_shape=(shape[0], shape[1]), return_sequences=False))
    model.add(Dropout(dropout))

    model.add(Dense(neurons[2],kernel_initializer="uniform",activation='relu'))
    model.add(Dense(neurons[3],kernel_initializer="uniform",activation='linear'))
    # model = load_model('my_LSTM_stock_model1000.h5')
    adam = keras.optimizers.Adam(decay=decay)
    model.compile(loss='mse',optimizer='adam', metrics=['accuracy'])
    model.summary()
    return model

model = build_model(shape, neurons, dropout, decay)

model.fit(
    X_train,
    y_train,
    batch_size=512,
    epochs=epochs,
    validation_split=0.01,
    verbose=1)

def model_score(model, X_train, y_train, X_test, y_test):
    trainScore = model.evaluate(X_train, y_train, verbose=0)
    print('Train Score: %.5f MSE (%.2f RMSE)' % (trainScore[0], math.sqrt(trainScore[0])))

    testScore = model.evaluate(X_test, y_test, verbose=0)
    print('Test Score: %.5f MSE (%.2f RMSE)' % (testScore[0], math.sqrt(testScore[0])))
    return trainScore[0], testScore[0]

model_score(model, X_train, y_train, X_test, y_test)

def percentage_difference(model, X_test, y_test):
    percentage_diff=[]

    p = model.predict(X_test)
    for u in range(len(y_test)): # for each data index in test data
        pr = p[u][0] # pr = prediction on day u

        percentage_diff.append((pr-y_test[u]/pr)*100)
    print('Prediction duration: ', str(datetime.timedelta(seconds=(time.time() - global_start_time))) )
    return p

def denormalize(stock_name, normalized_value):
    """
    Return a dataframe of that stock and normalize all the values. 
    (Optional: create moving average)
    """
    df = quandl.get_table('WIKI/PRICES', ticker = stock_name)
    df.drop(['ticker', 'open', 'high', 'low', 'close', 'ex-dividend', 'volume', 'split_ratio'], 1, inplace=True)
    df.set_index('date', inplace=True)
    
    # Renaming all the columns so that we can use the old version code
    df.rename(columns={'adj_open': 'Open', 'adj_high': 'High', 'adj_low': 'Low', 'adj_volume': 'Volume', 'adj_close': 'Adj Close'}, inplace=True)
    
    df.dropna(inplace=True)
    df = df['Adj Close'].values.reshape(-1,1)
    normalized_value = normalized_value.reshape(-1,1)

    #return df.shape, p.shape
    min_max_scaler = preprocessing.MinMaxScaler()
    a = min_max_scaler.fit_transform(df)
    new = min_max_scaler.inverse_transform(normalized_value)
      
    return new

def plot_result(stock_name, normalized_value_p, normalized_value_y_test):
    newp = denormalize(stock_name, normalized_value_p)
    newy_test = denormalize(stock_name, normalized_value_y_test)
    #newy_test = np.roll(newy_test,1,0)
    plt2.plot(newp, color='red', label='Prediction')
    plt2.plot(newy_test,color='blue', label='Actual')
    plt2.legend(loc='best')
    plt2.title('Global run time {}'.format(str(datetime.timedelta(seconds=(time.time() - global_start_time))) ) )
    plt2.xlabel('Days')
    plt2.ylabel('Adjusted Close')	
    plt2.show()

def predict_sequences_multiple(model, data, window_size, prediction_len):
    #Predict sequence of 50 steps before shifting prediction run forward by 50 steps
    prediction_seqs = []
    for i in range(int(len(data)/prediction_len)):
        curr_frame = data[i*prediction_len]
        predicted = []
        for j in range(prediction_len):
            predicted.append(model.predict(curr_frame[newaxis,:,:])[0,0])
            curr_frame = curr_frame[1:]
            curr_frame = np.insert(curr_frame, [window_size-1], predicted[-1], axis=0)
        prediction_seqs.append(predicted)
    
    print('Prediction duration: ', str(datetime.timedelta(seconds=(time.time() - global_start_time))) )
    return prediction_seqs

def plot_results_multiple(predicted_data, true_data, prediction_len):
    fig = plt.figure(facecolor='white')
    ax = fig.add_subplot(111)
    ax.plot(true_data, label='True Data')
    #Pad the list of predictions to shift it in the graph to it's correct start
    for i, data in enumerate(predicted_data):
        padding = [None for p in range(i * prediction_len)]
        plt.plot(padding + data, label='Prediction')
        plt.legend()
    plt.title('Global run time {}'.format(str(datetime.timedelta(seconds=(time.time() - global_start_time))) ) )
    plt.show()

#Single step prediction
#p = percentage_difference(model, X_test, y_test)
#plot_result(stock_name, p, y_test)

#Multiple step prediction
predictions = predict_sequences_multiple(model, X_test, seq_len, 2)
plot_results_multiple(predictions, y_test, 2)

One file in your Package called 'Old Version' has a serious problem

In this code, I dont know whether you thought it's wrong but forgot to delete it. I found that your X_train includes the target called 'adj_close' which is in y_train. So, in that code, you are not predicting it because the adj_close is for that day instead of the day after it, it's like a leakage. So I am confused and I want to hear your expaination if I am wrong. Thank you.

Link broken

Link in section "How to use Quandl" from README leads to 404.

What does Technical indicator contain?

Hello, when I was looking at your SP500 data preparation. ipynb tutorial, I saw three concepts: MACD (Trend), Stochastics (Momentum), Average True Range (Volume). What puzzles me is whether these three concepts are the methods you have come up with yourself or the ones commonly used in the financial field? Besides, I can't understand them very well. What are the meanings of the three concepts? Can you briefly summarize what they mean separately?

How to make predictions?

Hello,
How should I make predictions after I have finished training? What should my prediction be? If I want to predict future changes, what should my input be like? During the training, the training data has been calculated a lot, such as MACD, ART, etc., then my prediction data should be the same calculation? What should I predict?
Please guide me, thank you

Unable to implement your solution

Good day,

I am trying to implement your code. Python 3.6 64 bit environment. With keras, matplotlib, numpy, pandas, tensor flow installed onto the environment.

Getting the blow issue. The screen just stuck at "Using Tensorflow". After 5 minutes, I am able to press any key to continue however it does nothing. Do you have any suggestions? Thank you.

Thank you,
Skydrake123

Using Test Data to denormalize

Hello,

I am a little concerned about your methodology when it comes to denormalize, especially as you are using actual data to denormalize your predicted values. Of course these values will be very close to the predicted as you fit your denormalization function on the true values and then compare both.
Also, same when you normalize you use Test data which bias your scaling in the sense that for the training part you are not supposed to have the test data.

Can you please help me understand otherwise if I am missing something.

Thank you

have python format of codes

Hi,
I want to read all codes, so I do not understand how is the order of the files have been attached to run. Would you please help me?
And how can I have python format of codes to run in visual studio?

Download data! The URL Unable to read

Unable to read URL: http://ichart.finance.yahoo.com/table.csv?f=2017&e=3&a=0&s=%5EGSPC&d=5&c=1950&b=1&g=d&ignore=.csv

Using open price to predict the closing price?

Hello @BenjiKCF ,

Are you using open price (in x) to predict the closing price?

Data Preprocessing and Sentiment Analysis.ipynb code error

When I ran the file Data Preprocessing and Sentiment Analysis. ipynb you provided, I got an unexpected grammatical error:

What is the reason? How should I solve it?