scorpionhiccup / stockpriceprediction

Stock Price Prediction using Machine Learning Techniques

Stock Market Price Predictor using Supervised Learning

Aim

To examine a number of different forecasting techniques to predict future stock returns based on past returns and numerical news indicators to construct a portfolio of multiple stocks in order to diversify the risk. We do this by applying supervised learning methods for stock price forecasting by interpreting the seemingly chaotic market data.

Setup Instructions

    $ workon myvirtualenv                                  [Optional]
    $ pip install -r requirements.txt
    $ python scripts/Algorithms/regression_models.py <input-dir> <output-dir>

Download the Dataset needed for running the code from here.

Project Concept Video

Methodology

1. Preprocessing and Cleaning
2. Feature Extraction
3. Twitter Sentiment Analysis and Score
4. Data Normalization
5. Analysis of various supervised learning methods
6. Conclusions

Research Paper

• Machine Learning in Stock Price Trend Forecasting. Yuqing Dai, Yuning Zhang
• Stock Market Forecasting Using Machine Learning Algorithms. Shunrong Shen, Haomiao Jiang. Department of Electrical Engineering. Stanford University
• How can machine learning help stock investment?, Xin Guo

Datasets used

1. http://www.nasdaq.com/
2. https://in.finance.yahoo.com
3. https://www.google.com/finance

Useful Links

• Slides: http://www.slideshare.net/SharvilKatariya/stock-price-trend-forecasting-using-supervised-learning
• Video: https://www.youtube.com/watch?v=z6U0OKGrhy0
• Report: https://github.com/scorpionhiccup/StockPricePrediction/blob/master/Report.pdf

References

• Recurrent Neural Networks - LSTM Models
• ARIMA Models
• https://github.com/dv-lebedev/google-quote-downloader
• Book Value
• http://www.investopedia.com/articles/basics/09/simplified-measuring-interpreting-volatility.asp
• Volatility
• https://github.com/dzitkowskik/StockPredictionRNN
• Scikit-Learn
• Theano

stockpriceprediction's Issues

clarity on input and output folders

Hi i downloaded the 2 folders that you have given, i have placed in root folder of the repo and for running the repo,

I gave python regressionmodels.py input output, here what are the input and output directories?
when i downloaded in 1 folder i got all csv files and other folder with images, shd i mention those csv files folder as my input folder if so what shd i mention for output folder, it is asking there is no output/source.pickle file?

File "scripts/Algorithms/regression_models.py", line 81, in main with open(os.path.join(output_dir, 'scores.pickle'), 'wb') as handle: FileNotFoundError: [Errno 2] No such file or directory: 'output/scores.pickle

please help and there are so so many files and folder, where to start from, dont have clear idea and
sklearn is updated a lot, please check that also.

New complementary tool

My name is Luis, I'm a big-data machine-learning developer, I'm a fan of your work, and I usually check your updates.

I was afraid that my savings would be eaten by inflation. I have created a powerful tool that based on past technical patterns (volatility, moving averages, statistics, trends, candlesticks, support and resistance, stock index indicators).
All the ones you know (RSI, MACD, STOCH, Bolinger Bands, SMA, DEMARK, Japanese candlesticks, ichimoku, fibonacci, williansR, balance of power, murrey math, etc) and more than 200 others.

The tool creates prediction models of correct trading points (buy signal and sell signal, every stock is good traded in time and direction).
For this I have used big data tools like pandas python, stock market libraries like: tablib, TAcharts ,pandas_ta... For data collection and calculation.
And powerful machine-learning libraries such as: Sklearn.RandomForest , Sklearn.GradientBoosting, XGBoost, Google TensorFlow and Google TensorFlow LSTM.

With the models trained with the selection of the best technical indicators, the tool is able to predict trading points (where to buy, where to sell) and send real-time alerts to Telegram or Mail. The points are calculated based on the learning of the correct trading points of the last 2 years (including the change to bear market after the rate hike).

I think it could be useful to you, to improve, I would like to share it with you, and if you are interested in improving and collaborating I am also willing, and if not file it in the box.

Value Error: Could not convert string to float

Hi,
This is a good project to learn from :)
In the file preprocessing.py, I am getting the error,
ValueError: Could not convert string to float.
Not sure what to do.

Could not find a version that satisfies the requirement

➜  StockPricePrediction git:(master) pip install -r requirements.txt
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Collecting cycler==0.10.0
  Downloading cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
Collecting matplotlib==1.5.1
  Downloading matplotlib-1.5.1.tar.gz (54.0 MB)
     |████████████████████████████████| 54.0 MB 320 kB/s
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/local/opt/python@3.9/bin/python3.9 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/4c/13yp5nx15q1_vfhb_cswvjt00000gn/T/pip-install-zk0ma6jv/matplotlib_edbe4fa9feb94f2ea61385f02a55d9c2/setup.py'"'"'; __file__='"'"'/private/var/folders/4c/13yp5nx15q1_vfhb_cswvjt00000gn/T/pip-install-zk0ma6jv/matplotlib_edbe4fa9feb94f2ea61385f02a55d9c2/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/4c/13yp5nx15q1_vfhb_cswvjt00000gn/T/pip-pip-egg-info-o2iqiahf
       cwd: /private/var/folders/4c/13yp5nx15q1_vfhb_cswvjt00000gn/T/pip-install-zk0ma6jv/matplotlib_edbe4fa9feb94f2ea61385f02a55d9c2/
  Complete output (80 lines):
  ============================================================================
  Edit setup.cfg to change the build options

  BUILDING MATPLOTLIB
              matplotlib: yes [1.5.1]
                  python: yes [3.9.9 (main, Nov 21 2021, 03:23:44)  [Clang
                          13.0.0 (clang-1300.0.29.3)]]
                platform: yes [darwin]

  REQUIRED DEPENDENCIES AND EXTENSIONS
                   numpy: yes [version 1.21.4]
                dateutil: yes [using dateutil version 2.8.2]
                    pytz: yes [pytz was not found. pip will attempt to install
                          it after matplotlib.]
                  cycler: yes [using cycler version 0.10.0]
                 tornado: yes [tornado was not found. It is required for the
                          WebAgg backend. pip/easy_install may attempt to
                          install it after matplotlib.]
               pyparsing: yes [using pyparsing version 3.0.6]
                  libagg: yes [pkg-config information for 'libagg' could not
                          be found. Using local copy.]
                freetype: yes [version 2.11.1]
                     png: yes [version 1.6.37]
                   qhull: yes [pkg-config information for 'qhull' could not be
                          found. Using local copy.]

  OPTIONAL SUBPACKAGES
             sample_data: yes [installing]
                toolkits: yes [installing]
                   tests: yes [nose 0.11.1 or later is required to run the
                          matplotlib test suite. Please install it with pip or
                          your preferred tool to run the test suite / using
                          unittest.mock]
          toolkits_tests: yes [nose 0.11.1 or later is required to run the
                          matplotlib test suite. Please install it with pip or
                          your preferred tool to run the test suite / using
                          unittest.mock]

  OPTIONAL BACKEND EXTENSIONS
                  macosx: yes [installing, darwin]
                  qt5agg: no  [PyQt5 not found]
                  qt4agg: no  [PySide not found; PyQt4 not found]
                 gtk3agg: no  [Requires pygobject to be installed.]
  multiprocessing.pool.RemoteTraceback:
  """
  Traceback (most recent call last):
    File "/usr/local/Cellar/[email protected]/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 125, in worker
      result = (True, func(*args, **kwds))
    File "/usr/local/Cellar/[email protected]/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
      return list(map(*args))
    File "/private/var/folders/4c/13yp5nx15q1_vfhb_cswvjt00000gn/T/pip-install-zk0ma6jv/matplotlib_edbe4fa9feb94f2ea61385f02a55d9c2/setupext.py", line 1782, in backend_gtk3cairo_internal_check
      import cairocffi
    File "/usr/local/lib/python3.9/site-packages/cairocffi/__init__.py", line 48, in <module>
      cairo = dlopen(
    File "/usr/local/lib/python3.9/site-packages/cairocffi/__init__.py", line 45, in dlopen
      raise OSError(error_message)  # pragma: no cover
  OSError: no library called "cairo" was found
  no library called "libcairo-2" was found
  cannot load library 'libcairo.so.2': dlopen(libcairo.so.2, 0x0002): tried: '/usr/local/lib/libcairo.so.2' (no such file), 'libcairo.so.2' (no such file), '/usr/local/lib/libcairo.so.2' (no such file), '/usr/lib/libcairo.so.2' (no such file), '/private/var/folders/4c/13yp5nx15q1_vfhb_cswvjt00000gn/T/pip-install-zk0ma6jv/matplotlib_edbe4fa9feb94f2ea61385f02a55d9c2/libcairo.so.2' (no such file), '/usr/local/lib/libcairo.so.2' (no such file), '/usr/lib/libcairo.so.2' (no such file)
  cannot load library 'libcairo.2.dylib': dlopen(libcairo.2.dylib, 0x0002): tried: '/usr/local/lib/libcairo.2.dylib' (no such file), 'libcairo.2.dylib' (no such file), '/usr/local/lib/libcairo.2.dylib' (no such file), '/usr/lib/libcairo.2.dylib' (no such file), '/private/var/folders/4c/13yp5nx15q1_vfhb_cswvjt00000gn/T/pip-install-zk0ma6jv/matplotlib_edbe4fa9feb94f2ea61385f02a55d9c2/libcairo.2.dylib' (no such file), '/usr/local/lib/libcairo.2.dylib' (no such file), '/usr/lib/libcairo.2.dylib' (no such file)
  cannot load library 'libcairo-2.dll': dlopen(libcairo-2.dll, 0x0002): tried: '/usr/local/lib/libcairo-2.dll' (no such file), 'libcairo-2.dll' (no such file), '/usr/local/lib/libcairo-2.dll' (no such file), '/usr/lib/libcairo-2.dll' (no such file), '/private/var/folders/4c/13yp5nx15q1_vfhb_cswvjt00000gn/T/pip-install-zk0ma6jv/matplotlib_edbe4fa9feb94f2ea61385f02a55d9c2/libcairo-2.dll' (no such file), '/usr/local/lib/libcairo-2.dll' (no such file), '/usr/lib/libcairo-2.dll' (no such file)
  """

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/private/var/folders/4c/13yp5nx15q1_vfhb_cswvjt00000gn/T/pip-install-zk0ma6jv/matplotlib_edbe4fa9feb94f2ea61385f02a55d9c2/setup.py", line 167, in <module>
      result = package.check()
    File "/private/var/folders/4c/13yp5nx15q1_vfhb_cswvjt00000gn/T/pip-install-zk0ma6jv/matplotlib_edbe4fa9feb94f2ea61385f02a55d9c2/setupext.py", line 518, in check
      additional_info = self.check_requirements()
    File "/private/var/folders/4c/13yp5nx15q1_vfhb_cswvjt00000gn/T/pip-install-zk0ma6jv/matplotlib_edbe4fa9feb94f2ea61385f02a55d9c2/setupext.py", line 1828, in check_requirements
      success, msg = res.get(timeout=10)[0]
    File "/usr/local/Cellar/[email protected]/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 771, in get
      raise self._value
  OSError: no library called "cairo" was found
  no library called "libcairo-2" was found
  cannot load library 'libcairo.so.2': dlopen(libcairo.so.2, 0x0002): tried: '/usr/local/lib/libcairo.so.2' (no such file), 'libcairo.so.2' (no such file), '/usr/local/lib/libcairo.so.2' (no such file), '/usr/lib/libcairo.so.2' (no such file), '/private/var/folders/4c/13yp5nx15q1_vfhb_cswvjt00000gn/T/pip-install-zk0ma6jv/matplotlib_edbe4fa9feb94f2ea61385f02a55d9c2/libcairo.so.2' (no such file), '/usr/local/lib/libcairo.so.2' (no such file), '/usr/lib/libcairo.so.2' (no such file)
  cannot load library 'libcairo.2.dylib': dlopen(libcairo.2.dylib, 0x0002): tried: '/usr/local/lib/libcairo.2.dylib' (no such file), 'libcairo.2.dylib' (no such file), '/usr/local/lib/libcairo.2.dylib' (no such file), '/usr/lib/libcairo.2.dylib' (no such file), '/private/var/folders/4c/13yp5nx15q1_vfhb_cswvjt00000gn/T/pip-install-zk0ma6jv/matplotlib_edbe4fa9feb94f2ea61385f02a55d9c2/libcairo.2.dylib' (no such file), '/usr/local/lib/libcairo.2.dylib' (no such file), '/usr/lib/libcairo.2.dylib' (no such file)
  cannot load library 'libcairo-2.dll': dlopen(libcairo-2.dll, 0x0002): tried: '/usr/local/lib/libcairo-2.dll' (no such file), 'libcairo-2.dll' (no such file), '/usr/local/lib/libcairo-2.dll' (no such file), '/usr/lib/libcairo-2.dll' (no such file), '/private/var/folders/4c/13yp5nx15q1_vfhb_cswvjt00000gn/T/pip-install-zk0ma6jv/matplotlib_edbe4fa9feb94f2ea61385f02a55d9c2/libcairo-2.dll' (no such file), '/usr/local/lib/libcairo-2.dll' (no such file), '/usr/lib/libcairo-2.dll' (no such file)
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/8f/f4/c0c7e81f64d5f4d36e52e393af687f28882c53dcd924419d684dc9859f40/matplotlib-1.5.1.tar.gz#sha256=3ab8d968eac602145642d0db63dd8d67c85e9a5444ce0e2ecb2a8fedc7224d40 (from https://pypi.org/simple/matplotlib/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement matplotlib==1.5.1 (from versions: 0.86, 0.86.1, 0.86.2, 0.91.0, 0.91.1, 1.0.1, 1.1.0, 1.1.1, 1.2.0, 1.2.1, 1.3.0, 1.3.1, 1.4.0, 1.4.1rc1, 1.4.1, 1.4.2, 1.4.3, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 2.0.0b1, 2.0.0b2, 2.0.0b3, 2.0.0b4, 2.0.0rc1, 2.0.0rc2, 2.0.0, 2.0.1, 2.0.2, 2.1.0rc1, 2.1.0, 2.1.1, 2.1.2, 2.2.0rc1, 2.2.0, 2.2.2, 2.2.3, 2.2.4, 2.2.5, 3.0.0rc2, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.1.0rc1, 3.1.0rc2, 3.1.0, 3.1.1, 3.1.2, 3.1.3, 3.2.0rc1, 3.2.0rc3, 3.2.0, 3.2.1, 3.2.2, 3.3.0rc1, 3.3.0, 3.3.1, 3.3.2, 3.3.3, 3.3.4, 3.4.0rc1, 3.4.0rc2, 3.4.0rc3, 3.4.0, 3.4.1, 3.4.2, 3.4.3, 3.5.0b1, 3.5.0rc1, 3.5.0, 3.5.1)
ERROR: No matching distribution found for matplotlib==1.5.1

Tried with Python 3.9.9 and Python 2.7.18

dataset link

your dataset link is not working

Use of MAPE

Can this be updated with MAPE metrics, because as I see from the graph the reported RMSE doesn't seem to provide much insights. And as seen from this paper ARIMA doesn't work well. Most stocks in BSE are capped at 5% high or low for a day and with that in mind a model with a MAPE of greater than that value isn't of much help.

can you provide data FB.csv you used in stock market prediction

Does this works only with dataset provided?

Hi,

I would like to test this with Indian NSE market.
Will this works only with the dataset provided by you?

Do you have any scripts written to get data online and run through this?
If it's already included, please provide the steps to get the data.

Thanks in advance.

windows

Hi, can you tell me how to run this on windows?

Where did 50_day_moving_avg column come from ??

How did you calculate the columns (50_day_moving_avg, 10_day_volatility)? Did you extract them from the standard four columns (open, close, high, low) by any means?

This problem occurred when I was running

if name == 'main': #32
main(sys.argv[1], sys.argv[2]) #33

line 33, in
main(sys.argv[1], sys.argv[2])
IndexError: list index out of range，

How to solve？Thank you

This Program doesn't predict Stock market prices

This Alghorithm just matches the market close value of one day to the market open value of the next day and calls this a prediction...

SyntaxError: invalid syntax

python3 scripts/Algorithms/regression_models.py input output
  File "StockPricePrediction/scripts/Algorithms/regression_models.py", line 77
    except Exception, e:
                    ^
SyntaxError: invalid syntax

But, isn't the command like: python scripts/Algorithms/regression_models.py <input-dir> <output-dir>?