datasets / s-and-p-500

S&P 500 index data (aka Standard and Poor's index of 500 major US stocks)

S&P 500 index data including level, dividend, earnings and P/E ratio on a monthly basis since 1870. The S&P 500 (Standard and Poor's 500) is a free-float, capitalization-weighted index of the top 500 publicly listed stocks in the US (top 500 by market cap).

Data

The data provided here is a tidied and CSV'd version of that collected and prepared by the Economist Robert Shiller and made available on his website.

Source Data Construction

Details of the data construction as described on Shiller's website (and slightly reformatted):

Stock market data used in my book, Irrational Exuberance [Princeton University Press 2000, Broadway Books 2001, 2nd ed., 2005] are available for download, Excel file (xls). This data set consists of monthly stock price, dividends, and earnings data and the consumer price index (to allow conversion to real values), all starting January 1871.

The price, dividend, and earnings series are from the same sources as described in Chapter 26 of my earlier book (Market Volatility [Cambridge, MA: MIT Press, 1989]), although now I use monthly data, rather than annual data. Monthly dividend and earnings data are computed from the S&P four-quarter totals for the quarter since 1926, with linear interpolation to monthly figures. Dividend and earnings data before 1926 are from Cowles and associates (Common Stock Indexes, 2nd ed. [Bloomington, Ind.: Principia Press, 1939]), interpolated from annual data.

Stock price data are monthly averages of daily closing prices through January 2000, the last month available as this book goes to press. The CPI-U (Consumer Price Index-All Urban Consumers) published by the U.S. Bureau of Labor Statistics begins in 1913; for years before 1913 1 spliced to the CPI Warren and Pearson's price index, by multiplying it by the ratio of the indexes in January 1913. December 1999 and January 2000 values for the CPI-U are extrapolated. See George F. Warren and Frank A. Pearson, Gold and Prices (New York: John Wiley and Sons, 1935). Data are from their Table 1, pp. 11–14.

For the Plots, I have multiplied the inflation-corrected series by a constant so that their value in january 2000 equals their nominal value, i.e., so that all prices are effectively in January 2000 dollars.

License

No exact statement on license of original data but given size and factual nature believe one can assume these are public domain (and I, the maintainer, explicitly license under the ODC Public Domain Dedication and License (PDDL)).

That said, it would be natural to credit Robert Shiller for preparing this dataset and kindly making it publicly available.

s-and-p-500's Issues

turn off travis settings

Wrong command to run: python scripts/data.py

This is probably a typo. The right one should be
python scripts/process.py

After running this, I got something strange, though.

Traceback (most recent call last):
File "scripts/process.py", line 74, in
process()
File "scripts/process.py", line 70, in process
extract()
File "scripts/process.py", line 17, in extract
rows, metadata = xls.parse(open(fp))
File "E:\Programs\Anaconda\lib\site-packages\dataconverters\xls.py", line 32, in parse
table_set = xlsclass(stream, **kwargs)
File "E:\Programs\Anaconda\lib\site-packages\messytables\excel.py", line 77, in init
self.workbook = get_workbook()
File "E:\Programs\Anaconda\lib\site-packages\messytables\excel.py", line 50, in get_workbook
raise ReadError("Can't read Excel file: %r" % value, traceback)
messytables.error.ReadError: ('Can't read Excel file: XLRDError("Unsupported format, or corrupt file: Expected BOF record; found '\xd0\xcf\x11\xe0\xa1\xb1'",)', <traceback object at 0x00000000038303C8>)

I think it is probably something related to the dataconverters I installed. But I installed it through the command:
pip install dataconverters
And I also installed messytables already.
I don't know what's wrong with that. Anybody also has similar problem? Thanks.

new

My name is Luis, I'm a big-data machine-learning developer, I'm a fan of your work, and I usually check your updates.

I was afraid that my savings would be eaten by inflation. I have created a powerful tool that based on past technical patterns (volatility, moving averages, statistics, trends, candlesticks, support and resistance, stock index indicators).
All the ones you know (RSI, MACD, STOCH, Bolinger Bands, SMA, DEMARK, Japanese candlesticks, ichimoku, fibonacci, williansR, balance of power, murrey math, etc) and more than 200 others.

The tool creates prediction models of correct trading points (buy signal and sell signal, every stock is good traded in time and direction).
For this I have used big data tools like pandas python, stock market libraries like: tablib, TAcharts ,pandas_ta... For data collection and calculation.
And powerful machine-learning libraries such as: Sklearn.RandomForest , Sklearn.GradientBoosting, XGBoost, Google TensorFlow and Google TensorFlow LSTM.

With the models trained with the selection of the best technical indicators, the tool is able to predict trading points (where to buy, where to sell) and send real-time alerts to Telegram or Mail. The points are calculated based on the learning of the correct trading points of the last 2 years (including the change to bear market after the rate hike).

I think it could be useful to you, to improve, I would like to share it with you, and if you are interested in improving and collaborating I am also willing, and if not file it in the box.

If tou want, Please read the readme , and in case of any problem you can contact me ,
If you are convinced try to install it with the documentation.
https://github.com/Leci37/stocks-Machine-learning-RealTime-telegram/tree/develop I appreciate the feedback

Missing 2015 Data

You have a wonderful website. But data for the S&P 500 is missing for year 2015. Is this a known issue? Is there some special way to access that data all at once, rather than by individual months?