andrewrporter / yahoo-historical Goto Github PK
View Code? Open in Web Editor NEWDownloads historical EOD (end of day) prices from yahoo finance
License: MIT License
Downloads historical EOD (end of day) prices from yahoo finance
License: MIT License
Hi Andrew,
example:
#l_tickers = ["^DJI","^IXIC","STM.MI"] these works
l_tickers = ["EURGBP=X","EURUSD=X"] # these not
This is raised since recently when using Fetcher:
/home/.../venv3/lib/python3.7/site-packages/yahoo_historical/fetch.py:66: FutureWarning:
.ix is deprecated. Please use
.loc for label based indexing or
.iloc for positional indexing
See the documentation here:
http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#ix-indexer-is-deprecated
return self.getHistorical().ix[:,[0,4]]
/home/.../venv3/lib/python3.7/site-packages/pandas/core/indexing.py:822: FutureWarning:
.ix is deprecated. Please use
.loc for label based indexing or
.iloc for positional indexing
See the documentation here:
http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#ix-indexer-is-deprecated
retval = getattr(retval, self.name)._getitem_axis(key, axis=i)
I am looking at historic data for S&P 500 and Yahoo has it starting from 1950. time.mktime cannot handle dates before 1970-01-01. This is a rare case, but I think a very important one. The solution is to use calendar.timegm, which can handle everything, in your fetch.py file in init function.
Here is my proposed solution. It works for me using your original date format of (YYYY,MM,DD).
import calendar as cal
class Fetcher:
def __init__(self, ticker, start, end=None, interval="1d"):
self.start = int(cal.timegm(dt.datetime(*start).timetuple()))
if end is not None:
self.end = int(cal.timegm(dt.datetime(*start).timetuple()))
else:
self.end = int(time.time())
Doing:
data = Fetcher("ALV.F", [2016,9,27], [2017,9,27])
and getting:
Date Open High Low Close Adj Close \
0 2016-09-26 null null null null null
Hi.
About a week ago, I download your tool (pip install yada yada
) and it worked fine with the BEZQ.TA
share. Today it failed downloading the stock info
The head of the file reads:
from pandas_datareader import data as pdr
import fix_yahoo_finance as yf
import datetime
import os
and the relevant part of the code reads
# download dataframe
stock_name = 'BEZQ.TA'
start = datetime.datetime(2017,1,1)
end = datetime.datetime(2018,1,1)
I saw in #8 that you expect this to be a wrong ticker, but I the code also failed with the error:
File "stock_getter.py", line 68, in <module>
download_and_store(s)
File "stock_getter.py", line 37, in download_and_store
data = pdr.get_data_yahoo(stock_name, start, end)
File "/usr/local/lib/python2.7/dist-packages/fix_yahoo_finance/__init__.py", line 169, in download
actions=actions, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/fix_yahoo_finance/__init__.py", line 329, in download_chunk
get_yahoo_crumb()
File "/usr/local/lib/python2.7/dist-packages/fix_yahoo_finance/__init__.py", line 63, in get_yahoo_crumb
_YAHOO_COOKIE = res.cookies['B']
File "/home/yotam/.local/lib/python2.7/site-packages/requests/cookies.py", line 329, in __getitem__
return self._find_no_duplicates(name)
File "/home/yotam/.local/lib/python2.7/site-packages/requests/cookies.py", line 400, in _find_no_duplicates
raise KeyError('name=%r, domain=%r, path=%r' % (name, domain, path))
KeyError: "name='B', domain=None, path=None"
Sometimes I get instead of this an empty stock. How can I solve this?
Preferably through circle CI, can send emails upon failure
When I run the Fetcher with a list of stocks, I keep getting this ParserError but it's not consistent. It just happens randomly. I tried putting an exemption and attempt the same stock again in a loop and it would sometimes work after a few tries, but later it would indefinitely get stuck on one of the stocks.
ParserError Traceback (most recent call last)
<ipython-input-15-6c36ca50f58c> in <module>
2
3 print('Current time before download:', datetime.datetime.now())
----> 4 get_data(all_stocks,unqualified_stock_list,stock_list_idx=0)
5 print('Current time after download:', datetime.datetime.now())
<ipython-input-13-f19ec7bb5b8b> in get_data(stock_list, unqualified_stock_list, save_limit, stock_list_idx)
9 stock = stock_list[stock_list_idx]
10 try:
---> 11 stock_data = Fetcher(stock, [stock_start_y,stock_start_m,stock_start_d],[stock_end_y,stock_end_m,stock_end_d]).getHistorical().values
12 except KeyError:
13 print('KeyError... Restarting Loop...')
~\Anaconda3\envs\tfgpu\lib\site-packages\yahoo_historical\fetch.py in getHistorical(self)
49 DeprecationWarning,
50 )
---> 51 return self._get("history")
52
53 def getDividends(self):
~\Anaconda3\envs\tfgpu\lib\site-packages\yahoo_historical\fetch.py in _get(self, events)
34 data = requests.get(url, headers={"User-agent": ""})
35 content = StringIO(data.content.decode("utf-8"))
---> 36 return pd.read_csv(content, sep=",")
37
38 def getData(self, events):
~\Anaconda3\envs\tfgpu\lib\site-packages\pandas\io\parsers.py in parser_f(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, dialect, error_bad_lines, warn_bad_lines, delim_whitespace, low_memory, memory_map, float_precision)
674 )
675
--> 676 return _read(filepath_or_buffer, kwds)
677
678 parser_f.__name__ = name
~\Anaconda3\envs\tfgpu\lib\site-packages\pandas\io\parsers.py in _read(filepath_or_buffer, kwds)
452
453 try:
--> 454 data = parser.read(nrows)
455 finally:
456 parser.close()
~\Anaconda3\envs\tfgpu\lib\site-packages\pandas\io\parsers.py in read(self, nrows)
1131 def read(self, nrows=None):
1132 nrows = _validate_integer("nrows", nrows)
-> 1133 ret = self._engine.read(nrows)
1134
1135 # May alter columns / col_dict
~\Anaconda3\envs\tfgpu\lib\site-packages\pandas\io\parsers.py in read(self, nrows)
2035 def read(self, nrows=None):
2036 try:
-> 2037 data = self._reader.read(nrows)
2038 except StopIteration:
2039 if self._first_chunk:
pandas\_libs\parsers.pyx in pandas._libs.parsers.TextReader.read()
pandas\_libs\parsers.pyx in pandas._libs.parsers.TextReader._read_low_memory()
pandas\_libs\parsers.pyx in pandas._libs.parsers.TextReader._read_rows()
pandas\_libs\parsers.pyx in pandas._libs.parsers.TextReader._tokenize_rows()
pandas\_libs\parsers.pyx in pandas._libs.parsers.raise_parser_error()
ParserError: Error tokenizing data. C error: Expected 1 fields in line 4, saw 2
I tested the code in Python3.7:
from yahoo_historical import Fetcher
import datetime
import time
# create unix timestamp representing January 1st, 2007
timestamp = time.mktime(datetime.datetime(2007, 1, 1).timetuple())
data = Fetcher("AAPL", timestamp)
print(data.get_historical())
And I got:
TypeError Traceback (most recent call last)
/var/folders/5z/nfxgl_ps6qlf5tvrct964q980000gn/T/ipykernel_77052/2760101674.py in
6 timestamp = time.mktime(datetime.datetime(2007, 1, 1).timetuple())
7
----> 8 data = Fetcher("AAPL", timestamp)
9 print(data.get_historical())
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/yahoo_historical/fetch.py in init(self, ticker, start, end, interval)
20 self.ticker = ticker.upper()
21 self.interval = interval
---> 22 self.start = int(cal.timegm(dt.datetime(*start).timetuple()))
23
24 if end is not None:
TypeError: type object argument after * must be an iterable, not float
Encountered when importing the module on python 2.7.12
Tried installing with pip and then by running the running the python code directly.
In both cases I get a SyntaxError: on line 13 position 53 (pointing to "interval").
You perhaps have a naming conflict?
Add fetch.Fetch to init.py file so the Fetch can be imported from yahoo_historical module
I'm using Fetcher to download daily data, for some symbols, I got this error:
local variable 'crumb' referenced before assignment
Hi Andrew
thanks for sharing this
do you know if it is possible to get historical data for "exchange rate"
I tried "EUR=" code but gave this error
"name='B', domain=None, path=None"
Hi (yes I am quite new to Python...) - while your code works very well - when I try to download long time series to csv files I always get it truncated btween the first 30 and the last 30 records. Any intentions to implement a 'download' function into your code? /Suggestions? That will help. Thx
from yahoo_historical import Fetcher
data = Fetcher("AAPL", [2007,1,1], [2017,1,1])
print(data.get_historical())
The above code return the following.
Empty DataFrame
Columns: [Forbidden]
Index: []
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.