rsvp / fecon235 Goto Github PK

Description of specific issue

We are expecting major disruption in getting data on
equities, mutual funds, and ETFs via pandas_datareader
due its dependency on API of both Yahoo and Google Finance.

• [/] Fail quote retrieval via our module lib/yi_stock.py, if no fix upstream.
• [/] Enhancement, if fecon235 takes independent action in its yi_stock module.

Observed behavior

Yahoo employee has confirmed that the free End-Of-Day data has been terminated, 2017
presumably due to acquisition by Verizon.

• Possible work-around: https://stackoverflow.com/questions/44030983/yahoo-finance-url-not-working

/r/algotrading on Google JSON termination
presumably due to cost-cutting by new product manager.
Our yi_stock module may appear to be working, but
please tail() your dataframe to verify whether quote retrieval
goes beyond 2017-09-05 [silent fail to get() current stock quotes].

Alternatives to enhance behavior

1. Switch over to Quandl, using our lib/yi_quandl.py module

   • See: https://blog.quandl.com/api-for-stock-data lists premium databases. But for free stock price data: https://www.quandl.com/product/WIKIP/WIKI/PRICES-Quandl-End-Of-Day-Stocks-Info.

2. Alpha Vantage: https://www.alphavantage.co/documentation - Requires user to get free API key

   • Python wrapper, MIT license: https://github.com/RomelTorres/alpha_vantage

3. tiingo: https://api.tiingo.com/docs/general/overview - Requires user to get free API key

4. $$$ EODhistoricaldata: https://eodhistoricaldata.com

5. Scrape Google Finance pages: https://github.com/CNuge/general_use_functions/blob/master/international_stock_scraper.py (currently also works for international stocks) -- which is not a robust solution (relative to API code) since the page layout is subject to UI changes: pydata/pandas-datareader#408

Why would the improvement be useful to most users?

Information from the equities markets is vital for financial economics.

Before releasing our own independent solution, we expect to
make a pull request to the pandas_datareader repository.

Please kindly propose alternative solutions
below, or at https://gitter.im/rsvp/fecon235
... and look into pandas_datareader issues

Check for revisions upstream

It is possible that a solution has been merged into pandas_datareader
and all that is necessary is an update of the package, see its CHANGELOG:
https://pandas-datareader.readthedocs.io/en/latest/whatsnew.html
and make sure the update is compatible with the latest fecon235.

Note: Anaconda distribution uses hyphen, not underscore:

$ conda update pandas-datareader

Additional helpful details for bugs

• [/] Problem started recently

• [/] Problem can be reliably reproduced

• fecon235 version: v5.17.0603

• pandas version: 0.19.2

• pandas_datareader version: 0.2.1

Major release v4.15.1230 benefits from the python3 compatibility changes
made during v3. All modules are now operational under
both Python 2 and 3. Also, code has been rewritten for
cross-platform performance (Linux, Mac, and Windows).

We MOVED the yi-modules from nb to a new directory: lib.
Python 3 uses absolute import and our python2 code
now conforms to that practice.

To update pre-2016 notebooks, please use import style
discussed in docs README: https://git.io/fecon-intro
The top-level module fecon235.py
(formerly known as nb/fecon.py) is also
explained in that introduction.
With adoption of python3 print_function,
the python2 print statement must be rewritten as a function.

We also highly recommend inclusion of PREAMBLE-p6.15.1223
which gives versioning requirements for successful
notebook replication. With those fixes, our notebooks
should run under both Python kernels in Jupyter,
without any dependency on Linux.

Description of specific issue

As of pandas 0.18 the use of its ewma() function will produce a warning which looks like:

    FutureWarning: pd.ewm_mean is deprecated for DataFrame and will be 
    removed in a future version, replace with
    DataFrame.ewm(ignore_na=False,span=30.007751938,min_periods=0,adjust=True).mean()

therefore, this deprecation will adversely impact fecon235 exponential moving average
function ema in the yi_1tools module -- when pandas goes to 0.19.

Expected behavior

No warning to be raised.

Observed behavior

Computed result has no errors. The warning is given only the first time when the
exponential moving average function is used.

Strangely, pd.ewm_mean?? produces no results, so the warning is cryptic.

Why would the improvement/fix be useful to most users?

• Bug prevention for pandas 0.19 and above

Additional helpful details for bugs

• Problem started recently, but not in older versions
• Problem happens with all files, not only some files
• Problem can be reliably reproduced
• Problem happens randomly
• fecon235 version: v4.16.0525
• pandas version: 0.18.0
• Python version: 2.7.11

Description of specific issue

Mathematical equations, in both inline and display modes,
render correctly when viewed in local browser --
however, when a Jupyter notebook with such equations
is viewed at GitHub the following message is generated:
"Math Processing Error"

The issue can be traced solely to GitHub since
a notebook's source commited to a GH repo viewed through
http://nbviewer.jupyter.org renders the math correctly.

• Bug at Github itself, not here at fecon235

Expected behavior

Here's the correct renderings via
http://nbviewer.jupyter.org/github/rsvp/fecon235/blob/master/nb/qdl-spx-earn-div.ipynb
while noticing the actual source code resides at GitHub.

Observed behavior

Here's the problematic rendering at GitHub:
https://github.com/rsvp/fecon235/blob/master/nb/qdl-spx-earn-div.ipynb
where all the math is replaced by this message,
"Math Processing Error".

Steps towards resolution

1. Contact GitHub admin directly. Their notebook rendering library is apparently proprietary -- supposedly for security reasons.

Additional helpful details for bugs

• Problem started recently, but not in older versions: YES.
• Problem happens with all files, not only some files: notebooks with math.
• Problem can be reliably reproduced: YES, locally and above proof.
• Problem happens randomly: NO.
• fecon235 version: v4.16.0329

ATTN mention: @rsvp

Description of specific issue

Whenever I try to import any of the modules (python 3.5 anaconda, linux mint), I get "SystemError", parent module 'not loaded, cannot perform relative import

• Bug
• Enhancement

Expected behavior

Observed behavior

Steps which reproduce or enhance behavior

Optional: Markdown to include images...
![Screenshot for demonstration](url)

Why would the improvement be useful to most users?

Additional helpful details for bugs

• Problem started recently, but not in older versions
• Problem happens with all files, not only some files
• Problem can be reliably reproduced
• Problem happens randomly
• fecon235 version:
• pandas version:
• Python version:
• Operating system:

ATTN mention: @rsvp

Description of specific issue

Our code, currently compatible with both python27 and python3 series, is characterized as "straddling."

We are keen to use Python 3 to its full potential, and we currently accept the cost of writing cross-compatible code to allow a smooth transition, but we don’t intend to maintain this compatibility indefinitely. Although the transition has not been as quick as we hoped, we do see it taking place, with more and more people using, teaching and recommending Python 3.

The developers of the Python language extended support of Python 2.7 from 2015 to January 1, 2020, recognising that many people were still using Python 2. We believe that the extra 5 years is sufficient to transition off of Python 2, and our projects plan to stop supporting Python 2 when upstream support ends in 2020, if not before. We will then be able to simplify our code and take advantage of the many new features in the current version of the Python language and standard library.

We have pledged to drop support for Python 2.7 no later than 2020. Please see https://python3statement.org for details on the transition in the Python ecosystem.

• Deprecation notice: changes will be very gradual
• Enhancement: top-down refactoring of source code
• Support installation by pip, e.g. pip install --pre fecon236
• Rebuild Docker container using Miniconda3-latest-Linux-x86_64.sh as base

Expected behavior

The transition should be smooth, and we will take the opportunity to refactor and re-organize the code from top-down. Users using the Anaconda distribution based on python27 will have to install a new distribution based on python3. For the fecon235 Docker image, the transition will be announced and automatically be set up for you.

Installation by pip

We have intentionally avoided the pip because optimal installation of fecon235 dependencies has been difficult, esp. for binaries and newbies. Unfortunately, pip is incapable of true dependency resolution (whereas Anaconda conda installs are reliable). This may change towards the year 2020, where only python3 is involved, and pipenv matures. The major annoyances are summarized here: https://python3statement.org/practicalities -- and minor details, not mentioned, are even more tedious.

• fecon235 version: v6 notebooks will begin import of fecon236
• Python version: al least 3.4 through 2019, expect at least 3.6 thereafter.
• Operating systems: Linux, Mac, Windows

Description of specific issue

As of 2018, FRED is making changes to their API which
may alter the URL scheme to retrieve datasets.

This would critically impact the function makeURL()
in module lib/yi_fred.py

• Bug -- NOT yet, old URL still operational

Expected behavior

Data in CSV format gets properly downloaded and processed by fecon235.

Observed behavior

Temporary disruption reported January 2018.

Additional helpful details for bugs

• URL problem is discussed in detail at pydata/pandas-datareader#435

• Alternative: https://research.stlouisfed.org/docs/api/fred/ but this requires an API key

• Problem started recently, but not in older versions

• Problem happens with all files, not only some files

• Problem can be reliably reproduced

• Problem happens randomly

• fecon235 version: v5.17.0722

Description of specific issue

When resampling a time-series the following warning(s) will appear:

FutureWarning: how in .resample() is deprecated
the new syntax is .resample(...).median() fill_method=None)

FutureWarning: .resample() is now a deferred operation
use .resample(...).mean() instead of .resample(...)

It is somewhat cryptic until one realizes how='median'
was being used as an argument to the .resample function.
So how becomes the problem for yi_fred module,
specifically for our functions
daily(), monthly(), and quarterly() in fecon235.

(Sidenote: how='median' since it is more robust than 'mean'.)

The second cryptic warning can be traced to our use of
fill_method=None when upsampling. The new API
urges us to instead use methods:

• .backfill() : use NEXT valid observation to fill
• .ffill() : propagate last valid observation forward to next valid
• .fillna() : fill using nulls
• .asfreq() : convert TimeSeries to specified frequency

• Bug as of pandas 0.19
• Enhancement

Expected behavior

No such warning, possibly fatal termination.

Observed behavior

Warnings started as of pandas 0.18

Why would the improvement be useful to most users?

Because daily(), weekly(), and monthly() in fecon235
should just work without the casual user needing to learn
obscure flags and methods (subject to future API changes).

Additional helpful details for bugs

• Problem started recently, but not in older versions

• Problem happens with all files, not only some files

• Problem can be reliably reproduced

• Problem happens randomly

• fecon235 version: v4.16.1030

• pandas version: 0.18

• Python version: both 2.7 and 3

• Operating system: cross-platform