7astro7 / full_fred

Full Python interface to Federal Reserve Economic Data (FRED)

full_fred

full_fred is a Python interface to FRED (Federal Reserve Economic Data) that prioritizes user preference, flexibility, and speed. full_fred's API translates to Python every type of request FRED supports: each query for Categories, Releases, Series, Sources, and Tags found within FRED's web service has a method associated with it in full_fred. full_fred minimizes redundant queries for the sake of users and FRED's servers. After a request for data is made to FRED web service the retrieved data is stored in a dictionary, accessible and fungible

Installation

pip install full-fred

Testing

full_fred requires pytest. Tests can be run with FRED_API_KEY environment variable set and:

pytest

Usage

API Key

Queries to FRED web service require an API key. FRED has free API keys available with an account (also free)

You can tell full_fred about an api key in 2 secure ways:

1. fred.api_key_file can be set by passing it to the constructor

In [4]: from full_fred.fred import Fred

In [5]: fred = Fred('example_key.txt')

In [6]: fred.get_api_key_file()
Out[6]: 'example_key.txt'

This will set it too

In [3]: fred.set_api_key_file('example_key.txt')
Out[3]: True

If the file assigned to api_key_file can't be found, full_fred will say so immediately if api_key_file is set using the surefire fred.set_api_key_file()

2. FRED_API_KEY Environment Variable

full_fred will automatically detect your api key if it's assigned to an environment variable named FRED_API_KEY. To check that FRED_API_KEY environment variable is detected, you can use

In [7]: fred.env_api_key_found()
Out[7]: True

full_fred does not store your api key in an attribute for the sake of security: to send queries to FRED's databases, full_fred uses the value of FRED_API_KEY environment variable or the first line of fred.api_key_file

Fetching data

A pandas DataFrame stores observations when a request for data values is made

fred.get_series_df('GDPPOT')
    realtime_start realtime_end        date               value
0       2021-04-03   2021-04-03  1949-01-01         2103.179936
1       2021-04-03   2021-04-03  1949-04-01  2130.7327210000003
2       2021-04-03   2021-04-03  1949-07-01  2159.4478710000003
3       2021-04-03   2021-04-03  1949-10-01         2186.907265
4       2021-04-03   2021-04-03  1950-01-01          2216.07306
..             ...          ...         ...                 ...
327     2021-04-03   2021-04-03  2030-10-01            23219.35
328     2021-04-03   2021-04-03  2031-01-01            23318.31
329     2021-04-03   2021-04-03  2031-04-01            23417.38
330     2021-04-03   2021-04-03  2031-07-01            23516.38
331     2021-04-03   2021-04-03  2031-10-01            23615.28

[332 rows x 4 columns]

The fetched data is stored in fred.series_stack (see Accessing fetched data section for more on retrieving queried data)

fred.series_stack['get_series_df']
{'realtime_start': '2021-04-03',
 'realtime_end': '2021-04-03',
 'observation_start': '1600-01-01',
 'observation_end': '9999-12-31',
 'units': 'lin',
 'output_type': 1,
 'file_type': 'json',
 'order_by': 'observation_date',
 'sort_order': 'asc',
 'count': 332,
 'offset': 0,
 'limit': 100000,
 'series_id': 'GDPPOT',
 'df':     
realtime_start      realtime_end        date               value
 0       2021-04-03   2021-04-03  1949-01-01         2103.179936
 1       2021-04-03   2021-04-03  1949-04-01  2130.7327210000003
 2       2021-04-03   2021-04-03  1949-07-01  2159.4478710000003
 3       2021-04-03   2021-04-03  1949-10-01         2186.907265
 4       2021-04-03   2021-04-03  1950-01-01          2216.07306
 ..             ...          ...         ...                 ...
 327     2021-04-03   2021-04-03  2030-10-01            23219.35
 328     2021-04-03   2021-04-03  2031-01-01            23318.31
 329     2021-04-03   2021-04-03  2031-04-01            23417.38
 330     2021-04-03   2021-04-03  2031-07-01            23516.38
 331     2021-04-03   2021-04-03  2031-10-01            23615.28
 
 [332 rows x 4 columns]}

To find a specific category_id or to search FRED categories from most general to most specific start with the root category 0. A search along the lines of the following can help to pinpoint different category_ids:

In [4]: fred.get_child_categories(0)
Out[4]: 
{'categories': [{'id': 32991,
   'name': 'Money, Banking, & Finance',
   'parent_id': 0},
  {'id': 10,
   'name': 'Population, Employment, & Labor Markets',
   'parent_id': 0},
  {'id': 32992, 'name': 'National Accounts', 'parent_id': 0},
  {'id': 1, 'name': 'Production & Business Activity', 'parent_id': 0},
  {'id': 32455, 'name': 'Prices', 'parent_id': 0},
  {'id': 32263, 'name': 'International Data', 'parent_id': 0},
  {'id': 32213, 'name': 'Greenbook Projections', 'parent_id': 0},
  {'id': 3008, 'name': 'U.S. Regional Data', 'parent_id': 0},
  {'id': 33060, 'name': 'Academic Data', 'parent_id': 0}]}

In [5]: fred.category_stack['get_child_categories']
Out[5]: 
{'categories': [{'id': 32991,
   'name': 'Money, Banking, & Finance',
   'parent_id': 0},
  {'id': 10,
   'name': 'Population, Employment, & Labor Markets',
   'parent_id': 0},
  {'id': 32992, 'name': 'National Accounts', 'parent_id': 0},
  {'id': 1, 'name': 'Production & Business Activity', 'parent_id': 0},
  {'id': 32455, 'name': 'Prices', 'parent_id': 0},
  {'id': 32263, 'name': 'International Data', 'parent_id': 0},
  {'id': 32213, 'name': 'Greenbook Projections', 'parent_id': 0},
  {'id': 3008, 'name': 'U.S. Regional Data', 'parent_id': 0},
  {'id': 33060, 'name': 'Academic Data', 'parent_id': 0}]}

The whole gamut of requests on FRED web service is implemented. The example below is one among many other methods in the API, listed in the next section

In [1]: from full_fred.fred import Fred

In [2]: fred = Fred()

In [3]: fred.get_series_vintagedates('FYFSD', limit = 15)
Out[3]: 
{'realtime_start': '1776-07-04',
 'realtime_end': '9999-12-31',
 'order_by': 'vintage_date',
 'sort_order': 'asc',
 'count': 46,
 'offset': 0,
 'limit': 15,
 'vintage_dates': [
    '1998-02-02',
    '1998-10-26',
    '1999-02-01',
    '1999-10-25',
    '2000-02-07',
    '2000-10-20',
    '2001-04-09',
    '2001-10-24',
    '2002-02-04',
    '2002-10-23',
    '2003-02-03',
    '2003-10-15',
    '2004-02-02',
    '2004-10-12',
    '2005-02-23']}

In [4]: fred.series_stack['get_series_vintagedates']
Out[4]: 
{'realtime_start': '1776-07-04',
 'realtime_end': '9999-12-31',
 'order_by': 'vintage_date',
 'sort_order': 'asc',
 'count': 46,
 'offset': 0,
 'limit': 15,
 'vintage_dates': [
    '1998-02-02',
    '1998-10-26',
    '1999-02-01',
    '1999-10-25',
    '2000-02-07',
    '2000-10-20',
    '2001-04-09',
    '2001-10-24',
    '2002-02-04',
    '2002-10-23',
    '2003-02-03',
    '2003-10-15',
    '2004-02-02',
    '2004-10-12',
    '2005-02-23']}

Accessing fetched data

There are 5 stacks:

fred.category_stack fred.release_stack fred.series_stack fred.source_stack fred.tag_stack

After a method is called the returned data is stored using the method name for its key

Methods that store data in category stack:

fred.category_stack["get_a_category"]
fred.category_stack["get_child_categories"]
fred.category_stack["get_related_categories"]
fred.category_stack["get_series_in_a_category"]
fred.category_stack["get_tags_for_a_category"]
fred.category_stack["get_related_tags_for_a_category"]

Methods that store data in release stack:

fred.release_stack["get_a_release"]
fred.release_stack["get_tags_for_a_release"]
fred.release_stack["get_series_on_a_release"]
fred.release_stack["get_sources_for_a_release"]
fred.release_stack["get_related_tags_for_release"]
fred.release_stack["get_release_dates_all_releases"]
fred.release_stack["get_release_tables"]
fred.release_stack["get_release_dates"]
fred.release_stack["get_all_releases"]

Methods that store data in series stack:

fred.series_stack["get_a_series"]
fred.series_stack["get_categories_of_series"]
fred.series_stack["get_series_df"]
fred.series_stack["get_release_for_a_series"]
fred.series_stack["search_for_series"]
fred.series_stack["get_tags_for_series_search"]
fred.series_stack["get_related_tags_for_series_search"]
fred.series_stack["get_tags_for_a_series"]
fred.series_stack["get_series_updates"]
fred.series_stack["get_series_vintagedates"]

Methods that store data in source stack:

fred.source_stack["get_all_sources"]
fred.source_stack["get_releases_for_a_source"]
fred.source_stack["get_a_source"]

Methods that store data in tag stack:

fred.tag_stack["get_all_tags"]
fred.tag_stack["get_related_tags_for_a_tag"]
fred.tag_stack["get_series_matching_tags"]

full_fred realtime period and observation start/end defaults

By default fred.realtime_start and fred.realtime_end are set to None. realtime_start and realtime_end arguments override fred.realtime_start and fred.realtime_end.

fred.observation_start and fred.observation_end are also None by default. observation_start and observation_end arguments override fred.observation_start and fred.observation_end.

Contributing

The full_fred project welcomes feature requests, bug reports, bug fixes, documentation improvements, contributions of all kinds. full_fred aims to be responsive in integrating patches and listening to your feedback to be a community-driven API. This project is also new and while full_fred is still young there's great opportunity to contribute elements that may have disproportionate impact in the long run

License

GPLv3

full_fred's Issues

fred.get_series_df('GOLDPMGBD229NLBM') -> Error cannot create DataFrame

When I run this line of code:
fred.get_series_df('GOLDPMGBD229NLBM')

I get the following error message:
No key 'observations' found, cannot make DataFrame

KeyError Traceback (most recent call last)
in
----> 1 df_gold = fred.get_series_df('GOLDPMGBD229NLBM')
2 df_gold

~\anaconda3\lib\site-packages\full_fred\series.py in get_series_df(self, series_id, realtime_start, realtime_end, limit, offset, sort_order, observation_start, observation_end, units, frequency, aggregation_method, output_type, vintage_dates)
339 e = "No key 'observations' found, cannot make DataFrame"
340 print(e)
--> 341 self.series_stack["get_series_df"].pop("observations")
342 self.series_stack["get_series_df"]["df"] = df
343 return self.series_stack["get_series_df"]["df"]

KeyError: 'observations'

integrate codecoov

Integrate code coverage to incentivize tests and measure coverage

observation_date

Is it possible to alter the observation_date rather than realtime_date?

For example, on series UNRATE I'd like to just grab the last 12 months - but realtime_date wouldn't effect this selection:

realtime_start realtime_end        date value
0      2021-05-07   2021-05-07  1948-01-01   3.4
1      2021-05-07   2021-05-07  1948-02-01   3.8
2      2021-05-07   2021-05-07  1948-03-01   4.0
3      2021-05-07   2021-05-07  1948-04-01   3.9
4      2021-05-07   2021-05-07  1948-05-01   3.5
5      2021-05-07   2021-05-07  1948-06-01   3.6
6      2021-05-07   2021-05-07  1948-07-01   3.6
7      2021-05-07   2021-05-07  1948-08-01   3.9
8      2021-05-07   2021-05-07  1948-09-01   3.8
9      2021-05-07   2021-05-07  1948-10-01   3.7
10     2021-05-07   2021-05-07  1948-11-01   3.8
11     2021-05-07   2021-05-07  1948-12-01   4.0

Thank you!

integrate doctest

use doctest to test code embedded in docs

A part of critical information inaccessible

Hi, if you pay close attention to a FRED page like this:
https://fred.stlouisfed.org/series/GDPPOT
There is also some textual data attached to this time series about the Units, the Frequency, and a one line description, in the notes section. this part of data is also critical for the timeseries to be meaningful, but seemingly it is not accessible through this API.
Can you develop an integrated workaround please?

fred.env_api_key_found() returning false