tomaugspurger / pycps

Download CPS files

PyCPS

A python package for working with the Current Population Survey.

Documentation is available at readthedocs.

Neither the Census Bureau nor the NBER provide a clean, RESTful API for getting CPS data. This makes working with the CPS a pain, and reproducibility nearly impossible.

What does it do?

There's a few related functions PyCPS provides:

1. Downloading data dictionaries and monthly data files
2. Standardizing variables across months
3. Merging to create time series

Installation

• From pip: pip install pycps

• From source:

  git clone https://github.com/TomAugspurger/pycps
  cd pycps
  pip install .
  pip install -r requirements.txt
  pip install git+https://github.com/PyTables/PyTables
  python setup.py install
  

Dependencies

See requirements.txt

Python

Developed with python 3, aims to be compatible with python 2.

pycps's Issues

BUG: Make paths before using

REF/API: Add option to append info files to default

Similar to the refactor for data fixups. Just merge the lists by key.

BUG/COMPAT: python2 doesn't substitute paths in readsettings

FAIL: test_substitue (pycps.pycps.tests.test_parsers.TestReaderSettings)

Traceback (most recent call last):
File "/Users/tom/Envs/build/lib/python2.7/site-packages/pycps/pycps/tests/test_parsers.py", line 67, in test_substitue
self.assertEqual(result, expected)
AssertionError: u'{data_path}/data_dictionaries/' != 'data/data_dictionaries/'

Data Dictionary Errors

Compiling a list here. If these are fixed upstream, the workarounds needn't be applied.

File	Column	Line Number	Description	Reported	Status
1992-01	31	221	Gap in DD	No	To Report
1992-01	157	755	Gap in DD	No	To Report
1992-01	259	1304	Gap in DD	No	To Report
1992-01	171	2803	Gap in DD	No	To Report
1994-01	135	749	Wrong start column	No	To Report
1994-01	679	3910	Wrong start column	No	To Report
1994-04	135	713	Wrong start column	No	To Report
1994-04	679	3874	Wrong start column	No	To Report
1995-06	135	665	Wrong start column	No	To Report
1994-04	679	3826	Wrong start column	No	To Report
1998-01	149	~757	Gap in DD	Yes	Waiting
1998-01	535	~3130	Gap in DD	Yes	Waiting
1998-01	556	~3162	Gap in DD	Yes	Waiting
1998-01	632	~3313	Gap in DD	Yes	Waiting
1998-01	680	~3427	Gap in DD	Yes	Waiting
1998-01	786	~3733	Gap in DD	Yes	Waiting
2004-05	410	2438	Wrong Start; should be 410	Yes	Waiting
2005-08	411	2421	Wrong Start; should be 410	Yes	Waiting
2009-01	399	4930	Wrong Width: Should be 19	Yes	Waiting
2012-05*	114	954	Duplicate at col 114	Yes	Waiting
2012-05	637	4243	Gap in DD	Yes	Waiting

• File does say "Starting in Feb. 2005” on line 114, which seems like a copy paste error

In addition to the above:

• New questions were added in November 2005 (Katrina. This should result in a new data dictionary

REF: Refactor Data Adjustment

Right now they're all in pycps.parsers.fixup_by_dd. Do something similar to the dd adjustments.

Will have to have a different naming method since the functions should be generic.

"import pycps" does not work

import pycps yields error [...]/logging.json not found.

December 2007 Revised Weights

Figure out what to do with: http://www.nber.org/cps-basic/cpsrwdec07.ddf

Skipping in the code right now.

API: Fix `store_name`s to be valid python identifiers

Do this just before the 0.4 release.

Add support for Jan 1994 DD

pdf only on the nber website.

going to special case in the code and use: http://thedataweb.rm.census.gov/pub/cps/basic/199401-199403/jan94_mar94_dd.txt

Also adding DDS from http://thedataweb.rm.census.gov/pub/cps/basic/199404-199505/apr94_may95_dd.txt and http://thedataweb.rm.census.gov/pub/cps/basic/199506-199508/jun95_aug95_dd.txt which gives us full dd coverage from January 1994 through the present.

DATA: Downloader skips May 13 and Jan 14

From http://www.nber.org/data/cps_basic.html

May 2013 skipped because .txt extension http://www.nber.org/cps-basic/January_2013_Record_Layout.txt

Jan 2014 skipped because Word 👎 http://www.nber.org/cps-basic/January_2014_Record_Layout.doc

DATA: Finish col_rename_by_dd

In data.json, with optional functions needed to align things.
verify the ones that are there.

ENH: Setup logging

need to log

• every print statement.
• every row dropped due to NaNs, etc.
• data transformations? e.g. binning a year?
• writes to stores

DATA: Filter NaNs in Index

Or try to infer if its just the lineno that's NaN