newsdev / nyt-fec

a smaller, cleaner, campaign finance app that complements the new FEC site

Archived January 2024

fec

Note!

VERSION 1.1 INCLUDES BREAKING CHANGES THAT COULD LEAD TO DATA LOSS! The previous working version is tagged working If you would like to update to the current version, which supports multiple cycles, and you have data in your donors table, please update to v1.0, run the sql updates described here, and then go up to v1.1. Also strongly recommended to backup your whole database before doing any of this.

App is still kind of in the scratchpad phase, absolutely no promises that it works or I won't make breaking changes.

About

This app allows for importing and searching expenditures, independent expenditures and contributions from electronic FEC filings. It relies on the NYT's fec2json library.

Why not just use the FEC website?

The FEC website has been substantially improved recently, but it still lacks several main features we desire.

1. It takes several days for itemizations to be processed, so it is impossible to search transactions right away
2. There are some search fields that are important to me that do not exist in the FEC
3. We want to be able to do more with independent expenditure summing and categorizing
4. We want to be able to add additional data, such as our own donor ids

If you don't really need to deploy and maintain your own standalone campaign finance infrastructure, however, I recommend using tools developped by the FEC including their site, their api or their bulk data. Or use ProPublica's site or api.

Setup instructions

1. pull this repo
2. mkvirtualenv fec --python $(which python3)
3. get a FEC API key here
4. email [email protected] and ask them to upgrade you to 120 api calls per minute
5. add the following to your $VIRTUAL_ENV/bin/postactivate:

   export DJANGO_SETTINGS_MODULE=config.dev.settings
   export fec_DB_NAME=nyt_dev_fec
   export fec_DB_USER=nyt_dev_fec
   export FEC_API_KEY=your-api-key

6. pip install -r requirements.txt
7. createuser -s nyt_dev_fec
8. createdb -U nyt_dev_fec nyt_dev_fec
9. add2virtualenv . && add2virtualenv config && add2virtualenv fec
10. django-admin migrate

nyt-fec's Issues

admin is very slow

can we do anything about that?

add some kind of sort option

for filings(?) but def itemizations

check filing date from first line of file before processing itemizations

we're currently running the whole filing through fec2json before checking whether the coverage_through date makes it rejectable. We should reject before processing, probably in evaluate_filing_file

make donors searchable by address

make refunds into contribs

this needs to be done so we get accurate totals for individuals when we search contribs.

ies by race

we probably want a better way to look at IEs by race (maybe also fundraising).

I think we're going to need a view of some sort in order to do this, because computing the summary stats on the fly will be too slow. Basically I think I want a list of races with amounts, and then a detail page where you can see the items

rss scraper seems busted

take a look, please

use tsquerys for filters

figure out and implement tsquery indexes for contribution and expenditure filters

amendments

make bad things not happen if amendments come in

we're currently holding the same status in two places. fix that.

add fec link to failed filings

put dates in eastern time

currently import dates are utc

backup scraper that hits the fec html page

we've had trouble with the api being behind. i think i'd generally like this to have this available and maybe make a management command as a suspended cron but crank it up on filing nights.

Filing list page

• include date filed
• make it searchable, with options to include at least:
  • committee
  • amended
  • form type
  • min raised/spent

kitty requests

• add address search for expends
• separate occupation and employer search on contribs

ie search

implement it

make donor standardization persist if there are amendments

if an amendment comes in, we need to check whether the underlying filing had any standardized donors and transfer them to the new filing's transactions

new filings api

add new filings endpoint (and then hook up slackbot)

load/recreate donor matches

either find a way to load from old campfin or redo

add util to load a single filing

logging

ugh what I'm doing now is a terrible hack, log to datadog.

make status queue

this is currently on the filing, but we want another table for filing import status to keep clutter down and so we can also store filings we decided not to import so we don't re-download them.

make donor admin easier to edit

too wide for laptop screen and donor name too far from the name on the sked a

do not load amended F24s with missing base form

since they don't have an election date, this is resulting in a lot of old IEs. We should fix this, and delete the old ones.

pagination issues

when I paginate from a filing's expenditures, I get all expenditures.

fix amendment filter

not currently working

election year filtering

don't load filings from old election years? or don't display them?

transactions did not deactivate

Trump's 2017 july quarterly filing amendment did not deactivate transactions.

fix text search (and everything else)

everything is broken until we figure out how to get the english dictionaries onto our google cloud postgres instance see https://groups.google.com/forum/#!topic/google-cloud-sql-discuss/PYStzRDIkfU

take everything out of the main loop

everything is currently being done in the scrape_filings management command, make that not happen

add f24s

currently only parsing f3 type forms, add f24s, and add itemization turning off

pull in candidate info

so we can match committees to races

read/write to google cloud

we are currently reading/writing to a static dir which is definitely bad, we should read/write from google cloud storages

looks like removing a donor is not triggering re-save

should be checked and fixed if true

amended filing standardized contribs do not seem to be transferring

fix this

load F5s

some IEs are coming through on F5s.

deal with pac/party cycle to date totals

I don't think we're displaying them anywhere, but that doesn't mean they should be incorrect. To fix:

1. write method to find last odd year filing
2. check whether incoming filing year is even or odd
3. if odd, add cycle totals from last odd year filing to current filing totals.

top donor sort is slow

should we pre-compute totals and put on the donor model?

link from a filing to the admin page to standardize donors

expenditure search

implement it

show SA17s in admin

Priorities got all their money as SA17s(???), we need to be able to standardize

add date range to search options

for contrib and expend, also for IE if possible although those have like 4 date fields.

csv downloads

I do actually use these and would miss them, bring them back.

add committee page

with toplines and a way to get to itemizations quickly

donor standardization

I think we want to do this through the admin

• add donor model
• make it possible to add donor to ScheduleA in admin via an auto-complete
• come up with the easiest possible way to add a new donor
• make a page that shows the top donors
• make a donor detail page

QC spotchecks

need to be done before old campfin dies

error handling

figure out what to do if a filing fails

in races, combine district 00 with district None

these are usually the same

also might want to make a model for district so I can add a note (eg: special election)

failed filings

• we're being inconsistent about our expectations of failed filings (we delete them with the intent to reload in one place but go looking for them in another. Fix that.
• better error handling for unexpected fails
• a place in the UI to see all failed filings

inaugural contribs

lots of requests to search trump inaugural contribs. This shouldn't be part of the core, but if we can one-time import and then make searchable separately, it'll be good for folks.