bosterbuhr / fec-data-wranglin

Dashboard for displaying political donations.

This project is rapidly evolving and the README will be updated soon.

FEC-Data-Wranglin

get_that_data.py pulls data from FEC API clean_that_data.py finds duplicate values from the same column using TF-IDF i.e. "Not Employeed" is replaced by "Not Employed" or "Apple INC" is replaced with "Apple"

If you want to make more than a handful of requests you need an api_key, visit fec.gov to get a key. Set your api_key as an environment variable. i.e. FEC_API_KEY=DEMO_KEY

When you run python3 get_that_data.py in the terminal:

Resulting data will be saved in a csv in FEC-Data-Wranglin/data/raw_data and will be structured as seen below.

Each row contains the information of one donation, the first five columns reference the contributor's information and party is the party of the candidate they donated to.

	contributor_occupation	contributor_employer	contributor_city	contributor_state	contributor_zip	party
0	RETIRED	RETIRED	BENTON	AR	72019	OTH
1	SYSTEMS MANAGER	AVANIR PHARMACEUTICALS	ALISO VIEJO	CA	92656	OTH
2	RETIRED	RETIRED	OSHKOSH	WI	549048984	OTH
3	INSURANCE REP	BRS FINANCIAL GROUP	FRESNO	CA	93701	IND
4	EDITOR	GLOBAL FINANCE MEDIA	GREENLAWN	NY	11740	OTH
5	FALSE	FORMATIV HEALTH	HOOSICK FALLS	NY	12090	OTH

After gathering your data you can try to clean it up a little.

The process I have used for cleaning the data can be found here

The more data you have the better the cleaning will work!

Run python3 clean_that_data.py after python3 get_that_data.py.

You can change the how many times a file is 'cleaned' by adding or removing this line to the if statement in clean_that_data.py:

return_df_as_csv(build_classes(csv, lowest_similarity, ngram_size), saved_file_name)

csv -- data/raw_data/ .csv -- File you want to clean. It must be a .csv and must be in data/raw_data/

lowest_similarity -- float between 0 and 1 -- similarity threshold between two values in a column, values with similarity greater than lowest_similarity will become the same value.

ngram_size -- int (ideally between 2 and 4) -- size of character chunks used to assess similarity. i.e. ngram_size of 3 for similarity: ' si' 'sim' 'imi' 'mil' 'ila' 'lar' 'ari' 'rit' 'ity' 'ty '

fec-data-wranglin's Issues

Party Missing for ActBlue and WinRed

Need to fill in party column when the committee name is "ACTBLUE" or "WINRED"

remove this

FEC-Data-Wranglin/src/data/data_fetcher.py

Line 266 in e0ce967

return self.complete_list

Testing

Added a commit that introduces a bit of unit testing. This is really good practice as well-- both in terms of Python Development and getting used to Git/GitHub

Configure Git

I don't use Macs, but if memory serves correctly, they should come with git, batteries-included. What you're going to want to do is configure it so that you can use your Terminal to interface with GitHub.

You do that by setting up your name/email and (optionally, but highly-recommended) setting up your SSH credentials so you don't have to put in your password each time you make a change

Get the Code Locally

Now what I'd like for you to do pull the code down locally by

1. Just like the first step in the other issue, make a new branch off of big_refactor, called big_refactor_test or something like that.
2. Click the Download button and copy the URL (I don't think there's a huge difference between HTTPS and SSH)

3. Open up the terminal, navigate to a directory where you want to drop this project (let's keep it separate from your existing code for now) and type git clone and then paste this URL. This should pull all of your code down locally.

4. Go into the directory with cd FEC-Data-Wranglin

5. Establish a new working branch with

git checkout -b <your branch name> origin/<your_branch_name>

Essentially what this is doing is:

• checkout: I want to work on the branch
• -b: A new branch, at that
• <your branch name>: that we'll call <your branch name>
• origin/<your_branch_name>: that's based on the code on GitHub (origin) called <your_branch_name>

Onto Python

As far as the project goes, I took this chunk of the code

https://github.com/BOsterbuhr/FEC-Data-Wranglin/blob/big_refactor/src/data/data_fetcher.py#L85-L91

and made it reference two new functions

https://github.com/BOsterbuhr/FEC-Data-Wranglin/blob/big_refactor/src/data/data_fetcher.py#L106-L140

What I'd like you to do is rewrite _handle_recipient_committee_type() such that it:

1. Accepts lower-case input arguments
2. Cleans up the length of the if/elif statements (hint: consider googling "python in list")

Doing so, will allow your code to pass the unit tests I wrote (hence the introduction of conftest.py and tests/ at the root.

To run them:

1. First, install pytest via pip install pytest
2. Then navigate to the root of your project /FEC-Data-Wranglin and type pytest. Doing so at the beginning, will yield a bunch of fails

along with printouts of what the result was vs what the expected value was.

3. Change the code, run the tests, change the code, run the tests, rinse, repeat, until you see a less-ominous pytest message

4. Once it's all good you'll run something that looks like

The final line should give you something "success"-sounding, like

Query speed

The data_fetcher step takes an extraordinary amount of time.
Find a way to process data as it comes in or, improve the speed dramatically.

Deploy

Dockerize!

Update sleep_timer default

Default should be 3600 seconds to equal 1 hour.

Update:
Line 175 sleep_timer: int = 3600
Line 181 sleep_timer: int (default=3600)
Line 201 time.sleep(sleep_timer)

FEC-Data-Wranglin/src/data/data_fetcher.py

Line 175 in 35f80d7

def gimmie_data(self, sleep_timer: int = 360, record_limit: int = None):

FEC-Data-Wranglin/src/data/data_fetcher.py

Line 181 in 35f80d7

sleep_timer: int (default=360)

FEC-Data-Wranglin/src/data/data_fetcher.py

Line 201 in 35f80d7

time.sleep(3)

Update to api_calls_per_hour?

FEC-Data-Wranglin/src/data/data_fetcher.py

Line 202 in e0ce967

self.little_count = 1

This should be updated to api_calls_per_hour right? To reset after the wait time.

Summarize Data

Create dashboard to display:

    - committee_name.value_counts()
    - Total contribution_receipt_amount for:
        1. Each committee_name
        2. Each party
        3. Compare PAC versus direct campaign donations
        4. Top 5 contributor_employers and contributor_occupations

self.pages_pulled

FEC-Data-Wranglin/src/data/data_fetcher.py

Line 188 in e0ce967

while self.pages_pulled < self.total_pages:

self.pages_pulled isn't increased should that happen on line 198?

What is this?

Why is -> str outside the ()?

FEC-Data-Wranglin/src/data/data_fetcher.py

Line 53 in e0ce967

def _make_api_url(

FEC-Data-Wranglin/src/data/data_fetcher.py

Line 54 in e0ce967

two_year_transaction_period: int, recipient_committee_type: str

What is with the -> str?

FEC-Data-Wranglin/src/data/data_fetcher.py

Line 55 in e0ce967

) -> str:

Which file?

Structure wise, where should this go?

FEC-Data-Wranglin/get_that_data.py

Lines 10 to 41 in 5e9adb8

	#### Need to write this into the other file
	# Create instance of data_fetcher class class
	# fec = data_fetcher(starting_url, complete_list)
	# # Inizialize get_the_data to work
	# work = get_the_data(fec)
	# # Starts looping over all transactions
	# work.gimmie_data()
	# # Once complete_list has all transaction we create a pandas dataframe
	# df = pd.DataFrame(
	# complete_list,
	# columns=[
	# "contributor_occupation",
	# "contributor_employer",
	# "contributor_city",
	# "contributor_state",
	# "contributor_zip",
	# "party",
	# ],
	# )
	# Saves DataFrame as serialized object, will import this to ML program
	# Add path and uncommet to save pickled data
	# df.to_pickle("./pickled_data.pkl")
	# Prints DataFrame, be careful, this can get bigggggg
	# print(df)
	# Thanks for reading my Ted Talk

Also,

FEC-Data-Wranglin/get_that_data.py

Line 36 in 5e9adb8

# df.to_pickle("./pickled_data.pkl")

Where would this be saved? /data is my guess.

What is the difference in the /data and /src/data? What is your idea for what goes in /src/analysis?

Ok I think that's it for now, more later I assume.

Stylize input and output

Transfer map.html components to either landing.html or generic.html

Design and implement the summary page

Add this

self.pages_pulled += 1
return self.complete_list

FEC-Data-Wranglin/src/data/data_fetcher.py

Line 198 in e0ce967