elaske / tsp-data Goto Github PK

Should this really be stored in the repository? It might be a good baseline to include with each revision so that less data has to be downloaded on everyone's end. However, it's not really that useful...

The only way to really get all of the historical data from your TSP account is to download the PDFs. On the website, it is only possible to get the data from the last statement.

There are two versions of the TSP statements that have been used. This is the current version.

Currently, it always looks at the current page to check for new data. It shouldn't check if it doesn't expect there to be new data from what it's already loaded.

This means that if it's already got data for today in the dictionary, don't look.
If it doesn't have data for today:

• Make sure it's not a weekend
• Make sure it's not a federal holiday
• Check to see if it's after 7PM EST (when new data is posted)

Sort the data in at least the output files. A change over to an OrderedDict might be warranted:
http://stackoverflow.com/questions/18871217/python-how-to-custom-sort-a-list-of-dict-to-use-in-json-dumps

In order to organize the data from the TSP statements, we should create some classes that will also be used in analysis. These might be based on the mufund repository's classes / representation.

It seems that there is a new way to retrieve data from the TSP share price history site in that it allows you to request a CSV by dates. We should be able to retrieve all of the data as one download instead of having to scrape their site.

Complete the appending of new data that is found from the TSP site.

On 12/26/14, the share price history page only showed 11/18 to 12/24, or 26 lines. This means that the logic that is checking when to terminate the retrieval of new data is fooled: it's looking for 30 items.

We should really compare the date information to what is already in the data structure.

The only way to really get all of the historical data from your TSP account is to download the PDFs. On the website, it is only possible to get the data from the last statement.

There are two versions of the TSP statements that have been used. This is the current version.

The most obvious thing is what is going to be faster. The CSV will have gains in not having to parse as much, but the HTML version might get posted back to us quicker (less server processing time?).

Need to measure this to be sure which is the best option. This will become the default option. The other will stay in the code as a user option.

The TSP website changed from this:

to this:

This should make it easier to download the data, too.

Once the data is available locally, the next step will be to wrap this into a class that packages this functionality so that it takes care of it itself without management from an external script. This will allow another script to easily use this data, and essentially query it as needed.