Investigate automated table pdf scraping with pdftables about ebola

[chendaniely]

wow this is pretty cool.

the last time I was did table scraping from pdf's I redirected less output to a file and went line by line using regex. It was not pretty... nor fun..

[donpdonp]

what is happening currently for getting new data from PDFs into _data/ csv files? is there any automation and are those scripts in the repo?

[donpdonp]

I just tried Tabula after a tip-off from someone at CodeForPortland last night.
http://tabula.nerdpower.org/

The tables are manually selected using the mouse, but the data in the tables are read into csv automatically. It seems to work fairly well. I'm not sure if thats any better than whats being done already.

[srinivvenkat]

I think these tools work fine, as long as the text in the PDF is readable. I don't think it would help if they are scanned pages (which, at least in the guinea dataset, is a majority). I suspect most such are manually done. (@caitlin, please clarify)

Tried a crude crowdsourced initiative to do it --> bit.ly/ebola_guinea (A few were digitized but, due to lack of incentives/spreading the 'crowd' didn't come). The key reason for using google doc is to lower the tech barrier (no need of github account), and partial contribution (fill a few columns until you're bored).

If anybody has ideas on extending this idea, shoot!

[cmrivers]

I use a combination of Tabula, a few little scripts to reformat the data,
and some manual work. The broad overview is here:
http://www.caitlinrivers.com/blog/the-setup-tools-i-use-to-track-emerging-infectious-diseases.
I keep meaning to make a screencast of my workflow, but...no time.

Each Liberia file takes about 3-5 minutes from download to upload, and
Sierra Leone takes a bit longer.

On Wed, Oct 15, 2014 at 11:28 PM, srinivvenkat [email protected]
wrote:

I suspect most are manually done. (@caitlin https://github.com/Caitlin,
please clarify)

I think these tools work fine, as long as the text in the PDF is readable.
I don't think it would help if they are scanned pages (which, at least in
the guinea dataset, is a majority).

Tried a crude crowdsourced initiative to do it --> bit.ly/ebola_guinea (A
few were digitized but, due to lack of incentives/spreading the 'crowd'
didn't come). The key reason for using google doc is to lower the tech
barrier (no need of github account), and partial contribution (fill a few
columns until you're bored).

If anybody has ideas on extending this idea, shoot!

—
Reply to this email directly or view it on GitHub
https://github.com/cmrivers/ebola/issues/29#issuecomment-59309406.[image:
Web Bug from
https://github.com/notifications/beacon/1302262__eyJzY29wZSI6Ik5ld3NpZXM6QmVhY29uIiwiZXhwaXJlcyI6MTcyOTA0OTI4OCwiZGF0YSI6eyJpZCI6NDQxNzY4NTl9fQ==--d66722f2a22b3ea35926f18e416190cd46d220be.gif]