danmelton / heydan Goto Github PK
View Code? Open in Web Editor NEWInspired by HomeBrew, write a data download/process recipe and then serve it through ElasticSearch
License: MIT License
Inspired by HomeBrew, write a data download/process recipe and then serve it through ElasticSearch
License: MIT License
Need the ability to search through data by specifying a tag and then perhaps entity type and year:
heydan search population --type all --year all
Dan, I found something!
population
american_community_survey_population, 1990 to 2013
decennial_census, 1990 to 2013
heydan search population --type all --year 2015 --type school_district
population
american_community_survey_estimates)population, 2015
Suggest renaming to:
@DaTa is the main variable used in each step
download
return @DaTa
#1, 1, 10, 20, 40
transform
return @DaTa
compile
import
simpler and cleaner methods, make more sense
Generalize update_files method in the script.rb to be leveraged in the
It should look at the column header for the id in the download.csv, and then map accordingly to th file structure needed. At the moment, the code is slightly different in each of the script implementations.
we have a ton of fields that we don't need:
too many descriptions
notes
etc
we can add any additional fields as needed, but get rid of the base number
Add total population for available years from the api:
http://www.census.gov/data/developers/data-sets/decennial-census-data.html
Here are some roadblocks I hit when attempting to install Heydan on a Windows 10 box:
(1) It seems that kgio and unicorn are not compatible with Windows
(2) I couldn’t perform downloads because Ruby returned a certificate error. I found a couple of web pages that had downloadable certificates and suggestions about setting up an SSL_CERT_FILE environment variable but couldn’t get it working. I suspect this could be resolved with more effort.
(3) A more minor issue is that I couldn’t invoke heydan from the command prompt, but could work around this with “ruby heydan”.
(4) Windows versions of Ruby apparently do not support process.fork which seems to be required for sync
Update template.rb.erb with comments and sample code from population example
Currently, the downloads process just checks to see if the download is present or not. If the download file gets updated (new data), then it won't be downloaded. Perhaps we need a version control system? I.e. census_decennial_population_1.csv?
We should create an md5 hash of the download file, and store it in the metadata at upload time. That way, we can also check to see if the file downloaded is exactly the same as the file we need to process.
heydan import
Create ElasticSearch helper HeyDan::ElasticSearch that creates an index, updates mapping and bulk posts json files to an elasticsearch server
Add the data (ids and names) from the Open Civic Identifiers project
https://github.com/opencivicdata/ocd-division-ids/blob/master/identifiers/country-us.csv
as part of the import into elasticsearch, include the source metadata files with linkages
Create the ability to sync a jurisdictions folder to a bucket
Add Birth data at the state and county levels for 1968 to 2013
ftp://ftp.cdc.gov/pub/Health_Statistics/NCHS/Datasets/DVS/natality/
the file names for sources/folder/scripts/ should be reduced to just the variable name, we can scope the script by its containing folder
after we call it, unset the name to prevent writing over
heydan upload
uploads data from the download folder to an s3 bucket
heydan server
starts a sinatra server, checks to see if the data has been processed and files are present in the jurisidictions folder first.
Data is updated over time at the source (i.e. catalogues). How do we surface the need to update from the source (i.e. a next updated at?). Maybe the metadata has a nextUpdate with a date stamp?
Add an md5 hexdigest for ocd-identifiers for elastic search and other types of ids where we can't have nonnumber/string characters
We need to tag and easily surface data sets to the developers. What tagging/data system can we develop for developers to quickly identify and add/grab data to the system?
Pull out mortality data from ftp://ftp.cdc.gov/pub/Health_Statistics/NCHS/Datasets/DVS/mortality/ from 1968 to 2013 for:
age
race
cause of death
Turn Parrellel into Helpers to make it less scary to use/call
HeyDan::Helpers.process_array...
heydan download
Download class that scans the file names in the datasets folder and attempts to download them from an s3 bucket
Census American Community Survey Total Population for 2007 to 2013
http://www.census.gov/data/developers/data-sets/acs-survey-5-year-data.html#notes
Get feedback on the initial implementation
Proposal needed for creating/merging a bunch of variables/years into central csv file
heydan generate [name] --variables population var2 var 3 --entitytype state:kansas --year 1990 --path optional
would generate a csv file for the state of kansas with identifiers and variables as columns
running heydan new name
Copies scripts/template.rb.erb and replaces the filename and class name to scripts/dataset_name.rb
Copies datasets/template.json.erb and replaces the filename to datasets/dataset_name.json
It also checks for name collisions
Add a Thor enabled commandline tools for heydan including, this is just the skeleton and does not include the actual processing:
heydan sync
#Downloads all data sets from the CDN & imports them into elasticsearch.
heydan start
#starts up the webserver for heydan
heydan download
#Downloads all datasets from the CDN
heydan process
#Grab all datasets from original source process them and output into downloads folder. Mostly used to test original download.
heydan import
#Bulk updates elasticsearch from the jurisdictions folder
heydan list
#Output a nicely formatted list of names
heydan upload
#uploads download folder to s3. If you have a heydan.yml file with keys, it will upload all the files to an s3 bucket
heydan new name
#generates new files to add a new dataset datasets/name.json, scripts/name.rb
How do we version scripts? As an example, developer 1 adds a script for a dataset. Developer 2 installs/processes that script. Then developer 1 updates the script, and developer 2 need to run the update. Is there a heydan update *names?
The downloads folder doesn't make a lot of sense...we should rename that to data sets.
Rename datasets to sources, because its really about metadata from the sources
Rename tmp to downloads, b/c we are really just downloading into that folder.
heydan process dataset_name
Create a central HeyDan::Process class with methods:
class HeyDan::Process
def get_data
#this method is used to connect to a source, like an api, ftp or download a csv, and then saves it into the tmp folder.
super
end
def transform_data
#this method can transform data, like, pulling in data from another file and computing new data like trends, sums, etc.
#note, you don't need to do anything here if you don't want to process any data
super
end
def save_data
#this method saves the file into downloads
super
end
def process_data
#this method loops through each item and saves it to the jurisdictions/entity_id
super
end
end
heydan sync
essentially runs the download and import commands
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.