mapoflife / gbifer

Shredding Darwin Core Archives with ferocity, strength, and Cascalog.

What is GBIFer?

GBIFer is a Map of Life project designed for processing the full GBIF index into specialized views for CartoDB. It's written in the Clojure programming language and rides on Cascading and Cascalog for processing "Big Data" on top of Hadoop using MapReduce.

Developing

CartoDB OAuth credentials

GBIFer depends on an authenticated connection to CartoDB. This requires adding the following file in resources/creds.json:

{
  "key": "your_cartodb_oauth_key",
  "secret": "your_cartodb_oauth_secret",
  "user": "your_cartodb_username",
  "password": "your_cartodb_password"
}

gbifer's Issues

Add tax_uuid and loc_uuid to occurrence table.

These fields wont be indexed, but for speed of metadata queries it would be good not to have to hit the tax_loc table given an occ_uuid. These ID's are in the occ table currently in use.

Upload GBIF data to S3

The file is on the dev server:

/mnt/hgfs/Data/mol/gbif/occurrence_20120802.txt

Upload to s3://gbifsource

Remove taxonomy and coordinate fields from occ table

These fields are in the tax and loc tables, respectively.

defbufferop for grabbing all unique points for a given name

Basically given a scientific name, grab all unique/valid points in a mult-point WKT representation.