dive2pro / pilgrim

Bookmarklet and manual webcrawler to aid in web research

Pilgrim by Are.na

Pilgrim is a prototype tool for assisting in web-based research.

This project was initiated with generous support from the Knight Foundation Prototype Fund.

Table of contents

• Installation
• API
• Credits

Installation

1. Install node modules: npm install
2. Install Memcached: brew install memcached
3. Install Redis: brew install redis
4. Create a new .env file and copy contents from .env.example
5. Run the server npm run dev

API

Routes can be found in server/api/index.js

/api/:url

Returns extracted content from a url, along with all inner links.

Example result

Request: /api/http%3A%2F%2Fwww.ribbonfarm.com%2F2016%2F02%2F11%2Fminimum-viable-superorganism%2F

Result:

{
  "html": (stripped html from webpage),
  "title": "Minimum Viable Superorganism",
  "hrefs": [
    "http://www.meltingasphalt.com/tears/",
    "https://en.wikipedia.org/wiki/Reciprocal_altruism",
    "https://en.wikipedia.org/wiki/Free_rider_problem",
    "http://www.ribbonfarm.com/2015/09/29/what-is-the-largest-collective-action-ever/",
    "http://peterturchin.com/blog/2016/01/19/naked-self-interest-is-a-recipe-for-social-dissolution-a-response-to-branko-milanovic/",
    "http://www.bartleby.com/17/1/67.html",
    "http://www.meltingasphalt.com/social-status-down-the-rabbit-hole/",
    "https://en.wikipedia.org/wiki/Competitive_altruism",
    "https://en.wikipedia.org/wiki/Enlightened_self-interest",
    "https://twitter.com/KevinSimler/status/650336275954860033",
    "https://twitter.com/mdc/status/573547255115968514",
    "https://en.wikipedia.org/wiki/Agent_detection",
    "http://billmoyers.com/2014/02/21/anatomy-of-the-deep-state/",
    "http://slatestarcodex.com/2014/09/14/does-class-warfare-have-a-free-rider-problem/",
    "https://en.wikipedia.org/wiki/Trade_association",
    "https://en.wikipedia.org/wiki/FWD.us",
    "http://www.amazon.com/Hierarchy-Forest-Evolution-Egalitarian-Behavior/dp/0674006917",
    "http://www.amazon.com/SuperCooperators-Altruism-Evolution-Other-Succeed/dp/1451626630",
    "http://www.amazon.com/Why-We-Talk-Evolutionary-Evolution/dp/0199276234"
  ],
  "url": "http://www.ribbonfarm.com/2016/02/11/minimum-viable-superorganism/",
  "host": "www.ribbonfarm.com",
  "cached": 1461013525246
}

Credits

Pilgrim is built on the shoulders of giants and we'd like to highlight a few of the open-source projects that made it possible (and enjoyable) to develop:

• readability
• redux
• reselect
• redux-thunk
• bluebird-q