hackergrrl / github-dependency-crawl

:beetle: Crawl GitHub issues to build a dependency graph

github-dependency-crawl

Crawl GitHub issues to build a dependency graph.

Usage

Let's see what this very repository's dependency tree looks like:

var crawl = require('github-dependency-crawl')

crawl('noffle/github-dependency-crawl', function (err, graph) {
  console.log(graph)
})

It'll look something like this:

{
  'noffle/github-dependency-crawl/2': [ 'noffle/github-dependency-crawl/3' ],
  'noffle/github-dependency-crawl/1': [ 'noffle/github-dependency-crawl/2', 'noffle/github-dependency-crawl/3' ],
  'noffle/github-dependency-crawl/3': [ 'noffle/ipget/18' ],
  'noffle/ipget/18': [ 'ipfs/ipget/24', 'ipfs/ipget/26', 'ipfs/ipget/20', 'ipfs/ipget/21' ],
  'ipfs/ipget/24': [],
  'ipfs/ipget/26': [],
  'ipfs/ipget/20': [],
  'ipfs/ipget/21': []
}

Where keys indicate issues in the graph, and each maps to a list of its dependencies.

API

var crawl = require('github-dependency-crawl')

crawl(opts, cb)

Asynchronously makes many GitHub API requests to crawl a given repository's dependency graph.

To simply get the dependency graph of a repo, opts can be a string of the form "org/repo" for a single repo, or "org" to crawl all issues of all repositories in an organization.

cb is of the form function (err, graph). graph contains an object of the form

{
  issueName: [ issueName ],
  issueName: [ issueName ],
  ...
}

where issueName is of the form org/repo/issue-num (e.g. noffle/latest-tweets/1).

Keys are entries in the dependency graph, and the issues it maps to are its dependencies.

For more flexible use, opts can be an object of the form

{
  repo: 'org/repo' || 'org',
  orgToRepos: function (orgName, cb) { ... },
  repoToGitHubIssues: function (repoName, cb) { ... },
  issueToGitHubIssues: function (issueName, cb) { ... },
  auth: {
    client_id: '...',
    client_secret: '...'
  }
}

repoName will be of the form org/repo and issueName of the form org/repo/issue-num.

auth provides the option to include GitHub API credentials, to be able to make a higher # requests / hour.

By default, the crawler will visit all pages of issues per-repo.

If not supplied, orgToRepos, repoToGitHubIssues and issueToGitHubIssues will default to the built-in functionality of querying the GitHub API. These functions are overwritable here so that the module can a) be easily unit tested, and b) you can crawl your own offline datasets by e.g. substituting github api requests for local filesystem reads.

Install

With npm installed, run

$ npm install github-dependency-crawl

License

ISC

github-dependency-crawl's Issues

Allow users to provide an oauth token or client_id/secret pair

These could be optional keys inside of opts, and would let users access much higher request rates against the GitHub API

test issue

Depends on #2
Depends on #3

Split out issue fetching into its own module

For full notes, see #13.

This means a fetch-github-issues module to do the heavy lifting.

Prior art analysis:

1. github-issues: readable stream + pagination, but no org-level fetching
2. issues-for-url: no pagination; no org fetching
3. ghissues: doesn't look like org fetching is here either
4. github-issues-stream: readable stream; no org fetching

Dependency metadata

Eventually it will be nice to let the pluggable functions associate metadata with dependencies, which will require more structured data. E.g.

{
  'noffle/github-dependency-crawl/12': {
    state: 'closed',
    dependencies: [ ... ]
  }
}

This may also mean more pluggable functions to e.g. decide whether to recursive more deeply (if, say, the issue is resolved).

test issue 2

Depends on #3

test issue 3

just a test issue

Depends on ipfs/ipget#18

other issue data here or elsewhere?

Do you want to use this module to grab other issue data? (status, labels, assignee, etc) Or should this be a separate module?

I would imagine this one to save on requests, but then again, we could use a proxy in between that caches requests or something. so yeah, prob separate, but figured i'd ask.

Repo issue pagination

Currently the module just grabs the first page and bails.

Allow organizations to be queried

I imagine the API for this would just be an invocation of the form

crawl('ipfs', function (err, graph) { ...})

which will query [ 'ipfs/go-ipfs', 'ipfs/js-ipfs', ... ]

Input Github URLs

As a user of this module, it would be really nice to be able to input direct github URLs, like:

// repo
crawl("https://github.com/noffle/github-dependency-crawl", cb)

// single issue
crawl("https://github.com/noffle/github-dependency-crawl/issues/3", cb)

// whole user/org (expensive!)
crawl("https://gitehub.com/noffle", cb)

You can use this regexp to grab out the relevant details, i tested it some:

var re = /^(https?:)?(\/\/)?github\.com\/([^/]+)(\/([^/]+)(\/(issues|pulls?)(\/([0-9]+))?)?)?\/?$/

var getParts = function(s) {
  var g = re.exec(s)
  var obj = {}

  if (g) {
    if (g[3]) obj.user = g[3]
    if (g[5]) obj.repo = g[5]
    if (g[9]) obj.issue = g[9]
  }
  return obj
}

1.4.0 Roadmap

Depends on #4
Depends on #5
Depends on #6

opts bug?

i'm getting this:

/Users/jbenet/git/issue-dep-graph/node_modules/github-dependency-crawl/index.js:181
  if (opts.auth && opts.auth.client_id && opts.auth.client_secret) {
      ^

ReferenceError: opts is not defined
    at Object.issueToGitHubIssue (/Users/jbenet/git/issue-dep-graph/node_modules/github-dependency-crawl/index.js:181:7)
    at issueToDependencyGraph (/Users/jbenet/git/issue-dep-graph/node_modules/github-dependency-crawl/index.js:80:10)
    at reduce (/Users/jbenet/git/issue-dep-graph/node_modules/github-dependency-crawl/index.js:56:9)
    at /Users/jbenet/git/issue-dep-graph/node_modules/github-dependency-crawl/node_modules/async/dist/async.js:2476:13
    at replenish (/Users/jbenet/git/issue-dep-graph/node_modules/github-dependency-crawl/node_modules/async/dist/async.js:884:21)
    at /Users/jbenet/git/issue-dep-graph/node_modules/github-dependency-crawl/node_modules/async/dist/async.js:888:13
    at eachOfLimit (/Users/jbenet/git/issue-dep-graph/node_modules/github-dependency-crawl/node_modules/async/dist/async.js:915:26)
    at /Users/jbenet/git/issue-dep-graph/node_modules/github-dependency-crawl/node_modules/async/dist/async.js:920:20
    at reduce (/Users/jbenet/git/issue-dep-graph/node_modules/github-dependency-crawl/node_modules/async/dist/async.js:2475:9)
    at recursiveResolveGraph (/Users/jbenet/git/issue-dep-graph/node_modules/github-dependency-crawl/index.js:53:5)

looks like opts is not defined here: https://github.com/noffle/github-dependency-crawl/blob/a574b6ffae898d1e72883e8ab7d197d01135cc1f/index.js#L181

Pluggable regex for identifying dependencies

Right now the module hard-codes the regex /^Depends on http/ and then does URL matching on the line. This could be much more powerful if users could plug in their own function for dependency detection, e.g.

var opts = {
  repo: 'noffle/github-dependency-crawl',
  dependencyDetector: function (line) {
    // look for dependencies and return a github URL
  }
}

crawl(opts, ...)