yoppi / jumpy

Jumpy is a versatile web cralwer library.

Jumpy

Jumpy is a versatile web cralwer library. On the other hand run stand-alone in terminal as command.

Install

To use as library:

go get github.com/yoppi/jumpy

Otherwise, to use as command:

go get github.com/yoppi/jumpy/jumpy

Usage

Use as library in your code.

import (
  "github.com/yoppi/jumpy"
  "fmt"
)

func main() {
  jumpy.Crawl("http://golang.org/", map[string]string{}, func(page *jumpy.Page) {
    fmt.Println(page.Url)
  })
}

Use as command in terminal.

$ jumpy --root=http://golang.org/ --command=url

jumpy's Issues

crawl below root path

Fix variable names

Setting user agent

Reduce using memory on Page

Currently, Page has whole parsed Object(gokogiri HtmlDocument). This is huge.
For goal is,

• Page should not have parsed gokogiri object
• Page should have usable attributes in jumpy