seratch / gyotaku

Saving complete web pages by using Selenium Web Driver

Gyotaku - 魚拓（ぎょたく）

What's this?

Gyotaku is a simple tool to completely save web pages.

Requirement

• Mac OS/Linux/Windows
• Java Runtime Environment
• Firefox

Usage

Get Gyotaku

Download gyotaku.zip and unzip it.

https://github.com/seratch/gyotaku/downloads

Invoke Gyotaku

Using Gyotaku UI (Swing Application) is the easiest way.

./gyotaku_ui

Authentication

If you want to get a page which requires authentication, use the selenium web driver which is customized by yourself.

input/tumblr-login.scala

Added the following source code:

import org.openqa.selenium._
val driver = new firefox.FirefoxDriver
driver.get("https://www.tumblr.com/login")
driver.findElement(By.id("signup_email")).sendKeys("YOUR_EMAIL")
driver.findElement(By.id("signup_password")).sendKeys("YOUR_PASSWORD")
driver.findElement(By.id("signup_form")).submit()
driver

input/tumblr.yml

name: tumblr-dashbord
url: http://www.tumblr.com/dashboard
driver: { path: input/tumblr-login.scala }

Configuration

name: example
url: http://www.example.com/
driver: input/login_operation.scala
charset: UTF-8
prettify: false
replaceNoDomainOnly: false

name

The name of gyotaku. It'll be used as directory name under output directory.

url

The url to download.

driver

How to create a org.openqa.selenium.WebDriver instance.

FirefoxDriver will be used if it's omitted.

driver
  path: path/to/driver.scala

charset

Charset which is used for the downloaded html and modified css files.

"UTF-8" if it's omitted.

prettify

Modify the html using HtmlCleaner or not.

false if it's omitted.

replaceNoDomainOnly

Replace urls in html/css only when they don't start with 'http://' or 'https://'.

true if it's omitted.

gyotaku's Issues

Proxy setting is not used for img/css download

It's a bug. The reason is that it uses java.net.URL simply for download.

rel="stylesheet" is removed

rel="stylesheet" is removed. Maybe path replacement logic has a bug.

Dropping support for JSON format configuration

Only YAML format will be support since next version(0.1.5) beacause of

• maintenance cost
• jar file size(lift-json will be removed)
• verbosity than YAML format

Replace all the url link to full url with domain

A new feature is requested. They say that it's useful if replacing all the url link to full url with domain is available.

• url(...) in css files should start with 'http'
• links in html should point __local__ directory

Replaced url path is invalid in same domain css files

__local__/css/style.css is copied from __local__/http__example.com/css/style.css.

So if the css file contains replaced relative path, the path is invalid at __local__/css.

Simply replacing relational path for #3 is a too rough solution

This issue is related by issue #3. Replacing relational path simply is a too rough solution...

Maybe Create the other one to place in the __local__ directory when downloading css is fine.

Sometimes failed download same domain contents

There seems to be some patterns though I've not investigated in detail yet.

• Failed to download some directories
• Faild to copy from __local__/http__example.com/css to __local__/css?