handcraftedbits / netshot

A simple REST service for taking web page screenshots via Electroshot

Netshot

A simple REST service for taking web page screenshots via Electroshot.

Features

• Based on Electron so you get a recent version of Chromium for web page rendering, including:
  • Webfonts
  • CSS and JavaScript injection
  • Device emulation
  • Network emulation
  • And more...
• A simple REST API for creating, retrieving, and deleting screenshots

Usage

Prerequisites

Linux

You should make sure your Linux installation has a complete X11 distribution. You may also want to install the Microsoft TrueType core fonts as many web pages rely on these fonts (or equivalents).

Headless Usage

When running netshot in a headless manner, you will also need to install Xvfb to provide an in-memory display for Electron.

On Ubuntu systems, the following command can be used to install the minimum set of dependencies required to run netshot in a headless manner:

apt-get install libasound2 libgconf-2-4 libgtk2.0-0 libnss3 libxss1 libxtst6 ttf-mscorefonts-installer xvfb

If you are not using Ubuntu, consult your distribution's package repository for equivalent packages.

Installation

Make sure you have Node.js installed and then install netshot with npm:

npm install -g netshot

Starting the netshot Server

The netshot server can be started by running

netshot -d <directory> -p <port>

Where <directory> is the directory where screenshots will be saved and <port> is the port used to listen for incoming connections. If -d is not specified, <directory> will default to %TEMP%\__netshot on Windows (or /tmp/__netshot on all other platforms). If -p is not specified, <port> will default to 8000.

Headless Usage

When running netshot in a headless manner, you must also create a virtual framebuffer for Electroshot. The recommended way to do this is to start netshot with xvfb-run:

xvfb-run --server-args "-screen 0 640x480x24" netshot

Docker Image

A Docker image for netshot is available. To run, please consult the documentation.

Logging

Netshot uses Bunyan for logging. It is recommended that you install Bunyan and pipe netshot through it for pretty logging output:

netshot | bunyan

REST API

• Retrieve a listing of all screenshots: GET /
• Create one or more screenshots: POST /
• Retrieve a screenshot: GET /:id
• Delete a screenshot: DELETE /:id
• Retrieve a list of emulated devices: GET /devices
• Retrieves a list of emulated networks: GET /networks

GET /

Retrieve a listing of all screenshots.

Response

Response Codes

• 200 if successful.
• 500 if an unexpected error occurs.

Headers

• Content-Type: application/json

Body

An array of objects containing information about all captured screenshots.

Schema

[
  {
    // A unique identifier for the screenshot.
    "id": "string",
    // The URL used to retrieve or delete the screenshot.
    "href": "url"
  }
]

Example

curl http://localhost:8000/

HTTP/1.1 200 OK
Content-Type: application/json
Connection: close

[
  {
    "id": "mTjgVBRPNPswZe4TQV5wpe.png",
    "href": "http://localhost:8000/mTjgVBRPNPswZe4TQV5wpe.png"
  }
]

POST /

Create one or more screenshots.

Request

Headers

• Content-Type: application/json

Body

An object containing information about the screenshot(s) to capture.

Schema

{
  // Optional. The cookie(s) that should be included when loading the webpage.
  "cookie": "string | [ string, ... ]",
  // Optional. The CSS rule(s) that should be injected into the webpage after it is loaded.
  "css": "string | [ string, ... ]",
  // Optional. The amount of time, in milliseconds, that should elapse before a screenshot is taken.  When multiple
  // delays are specified, multiple screenshots will be taken.
  "delay": "integer | [ integer, ... ]"
  // A JSON object describing an emulated device used to load the webpage.  See
  // https://github.com/mixu/chromium-emulated-devices/blob/master/index.json for an example of the object format.
  "device": "object",
  // Optional. The screenshot format to use. Default: png
  "format": "string(jpg | pdf | png)"
  // Optional, but required if width is not specified. The maximum height of the screenshot either in pixels or as a
  // device name.  By default, this will be the minimum height required to capture all the content on the webpage.
  "height": "integer | string",
  // Optional. Used to specify JPEG format settings. Only used if format=jpg.
  "jpg": {
    // Optional. JPEG quality level, expressed as an integer from 0 to 100. Default: 75.
    "quality": "integer"
  },
  // Optional. The JavaScript code that should be injected into the webpage after it is loaded.
  "js": "string | [ string,... ]",
  // Optional, can be either a string or an object.  If a string, this value denotes an emulated network preset (you
  // can find a complete listing of these presets by calling the /networks endpoint).  If an object, this value lets you
  // create a custom emulated network.
  "network": "string",
  "network": {
    // Optional.  The emulated network download speed, in Bps.
    "download": "integer",
    // Optional.  The emulated network latency, in ms.
    "latency": "integer",
    // Optional.  The emulated network upload speed, in Bps.
    "upload" : "integer"
  },
  // Optional. Used to specify PDF format settings. Only used if format=pdf.
  "pdf":
    // Optional. Whether or not CSS backgrounds should be captured.
    "background": "boolean",
    // Optional. The page margins to use.
    "margin": "string(default | minimum | none)",
    // Optional. The page orientation to use.
    "orientation": "string(landscape | portrait)",
    // Optional. The paper size to use.
    "page-size": "string(A3 | A4 | legal | letter | tabloid)"
  },
  // Optional. The CSS selector used to specify the DOM element that should be captured for the screenshot, instead of
  // the entire page.
  "selector": "string",
  // Required. The URL of the webpage to load.
  "url": "string",
  // Optional. The user agent to present when loading the webpage.
  "user-agent": "string",
  // Required, but optional if height is specified. The maximum width of the screenshot either in pixels or as a device
  // name.  By default, this will be the minimum width required to capture all the content on the webpage.
  "width": "integer | string",
  // Optional. The amount to zoom the webpage, expressed as a floating-point number from 1.0 to 3.0 (which corresponds
  // to 100% to 300% zoom).
  "zoom-factor": "number"
}

Consult the Electroshot documentation for additional information on these parameters.

Response

Response Codes

• 201 if successful.
• 500 if an unexpected error occurs.

Headers

• Content-Type: application/json

Body

An array of objects containing information about the captured screenshot(s).

Schema

[
  {
    // A unique identifier for the screenshot.
    "id": "string",
    // The URL used to retrieve or delete the screenshot.
    "href": "url"
  }
]

Examples

Take a single screenshot of www.google.com with the following settings:

• Width of 1920 pixels
• Minimum height required to capture all content
• PNG (default) format

curl -X POST -H "Content-Type: application/json" -d '{ "url": "www.google.com", "width": 1920 }' http://localhost:8000

HTTP/1.1 201 Created
Content-Type: application/json
Content-Length: 91
Connection: close

[
  {
    "id": "fN9au2BsDfrLfc5Z8TVp9T.png",
    "href": "http://dev01:8000/fN9au2BsDfrLfc5Z8TVp9T.png"
  }
]

Take multiple screenshots of www.google.com with the following settings:

• iPhone 6 dimensions
• 500ms delay before first screenshot and 1000ms delay before second screenshot
• JPEG format, maximum quality

curl -X POST -H "Content-Type: application/json" -d '{ "url": "www.google.com", "width": "Apple iPhone 6", "format": "jpg", "jpg": { "quality": 100 }, "delay": [ 500, 1000 ] }' http://localhost:8000

HTTP/1.1 201 Created
Content-Type: application/json
Content-Length: 197
Connection: close

[
  {
    "id":"sprYtfXc4p7yNzCc7nMmEb-1.jpg",
    "href":"http://localhost:8000/sprYtfXc4p7yNzCc7nMmEb-1.jpg"
  },
  {
    "id":"sprYtfXc4p7yNzCc7nMmEb-2.jpg",
    "href":"http://localhost:8000/sprYtfXc4p7yNzCc7nMmEb-2.jpg"
  }
]

Take a screenshot of www.google.com with the following settings:

• Width of 1920 pixels
• Minimum height required to capture all content
• PDF format
• No page margins, letter format

curl -X POST -H "Content-Type: application/json" -d '{ "url": "www.google.com", "width": 1920, "format": "pdf", "pdf": { "margin": "none", "page-size": "letter" } }' http://localhost:8000

HTTP/1.1 201 Created
Content-Type: application/json
Content-Length: 95
Connection: close

[
  {
    "id":"mUaGfeAs4QAke5oQDm4kjU.pdf",
    "href":"http://localhost:8000/mUaGfeAs4QAke5oQDm4kjU.pdf"
  }
]

GET /:id

Retrieve a screenshot.

Request

Parameters

• id: the ID of the screenshot to retrieve.

Response

Response Codes

• 200 if successful.
• 404 if the screenshot cannot be found.

Headers

• Content-Type: application/pdf, image/jpeg, or image/png depending on format; application/json if an error occurs.

Body

Raw binary data containing the screenshot.

Example

curl -o screenshot.pdf http://localhost:8000/mUaGfeAs4QAke5oQDm4kjU.pdf

DELETE /:id

Delete a screenshot.

Request

Parameters

• id: the ID of the screenshot to delete.

Response

Response Codes

• 204 if successful.
• 404 if the screenshot cannot be found.

Headers

• Content-Type: application/json

Example

curl -X DELETE http://localhost:8000/mUaGfeAs4QAke5oQDm4kjU.pdf

GET /devices

Retrieve a list of emulated devices understood by netshot. These devices can be referenced via the height and width properties when creating a screenshot.

Response

Response Codes

• 200 if successful.

Headers

Content-Type: application/json

Example

curl http://localhost:8000/devices

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 10365
Connection: close

{
  "Apple iPhone 4": {
    "type": "phone",
    "width": 320,
    "height": 480,
    "pixel-ratio": 2,
    "user-agent": "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_1 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5"
  },
  "horizontal Apple iPhone 4": {
    "type": "phone",
    "width": 480,
    "height": 320,
    "pixel-ratio": 2,
    "user-agent": "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_1 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5"
  },
  "Etc...": {
  }
}

GET /networks

Retrieve a list of emulated networks understood by netshot. These networks can by referenced via the network property when creating a screenshot.

Response

Response Codes

• 200 if successful.

Headers

Content-Type: application/json

Example

curl http://localhost:8000/networks

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 514
Connection: close

{
  "Offline": {
    "latency": 0,
    "download": 0,
    "upload": 0
  },
  "GPRS": {
    "latency": 500,
    "download": 6400,
    "upload": 6400
  },
  "Etc...": {
  }
}

netshot's Issues

Doc typo: -d and -p are not listed

Add timeout

Some sites with Electroshot seem to hang -- investigate those and also add a timeout feature.

Large screenshot height and delay problems

Hi,
I'm trying to use Netshot (in a container on GCloud) to grab a screenshot of a complicated page with 10k px height and 60s delay - lot's of processing after load. The resulting PNG has about 2-3 MB.

Netshot either completes successfully and returns image url (which then throws 404, while other images work), or says Electron returned null.

When trying in Electroshot on command line, it works fine. I'm unable to test Netshot, since it just tells me "unable to start electroshot" when running it in Windows.

I'd be glad for any pointers to what I can try...

Link to docker-netshot project in documentation