schickling / chromeless

🖥 Chrome automation made simple. Runs locally or headless on AWS Lambda.

This project is deprecated in favor for Puppeteer. Thanks to all the contributors who made this project possible.

Chromeless

Chrome automation made simple. Runs locally or headless on AWS Lambda. (See Demo)

Chromeless can be used to...

• Run 1000s of browser integration tests in parallel ⚡️
• Crawl the web & automate screenshots
• Write bots that require a real browser
• Do pretty much everything you've used PhantomJS, NightmareJS or Selenium for before

Examples

• JSON of Google Results: Google for chromeless and get a list of JSON results
• Screenshot of Google Results: Google for chromeless and take a screenshot of the results
• prep: Compile-time prerendering for SPA/PWA (like React, Vue...) instead of server-side rendering (SSR)
• See the full examples list for more

▶️ Try it out

You can try out Chromeless and explore the API in the browser-based demo playground (source).

Contents

1. How it works
2. Installation
3. Usage
4. API Documentation
5. Configuring Development Environment
6. FAQ
7. Contributors
8. Credits
9. Help & Community

How it works

With Chromeless you can control Chrome (open website, click elements, fill out forms...) using an elegant API. This is useful for integration tests or any other scenario where you'd need to script a real browser.

There are 2 ways to use Chromeless

1. Running Chrome on your local computer
2. Running Chrome on AWS Lambda and controlling it remotely

1. Local Setup

For local development purposes where a fast feedback loop is necessary, the easiest way to use Chromeless is by controlling your local Chrome browser. Just follow the usage guide to get started.

2. Remote Proxy Setup

You can also run Chrome in headless-mode on AWS Lambda. This way you can speed up your tests by running them in parallel. (In Graphcool's case this decreased test durations from ~20min to a few seconds.)

Chromeless comes out of the box with a remote proxy built-in - the usage stays completely the same. This way you can write and run your tests locally and have them be executed remotely on AWS Lambda. The proxy connects to Lambda through a Websocket connection to forward commands and return the evaluation results.

Installation

npm install chromeless

Proxy Setup

The project contains a Serverless service for running and driving Chrome remotely on AWS Lambda.

1. Deploy The Proxy service to AWS Lambda. More details here
2. Follow the usage instructions here.

Usage

Using Chromeless is similar to other browser automation tools. For example:

const { Chromeless } = require('chromeless')

async function run() {
  const chromeless = new Chromeless()

  const screenshot = await chromeless
    .goto('https://www.google.com')
    .type('chromeless', 'input[name="q"]')
    .press(13)
    .wait('#resultStats')
    .screenshot()

  console.log(screenshot) // prints local file path or S3 url

  await chromeless.end()
}

run().catch(console.error.bind(console))

Local Chrome Usage

To run Chromeless locally, you need a recent version of Chrome or Chrome Canary installed (version 60 or greater). By default, chromeless will start Chrome automatically and will default to the most recent version found on your system if there's multiple. You can override this behavior by starting Chrome yourself, and passing a flag of launchChrome: false in the Chromeless constructor.

To launch Chrome yourself, and open the port for chromeless, follow this example:

alias canary="/Applications/Google\ Chrome\ Canary.app/Contents/MacOS/Google\ Chrome\ Canary"
canary --remote-debugging-port=9222

Or run Chrome Canary headless-ly:

canary --remote-debugging-port=9222 --disable-gpu --headless

Or run Chrome headless-ly on Windows:

cd "C:\Program Files (x86)\Google\Chrome\Application"
chrome --remote-debugging-port=9222 --disable-gpu --headless

Proxy Usage

Follow the setup instructions here.

Then using Chromeless with the Proxy service is the same as running it locally with the exception of the remote option. Alternatively you can configure the Proxy service's endpoint with environment variables. Here's how.

const chromeless = new Chromeless({
  remote: {
    endpointUrl: 'https://XXXXXXXXXX.execute-api.eu-west-1.amazonaws.com/dev',
    apiKey: 'your-api-key-here',
  },
})

API Documentation

Chromeless constructor options

• new Chromeless(options: ChromelessOptions)

Chromeless methods

• end()

Chrome methods

• goto(url: string, timeout?: number)
• setUserAgent(useragent: string)
• click(selector: string, x?: number, y?: number)
• wait(timeout: number)
• wait(selector: string)
• [wait(fn: (...args: any[]) => boolean, ...args: any[])] - Not implemented yet
• clearCache()
• clearStorage(origin: string, storageTypes: string)
• focus(selector: string)
• press(keyCode: number, count?: number, modifiers?: any)
• type(input: string, selector?: string)
• back() - Not implemented yet
• forward() - Not implemented yet
• refresh() - Not implemented yet
• mousedown(selector: string)
• mouseup(selector: string)
• scrollTo(x: number, y: number)
• scrollToElement(selector: string)
• setHtml(html: string)
• setExtraHTTPHeaders(headers: Headers)
• setViewport(options: DeviceMetrics)
• evaluate<U extends any>(fn: (...args: any[]) => void, ...args: any[])
• inputValue(selector: string)
• exists(selector: string)
• screenshot(selector: string, options: ScreenshotOptions)
• pdf(options?: PdfOptions)
• html()
• cookies()
• cookies(name: string)
• cookies(query: CookieQuery) - Not implemented yet
• allCookies()
• setCookies(name: string, value: string)
• setCookies(cookie: Cookie)
• setCookies(cookies: Cookie[])
• deleteCookies(name: string)
• clearCookies()
• clearInput(selector: string)
• setFileInput(selector: string, files: string | string[])

Configuring Development Environment

Requirements:

• NodeJS version 8.2 and greater

1. Clone this repository
2. Run npm install
3. To build: npm run build

Linking this NPM repository

1. Go to this repository locally
2. Run npm link
3. Go to the folder housing your chromeless scripts
4. Run npm link chromeless

Now your local chromeless scripts will use your local development of chromeless.

FAQ

How is this different from NightmareJS, PhantomJS or Selenium?

The Chromeless API is very similar to NightmareJS as their API is pretty awesome. The big difference is that Chromeless is based on Chrome in headless-mode, and runs in a serverless function in AWS Lambda. The advantage of this is that you can run hundreds of browsers in parallel, without having to think about parallelisation. Running integration Tests for example is much faster.

I'm new to AWS Lambda, is this still for me?

You still can use this locally without Lambda, so yes. Besides that, here is a simple guide on how to set the lambda function up for Chromeless.

How much does it cost to run Chromeless in production?

The compute price is $0.00001667 per GB-s and the free tier provides 400,000 GB-s. The request price is $0.20 per 1 million requests and the free tier provides 1M requests per month.

This means you can easily execute > 100.000 tests for free in the free tier.

Are there any limitations?

If you're running Chromeless on AWS Lambda, the execution cannot take longer than 5 minutes which is the current limit of Lambda. Besides that, every feature that's supported in Chrome is also working with Chromeless. The maximal number of concurrent function executions is 1000. AWS API Limits

Are there commercial options?

Although Chromeless is the easiest way to get started running Chrome on Lambda, you may not have time to build and manage your own visual testing toolkit. Commercial options include:

• Chromatic: Visual snapshot regression testing for Storybook.

Troubleshooting

Error: Unable to get presigned websocket URL and connect to it.

In case you get an error like this when running the Chromeless client:

{ HTTPError: Response code 403 (Forbidden)
    at stream.catch.then.data (/code/chromeless/node_modules/got/index.js:182:13)
    at process._tickDomainCallback (internal/process/next_tick.js:129:7)
  name: 'HTTPError',
...
Error: Unable to get presigned websocket URL and connect to it.

Make sure that you're running at least version 1.19.0 of serverless. It is a known issue, that the API Gateway API keys are not setup correctly in older Serverless versions. Best is to run npm run deploy within the project as this will use the local installed version of serverless.

Resource ServerlessDeploymentBucket does not exist for stack chromeless-serverless-dev

In case the deployment of the serverless function returns an error like this:

  Serverless Error ---------------------------------------

  Resource ServerlessDeploymentBucket does not exist for stack chromeless-serverless-dev

Please check, that there is no stack with the name chromeless-serverless-dev existing yet, otherwise serverless can't correctly provision the bucket.

No command gets executed

In order for the commands to be processed, make sure, that you call one of the commands screenshot, evaluate, cookiesGetAll or end at the end of your execution chain.

Contributors

A big thank you to all contributors and supporters of this repository 💚

Credits

• chrome-remote-interface: Chromeless uses this package as an interface to Chrome
• serverless-chrome: Compiled Chrome binary that runs on AWS Lambda (Azure and GCP soon, too.)
• NightmareJS: We draw a lot of inspiration for the API from this great tool

Help & Community

Join our Slack community if you run into issues or have questions. We love talking to you!

chromeless's Issues

Implement non-promise eval.code

Currently it's not a nice experience when you want to evaluate code, but continue with the method chain. This is an example of tests that I implemented with chromeless. Here it gets pretty clear, that we should have a variant of eval.code that returns the Chromeless instance. One possibility would be to call it directly .code()

export default async (cookies: any[]): Promise<any> => {

  const chromeless = new Chromeless({
    runRemote,
    waitTimeout,
    viewport,
  })

  await chromeless
    .cookies.set(cookies)
    .goto(CONSOLE_URL)
    .wait(3200)
    .wait('a[data-test="sidenav-databrowser"]')
    .click('a[data-test="sidenav-databrowser"]')
    .wait(2200)
    .wait('a[data-test="sidenav-databrowser-model-Post"]')
    .click('a[data-test="sidenav-databrowser-model-Post"]')
    .wait(2200)
    .click('div[class^="NewRowInactive__add"]')
    .type('test url')
    .eval.code(() => document.querySelector('div[data-test="new-row-cell-imageUrl"] input').blur())

  await chromeless
    .click('div[data-test="new-row-cell-description"] > div > div')
    .wait(400)
    .wait('div[data-test="new-row-cell-description"] input')
    .type('some description', 'div[data-test="new-row-cell-description"] input')
    .eval.code(() => document.querySelector('div[data-test="new-row-cell-description"] input').blur())

  await chromeless
    .wait(700)
    .click('button[data-test="add-node"]')
    .wait(300)
    .click('div[data-test="edit-field-imageUrl"]')
    .click('div[data-test="cell-imageUrl"]')
    .type('123')
    .eval.code(() => document.querySelector('input').focus())

  return chromeless
    .press(13)
    .wait(400)
    .click('div[data-test="checkbox-row-0"]')
    .wait(400)
    .click('div[data-test="delete-button"]')
    .wait(400)
    .click('div.button.warning')
    .end()
}

If you're interested in more test examples, here are our console tests: https://github.com/graphcool/console/tree/dev/integration-tests/src/tests

Scripts that trigger pageloads fail

Not seeing a wait-like API for pageload. Might be nice to have chromeless "pause" execution if a pageload happens or retry failed actions when a pageload occurs

Add integration test use case to Examples section in README

Since running integration tests in parallel is one of the great benefits of the project it'd be good to have a small example showing how it can be done. The examples available right now enlighten the uses in automation and scraping, but not so much on tests.

Save images or links on element

Is there any way to save image or link on an element?
Or maybe I can use evaluate() to download them 🤔

ECONNREFUSED

Error: connect ECONNREFUSED 127.0.0.1:9222

?

Form element API's

Things like check, uncheck, select and more for form elements

Implement missing features

• back
• forward
• refresh
• mousedown
• mouseup
• viewport
• cookies
• pdf
• inject
• mouseover

Automatically "wait" for elements to cut down on script noise

If you're writing a test that does a lot of interactive JS, there's going to be a lot of wait-type calls inside of scripts. It might be nice to automatically wait for elements as opposed to having to tell chromeless every time to wait for something.

Revisit screenshot handling

Currently we're uploading to our own Graphcool project - maybe we just want to return the b64 string so people can save it with fs.writeFile on their own

viewport is broken. Change documentation or fix documented solution

Currently viewport parameters has to be set as options like this:

https://chromeless.netlify.com/#src=const%20chromeless%20=%20new%20Chromeless(%7B%20remote:%20true,%20viewport:%20%7Bwidth:%201024,%20height:%20800%7D%20%7D)%0A%0Aconst%20screenshot%20=%20await%20chromeless%0A%20%20.goto('https://www.graph.cool')%0A%20%20.scrollTo(0,%202000)%0A%20%20.screenshot

but the documentation states that you should be calling the viewport method:

https://github.com/graphcool/chromeless/blob/master/docs/api.md#viewportwidth-number-height-number

If you do that, the function will time out.

Implement interactive remote REPL mode

Chrome extension to record session + emit code instructions

Similar to https://github.com/segmentio/daydream

Rewrite README/docs

Cleanup Logging

For example introduce debug

.evaluate should return values

const evalRes = await chromeless
  .goto('https://www.graph.cool')
  .evaluate(() => {
      return new Promise((resolve) => {
          setTimeout(() => {
              resolve('ok');
          }, 10);
      });
  })

console.log(evalRes)

await chromeless.end()

It's possible to return even async stuff as well: https://chromedevtools.github.io/devtools-protocol/tot/Runtime/#method-evaluate. There's a param called awaitPromise to indicate it's async. You can also "assume" it's async by default even if it's not and just return the value immediately.

Missing Fonts

I tried demo page to get Taiwan google homepage screenshot, but the content have encoding problem.

close tabs

when done

ServerlessError: The request signature we calculated does not match the signature you provided.

ServerlessError: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

Replace Ably with AWS IoT

Ideally the entire AWS IoT setup would be contained in the serverless.yml file of the remote serverless stack.

Garbage urls in `goto` hang

Probably want to fail quickly if goto gets a bad URL (also might want it to return a value for the resolved URL in case of redirects)

add tests

Allow setting of `headers`

https://chromedevtools.github.io/devtools-protocol/tot/Network/#method-setExtraHTTPHeaders

Consumers should probably be allowed to set headers for requests (helps with functional tests so that you can auth once and use headers on subsequent requests).

Make timeout of wait() customizable

I have been playing around with chromeless and I noticed, that there is a timeout for the wait() function (https://github.com/graphcool/chromeless/blob/master/src/api.ts#L21).

Would be great to be able to customize that. Not sure if it's better via an options object directly passed to the wait() function or via a global config object passed to the new Chromeless().

I can also make a PR for the chosen option.

Auto viewport resize in headless mode

I'm wondering why viewport size is set to 1440x900 by default when in headless mode. Is it not possible to do this in headless mode too?

Scroll to click position

When clicking on an element that is not in the screen, chromeless should scroll to that position

.coverage API

Chrome exposes a mechanism in which to get coverage for scripts. This type of information is crucial to front-end devs as it exposes unused code-paths. Would love to see an API for this

better solution for click

As the Input.dispatchMouseEvent didn't work for all cases, we added the artificial element.click().
This should have a cleaner solution

More "helper" methods like `attr` and `text`

Evaluate is a good hook for consumers to do what they wish for DOM stuff, but it's probably a good idea to have some sugar methods like attr, text, and html so that folks can write scrapers more easily.

parameterize if new tab should be created

I would like to see the source code of the demo front end of the website

Is this demo open source? I'd like to look at the code of the page, especially the editor section.

https://chromeless.netlify.com

How to retrieve html by selector/xpath

?

.visible API

Ensures that's elements are visible (which is a hard thing to implement in a .evaluate as there's a whole host of things that can make something invisible).

Release TODOs

• Finish readme #13
• process.end() on chrome instance. kill chrome each time for now to avoid zombies
• chrome-plugin option to select which functions to wrap

Optional

• refactor the on.message -> filter topic -> message.toString() stuff into a wrapper, like ably with channels, with builtin topic scopes
• optimise what’s included in the deployment package

.size API

Would be nice to have a size method for changing the browser window size (helpful for screenshots and stuff where users want to capture responsive sites)

Add more examples

Use chrome-launcher to launch Chrome when using Chromeless locally

Use chrome-launcher to launch Chrome when using Chromeless locally. Then we don't have to explain to users that they need to have Chrome running for Chromeless to work locally.

Write tests

Implement inject() and refresh()

These are critical for my tests, and could very well be for others. It may not seem so important, but it is when actual pages don't matter. I test some of my libraries with a single instance on about:blank with inject(); refresh()'ing before each test.

https://github.com/stevenvachon/universal-url/blob/master/test.js#L179-L184

Chromeless docker container

It would be great to quickly spin up a farm of chromeless images to run a test suite in parallel.
Using containers would remove the AWS requirement for scalability.
I'm not a docker expert, just started to evaluate docker-based solution as our test suite has grown significantly in the last couple of months, and we need a more elastic approach.

HTTPS error on Win7

Hi guys,

Just FYI: the demo playground may not work on some Win7 machines because the Root Cisco Cert you're using to sign your certificate with is not known.

I tried opening the same website on a Virtual Ubuntu on the same machine ­- all works perfectly. Not sure that you want to support all dinosaur-age operating systems, but still letting you know :­-)

Investigate periodic timeouts

Running chromeless remotely seems to timeout periodically.

Refactor constructor arguments

• Remote function name should be injectable and not hardcoded
• Rethink existing options (maybe viewport?)

Please add a license

It is unclear what terms this tool is distributed under. Without a license it defaults to being proprietary (All Rights Reserved). Can you please clarify by adding a LICENSE file?

Can't use screenshot after evaluate

I'm trying to evaluate some code before taking a screenshot, but when I do that the screenshot returns a completely white image.

You can reproduce this by going to https://chromeless.netlify.com and replacing line 5 with:

  .evaluate(() => { console.log('Hello') })

Investigate virtual time for deterministically replaying the network stack

resources:

• https://groups.google.com/a/chromium.org/forum/?utm_medium=email&utm_source=footer#!topic/headless-dev/v8GtCwr4X_o
• https://docs.google.com/document/d/1y9KDT_ZEzT7pBeY6uzVt1dgKlwc1OB_vY4NZO1zBQmo/edit

show console.log of evaluate

how can i get console.log from .evaluate() ?
i try to write this code, and i got 'undefined'
what's wrong with my code.

const { Chromeless } = require('chromeless');

async function run() {
  const chromeless = new Chromeless()

  const login = await chromeless
    .cookiesClearAll()
    .goto('http://localhost/project/')
    .type('admin, 'input[id="username"]')
    .type('admin', 'input[id="password"]')
    .click('#submit')
    .wait('#last-loaded')
    .click('#last-loaded')
    .wait('#menu-sidebar')
    .evaluate(() => {
      const Links = [].map.call(
        document.querySelector('.main-nav').querySelectorAll('a'),
        a => a.href
      )
      return JSON.stringify(Links)
    })

  console.log(login)

  await chromeless.end()
}

run()
  .catch(console.error.bind(console))

Support for authenticated proxies

Does this support authenticated proxies? Similar to how nightmarejs has the .authentication method

Finish chromeless-playground

Repo: https://github.com/graphcool/chromeless-playground

Demo: https://chromeless-homepage.netlify.com/#src=const%20chromeless%20=%20new%20Chromeless(%7B%20remote:%20true%20%7D)%0A%0Aconst%20screenshot%20=%20await%20chromeless%0A%20%20.goto('https://www.graph.cool')%0A%20%20.scrollTo(0,%202000)%0A%20%20.screenshot()%0A%0Aconsole.log(screenshot)%0A%0Aawait%20chromeless.end()

• Responsiveness
• Adding a new line should respect indentation
• Redo layout
• Auto detect links and make them clickable
• Add download script button
• Better loading spinner

Document how to actually run the example in the `usage` section

The usage section of the readme shows how to create a file with some code in, how to start chrome headlessly, but not how to actually run the example. Please update the README.

Take screenshot of an element

The screenshot() method returns the full-page screenshot, is there any way to get the screenshot of a DIV or any other element?