Would it be possible to run puppeteer-cluster on AWS Lambda? about puppeteer-cluster

[Kikobeats]

Hello,

I've recently discovered this library and I'm using it with AWS Lambda.

That's my configuration:

const { Cluster } = require('puppeteer-cluster')
const HEADLESS = !!process.env.LAMBDA_TASK_ROOT
const REQ_TIMEOUT = 8000

return Cluster.launch({
    concurrency: Cluster.CONCURRENCY_PAGE,
    maxConcurrency: 2,
    puppeteer: isLambda ? require('puppeteer-core') : require('puppeteer'),
    puppeteerOptions: {
      args: [
        ...chromium.args,
        '--disable-web-security',
        '--disable-dev-profile',
        '--font-render-hinting=medium', // could be 'none', 'medium'
        '--enable-font-antialiasing'
      ],
      defaultViewport: chromium.defaultViewport,
      executablePath: await chromium.executablePath,
      headless: HEADLESS,
      timeout: REQ_TIMEOUT,
      userDataDir: '/dev/null'
    }
  })

Also, I'm the author of browserless, a top-level wrapper around puppeteer.

I was thinking could be very great to adopt puppeteer-cluster inside browserless. Right now, I have a generic pool package, but this library is a better solution since you are handling errors, etc.

The premise of browserless is to provide sensible good defaults (e.g., the library has a builtin adblocker by default) so just I need to expose maxConcurrency and concurrency as part of the shared interface and any user can setup a cluster of instances automagically 🙂

[vidyanand]

Hi @thomasdondorf , I have been able to get puppeteer-cluster running on AW Lmabda with the intrsuctions provided above by @Kikobeats . However it works only when I set the concurrent to Cluster.CONCURRENCY_PAGE. IN the cases of both Cluster.CONCURRENCY_CONTEXT and Cluster.CONCURRENCY_BROWSER , it doesnt seem to work. It looks like the worker(s) are initialized but fails when it trys to get the browser page

puppeteer-cluster: Worker Error getting browser page (try: 5), message: Protocol error (Target.createTarget): Target closed.

I ran the lambda with DEBUG for puppeteer* and here is the log file from running it.

Can you advise on what I might be doing wrong ?

[thomasdondorf]

I have not tried it myself, but if you are able to launch two puppeteer instances in parallel, there should be no problems using the library.

What did you try and what errors did you get?

[krunalshah68]

I've tried to provide chrome executable path from https://github.com/Kikobeats/aws-lambda-chrome and getting this error

 UnhandledPromiseRejectionWarning: Error: Unable to get browser page
    at Worker.<anonymous> (/usr/local/load_test/node_modules/puppeteer-cluster/dist/Worker.js:43:31)
    at Generator.next (<anonymous>)
    at fulfilled (/usr/local/load_test/node_modules/puppeteer-cluster/dist/Worker.js:4:58)
    at process._tickCallback (internal/process/next_tick.js:68:7)

[krunalshah68]

any update on this @thomasdondorf ?

[thomasdondorf]

No. I'm not using this library on AWS Lambda.

In case you want to investigate:

• Does spawning a normal puppeteer instance work (without this library)?
• Does spawning two puppeteer instances in parallel work (without this library?
• What kind of concurrency option are you using?
• What does your code look like?
• What output does the debug log produce?

[krunalshah68]

Hey, @thomasdondorf Thanks for the reply,
Actually, I have moved my project to EC2 due to a deadline but will post try and post updates about your questions soon.

[mapi33]

Hi @krunalshah68,
I would love to know if you've managed to run puppeteer cluster on Lambda

[cmaycumber]

I was able to get it working in a lambda with this forked version of puppeteer-cluster: https://github.com/NikolaiT/puppeteer-cluster/ and https://github.com/alixaxel/chrome-aws-lambda

[mapi33]

Awesome! Thanks, I'll try it out

[thomasdondorf]

That commit/repo you are linking to does not change anything related to running on AWS, it just adds "per browser options". So, if you got it running on AWS with that repo, I am very positive, that you should be able to get it running with this repository. @cmaycumber Maybe you can share your code?

[thomasdondorf]

Nice, thank you for sharing your config.

[deldrid1]

@vidyanand - I'm running into the same problem with Cluster.CONCURRENCY_CONTEXT. Did you ever get this resolved?

[vidyanand]

@deldrid1 Nope. I did not. I dont think you can do this with AWS Lambda right now with this package. This article (though not related to pupetter at all) may shed some light on this.

[tuukif]

Hi, did anyone succeeded to run this using Cluster.CONCURRENCY_CONTEXT ? I confirm @Kikobeats solution is working fine with Cluster.CONCURRENCY_PAGE