Comments (15)
Hello,
I've recently discovered this library and I'm using it with AWS Lambda.
That's my configuration:
const { Cluster } = require('puppeteer-cluster')
const HEADLESS = !!process.env.LAMBDA_TASK_ROOT
const REQ_TIMEOUT = 8000
return Cluster.launch({
concurrency: Cluster.CONCURRENCY_PAGE,
maxConcurrency: 2,
puppeteer: isLambda ? require('puppeteer-core') : require('puppeteer'),
puppeteerOptions: {
args: [
...chromium.args,
'--disable-web-security',
'--disable-dev-profile',
'--font-render-hinting=medium', // could be 'none', 'medium'
'--enable-font-antialiasing'
],
defaultViewport: chromium.defaultViewport,
executablePath: await chromium.executablePath,
headless: HEADLESS,
timeout: REQ_TIMEOUT,
userDataDir: '/dev/null'
}
})
Also, I'm the author of browserless, a top-level wrapper around puppeteer.
I was thinking could be very great to adopt puppeteer-cluster
inside browserless
. Right now, I have a generic pool package, but this library is a better solution since you are handling errors, etc.
The premise of browserless
is to provide sensible good defaults (e.g., the library has a builtin adblocker by default) so just I need to expose maxConcurrency
and concurrency
as part of the shared interface and any user can setup a cluster of instances automagically 🙂
from puppeteer-cluster.
Hi @thomasdondorf , I have been able to get puppeteer-cluster running on AW Lmabda with the intrsuctions provided above by @Kikobeats . However it works only when I set the concurrent to Cluster.CONCURRENCY_PAGE. IN the cases of both Cluster.CONCURRENCY_CONTEXT and Cluster.CONCURRENCY_BROWSER , it doesnt seem to work. It looks like the worker(s) are initialized but fails when it trys to get the browser page
puppeteer-cluster: Worker Error getting browser page (try: 5), message: Protocol error (Target.createTarget): Target closed.
I ran the lambda with DEBUG for puppeteer* and here is the log file from running it.
Can you advise on what I might be doing wrong ?
from puppeteer-cluster.
I have not tried it myself, but if you are able to launch two puppeteer instances in parallel, there should be no problems using the library.
What did you try and what errors did you get?
from puppeteer-cluster.
I've tried to provide chrome executable path from https://github.com/Kikobeats/aws-lambda-chrome and getting this error
UnhandledPromiseRejectionWarning: Error: Unable to get browser page
at Worker.<anonymous> (/usr/local/load_test/node_modules/puppeteer-cluster/dist/Worker.js:43:31)
at Generator.next (<anonymous>)
at fulfilled (/usr/local/load_test/node_modules/puppeteer-cluster/dist/Worker.js:4:58)
at process._tickCallback (internal/process/next_tick.js:68:7)
from puppeteer-cluster.
any update on this @thomasdondorf ?
from puppeteer-cluster.
No. I'm not using this library on AWS Lambda.
In case you want to investigate:
- Does spawning a normal puppeteer instance work (without this library)?
- Does spawning two puppeteer instances in parallel work (without this library?
- What kind of concurrency option are you using?
- What does your code look like?
- What output does the debug log produce?
from puppeteer-cluster.
Hey, @thomasdondorf Thanks for the reply,
Actually, I have moved my project to EC2 due to a deadline but will post try and post updates about your questions soon.
from puppeteer-cluster.
Hi @krunalshah68,
I would love to know if you've managed to run puppeteer cluster on Lambda
from puppeteer-cluster.
I was able to get it working in a lambda with this forked version of puppeteer-cluster: https://github.com/NikolaiT/puppeteer-cluster/ and https://github.com/alixaxel/chrome-aws-lambda
from puppeteer-cluster.
Awesome! Thanks, I'll try it out
from puppeteer-cluster.
That commit/repo you are linking to does not change anything related to running on AWS, it just adds "per browser options". So, if you got it running on AWS with that repo, I am very positive, that you should be able to get it running with this repository. @cmaycumber Maybe you can share your code?
from puppeteer-cluster.
Nice, thank you for sharing your config.
from puppeteer-cluster.
@vidyanand - I'm running into the same problem with Cluster.CONCURRENCY_CONTEXT
. Did you ever get this resolved?
from puppeteer-cluster.
@deldrid1 Nope. I did not. I dont think you can do this with AWS Lambda right now with this package. This article (though not related to pupetter at all) may shed some light on this.
from puppeteer-cluster.
Hi, did anyone succeeded to run this using Cluster.CONCURRENCY_CONTEXT ? I confirm @Kikobeats solution is working fine with Cluster.CONCURRENCY_PAGE
from puppeteer-cluster.
Related Issues (20)
- I think a timeout of `0` should disable timeouts HOT 2
- Clear up Concurrency wording incorrect usage HOT 5
- Feature: Lifetimes
- How To Stop Worker To Become Idle automatically
- Expose stats via prometheus HOT 2
- Screen shot getting stuck forever
- Use same URL but diffetent logic on each browser HOT 1
- Concurrency launch: CONCURRENCY_BROWSER definition slightly misleading HOT 3
- Error detection super slow with new Puppeteer versions HOT 1
- Support to new puppetter versions HOT 1
- share the dockerfile I'm using
- Suggestion: Allow pool of already instantiated browser workers
- how to open the progress view and monitoring statistics? HOT 1
- cluster concurrent seems not work HOT 1
- Regarding resource usage HOT 1
- Worker Error getting browser page HOT 1
- How to set args like .launch({ args: [] }) ? HOT 3
- Has anyone managed to use separate data for each browser?
- browser crushing due to "open /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory (2)"
- puppeteer-v22.0.0 breaking changes HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from puppeteer-cluster.