Comments (5)
i will give 100 urls as command line arguments ..
i have printed final count .
But i get only 98 or 99.
from puppeteer-cluster.
This is your second "low effort" question and you didn't even respond to the answer to your first question. Please format your code and be more precise with your question. Then I'll have a look.
from puppeteer-cluster.
i formated the code ..i am giving 100 urls as input .i will extract the outlinks from those 100 urls by using document.queryselector .and i will increment the total by one after processing each url..Atlast i print the total.it only prints 99 or 98.i even checked it by printing the input urls after crawling and they also printed only 99 urls ..has anyone had this issue before ?
from puppeteer-cluster.
I am crawling 100 urls with maximum concurrency =10 ..I launched chrome in headless mode and i noticed that some urls does not load at all .if i manually reload those pages,it works fine..This is why the count shows 98 or 99..
from puppeteer-cluster.
I had a look at your code (even though it is still not properly formatted). You have this before you increment the variable total
:
if (response.status() == 404) {
console.log('program encountered error');
return;
}
Are you sure the return statement is not the problem here? I'm closing this question for now as I don't think your problem is related to this library. Feel free to reopen, but then you should format your code properly.
from puppeteer-cluster.
Related Issues (20)
- Use same URL but diffetent logic on each browser HOT 1
- Concurrency launch: CONCURRENCY_BROWSER definition slightly misleading HOT 3
- Error detection super slow with new Puppeteer versions HOT 1
- Support to new puppetter versions HOT 1
- share the dockerfile I'm using
- Suggestion: Allow pool of already instantiated browser workers
- how to open the progress view and monitoring statistics? HOT 1
- cluster concurrent seems not work HOT 1
- Regarding resource usage HOT 1
- Worker Error getting browser page HOT 1
- How to set args like .launch({ args: [] }) ? HOT 3
- Has anyone managed to use separate data for each browser?
- browser crushing due to "open /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory (2)"
- puppeteer-v22.0.0 breaking changes HOT 2
- concurrent limit?
- Can minConcurrency be supported?
- (Question) Tasks separated into modules for each website
- Cluster.queue() await doesn't work
- Why was this approach chosen?
- Extension doesn't work
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from puppeteer-cluster.