multiple crawl does not crawl all my urls. about puppeteer-cluster

[saiprashanth24]

i will give 100 urls as command line arguments ..
i have printed final count .
But i get only 98 or 99.

[thomasdondorf]

This is your second "low effort" question and you didn't even respond to the answer to your first question. Please format your code and be more precise with your question. Then I'll have a look.

[saiprashanth24]

i formated the code ..i am giving 100 urls as input .i will extract the outlinks from those 100 urls by using document.queryselector .and i will increment the total by one after processing each url..Atlast i print the total.it only prints 99 or 98.i even checked it by printing the input urls after crawling and they also printed only 99 urls ..has anyone had this issue before ?

[saiprashanth24]

I am crawling 100 urls with maximum concurrency =10 ..I launched chrome in headless mode and i noticed that some urls does not load at all .if i manually reload those pages,it works fine..This is why the count shows 98 or 99..

[thomasdondorf]

I had a look at your code (even though it is still not properly formatted). You have this before you increment the variable total:

    if (response.status() == 404) {
        console.log('program encountered error');
        return;
    }

Are you sure the return statement is not the problem here? I'm closing this question for now as I don't think your problem is related to this library. Feel free to reopen, but then you should format your code properly.