tools is not running about paramspider

[Yunus96]

Same Issue, Even i'am facing.

[JPJ98]

same

[sushant-kamble]

same here, It is not running

[maverickNerd]

It seems to be happening because webarchive is having issue at that time.
Tool just fetches from wayback machine. When you get this issue, check manually

http://web.archive.org/cdx/search/cdx?url=google.com/*&output=txt&fl=original&collapse=urlkey&page=/ (example for google.com)

[devanshbatham]

Hello ,

Web Archive CDX is blocking the requests sent from ParamSpider , I am working on a workaround , Hope that works !

rgds
Devansh

[maverickNerd]

You can use a useragent header, I tried with curl and it is working fine. So may be curl or Mozilla as useragent will fix this.

[devanshbatham]

Hello, Well it seems tricky , as I am already using "rotating useragent technique" , you can see it in requester.py file (in core folder) , I tried changing useragents with every request but it isn't working (in 15% cases it works tho) , Yes with curl and in browser it works ! I will dig deep into this for sure , rn am kinda busywith some of my personal work rgds Devansh

…

On Mon, Aug 3, 2020, 6:04 PM maverickNerd ***@***.***> wrote: You can use a useragent header, I tried with curl and it is working fine. So may be curl or Mozilla as useragent will fix this. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#41 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGVX7BPM6GYWNLJNUF54JVTR62VEHANCNFSM4O24WCDA> .

[maverickNerd]

why not try with User-Agent: curl/7.54.1
Though today I am not seeing any issue with Paramspider
But In my local copy I tried with curl as useragent and it is working fine:

  user_agent_list = [
    #Chrome
-    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36',
-    'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36',
-    'Mozilla/5.0 (Windows NT 5.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36',
-    'Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36',
-    'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36',
-    'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36',
-    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36',
-    'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36',
-    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36',
-    'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36',
-    #Firefox
-    'Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1)',
-    'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko',
-    'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)',
-    'Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko',
-    'Mozilla/5.0 (Windows NT 6.2; WOW64; Trident/7.0; rv:11.0) like Gecko',
-    'Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko',
-    'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0)',
-    'Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko',
-    'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)',
-    'Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko',
-    'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)',
-    'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)',
-    'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)'
+    'curl/7.54.1'
 ] 

We can have multiple curl agents with different versions.

[devanshbatham]

Hello,

It is working fine now, there were some issues from the web archive's side, I am closing this issue as things are good now !

Thanks

rgds
Devansh