liuxingming / sinaspider Goto Github PK

View Code? Open in Web Editor NEW

3.2K 3.2K 1.5K 85 KB

新浪微博爬虫（Scrapy、Redis）

Python 100.00%

sinaspider's Introduction

Hi there!

我是幸明，一名爬虫工程师与交易工程师；
忙碌而无为是件多么无趣的事情；
壮年不过三十载，希望能留下一些东西；

sinaspider's People

Contributors

Stargazers

Watchers

Forkers

xieming 15310944349 miminus wlstpgo yappawu riverjiangtao cht2014 zheli fashtimedotcom jimso reticentfat jhshengxy zhiyxu gcybupt ilham1223 projoy sdsy888 dagou xiangbro chenkaigithub andelfin lvchunrong genilyc tinemar hexingwei vici0728 heitaoa guo-guoguo leohazy lmjmt sunbuck toints kaitoulee milesqli dhgrz jtwmyddsgx magic282 695540301 cbywxkj tonywu1991 liwei123o0 sirilanka jerome-wang boristype000 owenzx veterun liweijavakf libcorner wangxiaowang tifcty torans looio qilingu yulanggong ligoudanblabla log4me borye yuansmin lzhuo chengyanglei kobenji summerlove66 xiaopihao haika pois-0n egoole yanguanglan qq509103902 xhhmaster lnky79 home123wx zhugeburu lixuxian jingxichen adevil whwq2012 yeelerg learbre shartoo kaurala dzhuang nanakatta suishaojian hyp528 techstone likantour kongnuanzhao jackrool spirit0lh bobiscool seawaylee mychildhoods chenruimin jzd2010 punkcure rucky2013 dotnetage xiangshen-meng apo-j leyap

sinaspider's Issues

单次没成功获取cookies就跳出报错

我用的国外的VPS 在虚拟机上测试是可以用的但是在VPS上的时候就会单次获取不成功就直接Unhandled error in Deferred: 应该怎么解决

URLError:
2016-08-26 16:22:18 [boto] ERROR: Unable to read instance data, giving up
2016-08-26 16:22:18 [requests.packages.urllib3.connectionpool] INFO: Starting new HTTPS connection (1): login.sina.com.cn
2016-08-26 16:22:18 [requests.packages.urllib3.connectionpool] DEBUG: "POST /sso/login.php?client=ssologin.js(v1.4.15) HTTP/1.1" 200 None
Unhandled error in Deferred:
2016-08-26 16:22:18 [twisted] CRITICAL: Unhandled error in Deferred:

关于速度变慢问题

你好。
我使用SinaSpider1，连续运行24小时后速度会降为原来的1/2，这是为什么呢？

微博的点赞、转发、评论数爬取出错？

在爬取了部分数据后，我发现有些微博的点赞、转发、评论数是错误的，会达到几亿的级别。

自动登录问题

使用https://login.sina.com.cn/sso/login.php?client=ssologin.js(v1.4.15)登录时，每次都提示获取cookie成功，但每次以这个cookie请求微博数据的时候却又重定向到登录页？，有人遇到过么？

setting.py要怎么设置才能抓的快

我在公司抓，大概一天只能抓10万条微博，实在是太慢了。

KeyError: 'pop from an empty set'

各位大神，为什么我运行出现下面的错误啊？
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\scrapy\core\engine.py", line 126, in _next_request
request = next(slot.start_requests)
File "C:\Users\chinchilla77\SinaSpider\Sina_spider1\Sina_spider1\spiders\spiders.py", line 22, in start_requests
ID = self.scrawl_ID.pop()
KeyError: 'pop from an empty set'
新手，求帮助：（

有多个爬虫的时候如何控制每个爬虫的爬取频率

比如我注意到SinaSpider2里拆分为info和tweet两个爬虫去实现，但是用户info不需要每次都更新（只有微博数、粉丝数改变的频率比较高），而tweet需要每隔一段时间看看有没有更新，怎么将不同爬虫设置不同的爬取频率？

我想问一下你单机的时候跑一天不会被封iP么？

我想问一下你单机的时候跑一天不会被封iP么？
也没看到你用代理
你有遇到封IP的情况么
我正在发愁呢

不过我和你的唯一区别就在没用agent

mac 运行 sinaspider3 出错

请教大神们，代码出现以下报错是什么原因，redis启动了，weibo账号也能读取，但获取cookie之前检查redis中是否存在账号就会报错。。。。

/anaconda/bin/python SinaSpider-master/Sina_spider3/launch.py
2017-02-09 18:55:25 [scrapy] INFO: Scrapy 1.0.5 started (bot: ['Sina_spider3'])
2017-02-09 18:55:25 [scrapy] INFO: Optional features available: ssl, http11, boto
2017-02-09 18:55:25 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'Sina_spider3.spiders', 'CONCURRENT_REQUESTS': 1, 'SPIDER_MODULES': ['Sina_spider3.spiders'], 'BOT_NAME': ['Sina_spider3'], 'SCHEDULER': 'Sina_spider3.scrapy_redis.scheduler.Scheduler', 'REDIRECT_ENABLED': False, 'DOWNLOAD_DELAY': 10}
2017-02-09 18:55:25 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState
2017-02-09 18:55:25 [Sina_spider3] DEBUG: Reading URLs from redis list 'Sina_spider3:start_urls'
Unhandled error in Deferred:
2017-02-09 18:55:26 [twisted] CRITICAL: Unhandled error in Deferred:

2017-02-09 18:55:26 [twisted] CRITICAL:
Traceback (most recent call last):
File "/anaconda/lib/python2.7/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
result = g.send(result)
File "/anaconda/lib/python2.7/site-packages/scrapy/crawler.py", line 71, in crawl
self.engine = self._create_engine()
File "/anaconda/lib/python2.7/site-packages/scrapy/crawler.py", line 83, in _create_engine
return ExecutionEngine(self, lambda _: self.stop())
File "/anaconda/lib/python2.7/site-packages/scrapy/core/engine.py", line 68, in init
self.downloader = downloader_cls(crawler)
File "/anaconda/lib/python2.7/site-packages/scrapy/core/downloader/init.py", line 69, in init
self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)
File "/anaconda/lib/python2.7/site-packages/scrapy/middleware.py", line 56, in from_crawler
return cls.from_settings(crawler.settings, crawler)
File "/anaconda/lib/python2.7/site-packages/scrapy/middleware.py", line 34, in from_settings
mw = mwcls.from_crawler(crawler)
File "/Users/yang/Documents/Code/SinaSpider-master/Sina_spider3/Sina_spider3/middleware.py", line 40, in from_crawler
return cls(crawler.settings, crawler)
File "/Users/yang/Documents/Code/SinaSpider-master/Sina_spider3/Sina_spider3/middleware.py", line 36, in init
initCookie(self.rconn, crawler.spider.name)
File "/Users/yang/Documents/Code/SinaSpider-master/Sina_spider3/Sina_spider3/cookies.py", line 66, in initCookie
if rconn.get("%s:Cookies:%s--%s" % (spiderName, weibo[0], weibo[1])) is None: # 'SinaSpider:Cookies:账号--密码'，为None即不存在。
KeyError: 0

Process finished with exit code 0`

File "D:\Programs\Anaconda2\lib\site-packages\six.py", line 599, in iteritems return d.iteritems(**kw) AttributeError: 'list' object has no attribute 'iteritems'

ITEM_PIPELINES = {"Sina_spider2.pipelines.MongoDBPipleline": 300}

关于scrapy-redis分布式问题

您好,我最近遇到scrapy分布式处理的问题,对这个分布式处理总是处理不正确,您可以详细介绍下scrapy分布式中的注意和说明吗十分感谢

需要输入验证码，求楼主解答

现在WAP登录也需要输入验证码了，楼主能否写个获取验证码的模块，或者有其他什么方法？

运行Begin.py报错：'pop from an empty set'

报错如下
2017-04-29 11:04:01 [scrapy.core.engine] ERROR: Error while obtaining start requests
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 127, in _next_request
request = next(slot.start_requests)
File "/Users/voidwalker/Downloads/爬虫和web程序/SinaSpider/Sina_spider1/Sina_spider1/spiders/spiders.py", line 64, in start_requests
ID = self.scrawl_ID.pop()
KeyError: 'pop from an empty set'

单机版出现'list' object has no attribute 'iteritems'

Traceback (most recent call last):
File "Begin.py", line 3, in
cmdline.execute("scrapy crawl sinaSpider".split())
File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 108, in execute
settings = get_project_settings()
File "/usr/local/lib/python2.7/dist-packages/scrapy/utils/project.py", line 60, in get_project_settings
settings.setmodule(settings_module_path, priority='project')
File "/usr/local/lib/python2.7/dist-packages/scrapy/settings/init.py", line 285, in setmodule
self.set(key, getattr(module, key), priority)
File "/usr/local/lib/python2.7/dist-packages/scrapy/settings/init.py", line 260, in set
self.attributes[name].set(value, priority)
File "/usr/local/lib/python2.7/dist-packages/scrapy/settings/init.py", line 55, in set
value = BaseSettings(value, priority=priority)
File "/usr/local/lib/python2.7/dist-packages/scrapy/settings/init.py", line 91, in init
self.update(values, priority)
File "/usr/local/lib/python2.7/dist-packages/scrapy/settings/init.py", line 317, in update
for name, value in six.iteritems(values):
File "/usr/local/lib/python2.7/dist-packages/six.py", line 599, in iteritems
return d.iteritems(**kw)
AttributeError: 'list' object has no attribute 'iteritems'

Fans数量不全的问题

在Information表中有的Id拥有几千粉丝，但是存储的时候只能存几个，应该是翻页有限制，翻了几页之后就翻不下去了。不知道有没有什么比较好的解决方案。谢谢：）

微博分析的url是如何获取的

新手，请勿见怪，见笑

http://weibo.cn/%s/follow（简称一）这个是关注列表页面的url，请问如何得到的。

我自己查看只发现http://m.weibo.cn/page/tpl?containerid=1005052026034205_-_FOLLOWERS（简称二）这个url。

对一进行访问，发现可以通过302跳转到二，请问如何得到一那个url

谢谢！！

请问可不可以限制爬取条件？比如，限定城市和时间段？

SinaSpider3运行出错，[scrapy] ERROR: Error downloading <GET http://weibo.cn/1731838797/info>

爬虫一开始就遇到这个问题，老是说downloading网址的时候出错，有没有人遇到过同样问题的

单机版运行缺少模块问题？

哥们你好，我一运行就提示ImportError: cannot import name CrawlSpider，想问一下CrawlSpider模块是系统自带的吗？还是自己写的，要是自己写的话能不能分享一下？

运行时出现 Crawled (200) <GET https://m.weibo.cn/security/stolenReadOnly> (referer: None)

出现这个报错是新浪封了IP吗

sina_spider1

mongodb结果表中的Follows表和Fans表，_id后面的的1 10 100 101 ...... 这些列是什么意思？后面跟的都是关注人和粉丝的ID吗？

process_response() 函数中有语法错误

红圈的地方没有"location"这个关键词。
200的response如下：
{'Proc_Node': ['web359.mweibo.bx.sinanode.com'], 'Set-Cookie': ['_T_WM=339ff7f97fb5c147c8e465d4e9eb1aaa; expires=Fri, 07-Apr-2017 03:27:31 GMT; path=/; domain=.weibo.cn; httponly', 'WEIBOCN_FROM=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=.weibo.cn'], 'Lb_Node': ['layer7-002.mweibo.hk.sinanode.com'], 'Vary': ['Host,Accept-Encoding'], 'X-Log-Uid': ['6039348406'], 'Server': ['Tengine'], 'Date': ['Wed, 08 Mar 2017 03:27:31 GMT'], 'Content-Type': ['text/html; charset=utf-8'], 'Age': ['3']}

另外
raise IgnoreRequest会让scrapy直接忽略这个语法错误，不利于debug。

关于微博反爬机制

虽然说现在微博对爬取的频率做了限制，不过站在微博的角度思考，人家也很纠结，又想让搜索引擎爬取到数据，又要防止其他爬虫对服务器带来压力。

本人尝试过将爬虫对 UA 改为诸如百度爬虫之类的，可以在不模拟登录的情况下，高频率爬取到很多数据。可以把这条建议追加到 README 中。

单机版本运行不成功，求大神救助

/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 /Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py --multiproc --qt-support --client 127.0.0.1 --port 59058 --file /data/apps/codes/weibo/Begin.py
warning: Debugger speedups using cython not found. Run '"/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python" "/Applications/PyCharm.app/Contents/helpers/pydev/setup_cython.py" build_ext --inplace' to build.
pydev debugger: process 8313 is connecting

Connected to pydev debugger (build 163.10154.50)
/data/apps/codes/weibo/Sina_spider1/spiders/spiders.py:4: ScrapyDeprecationWarning: Module `scrapy.spider` is deprecated, use `scrapy.spiders` instead
  from scrapy.spider import CrawlSpider
2017-02-13 19:40:30 [scrapy] INFO: Scrapy 1.2.2 started (bot: Sina_spider1)
2017-02-13 19:40:30 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'Sina_spider1.spiders', 'SPIDER_MODULES': ['Sina_spider1.spiders'], 'LOG_LEVEL': 'INFO', 'DOWNLOAD_DELAY': 2, 'BOT_NAME': 'Sina_spider1'}
2017-02-13 19:40:31 [scrapy] INFO: Enabled extensions:
['scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.corestats.CoreStats']
Get Cookie Success!( Account:17012010793 )
Get Cookies Finish!( Num:1)
2017-02-13 19:40:31 [scrapy] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'Sina_spider1.middleware.UserAgentMiddleware',
 'Sina_spider1.middleware.CookiesMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-02-13 19:40:31 [scrapy] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-02-13 19:40:31 [scrapy] INFO: Enabled item pipelines:
['Sina_spider1.pipelines.MongoDBPipleline']
2017-02-13 19:40:31 [scrapy] INFO: Spider opened
2017-02-13 19:40:31 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-02-13 19:40:32 [scrapy] ERROR: Error downloading <GET http://weibo.cn/5235640836/follow>
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/Library/Python/2.7/site-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/Library/Python/2.7/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request
    defer.returnValue((yield download_func(request=request,spider=spider)))
  File "/Library/Python/2.7/site-packages/scrapy/utils/defer.py", line 45, in mustbe_deferred
    result = f(*args, **kw)
  File "/Library/Python/2.7/site-packages/scrapy/core/downloader/handlers/__init__.py", line 65, in download_request
    return handler.download_request(request, spider)
  File "/Library/Python/2.7/site-packages/scrapy/core/downloader/handlers/http11.py", line 60, in download_request
    return agent.download_request(request)
  File "/Library/Python/2.7/site-packages/scrapy/core/downloader/handlers/http11.py", line 285, in download_request
    method, to_bytes(url, encoding='ascii'), headers, bodyproducer)
  File "/Library/Python/2.7/site-packages/twisted/web/client.py", line 1631, in request
    parsedURI.originForm)
  File "/Library/Python/2.7/site-packages/twisted/web/client.py", line 1408, in _requestWithEndpoint
    d = self._pool.getConnection(key, endpoint)
  File "/Library/Python/2.7/site-packages/twisted/web/client.py", line 1294, in getConnection
    return self._newConnection(key, endpoint)
  File "/Library/Python/2.7/site-packages/twisted/web/client.py", line 1306, in _newConnection
    return endpoint.connect(factory)
  File "/Library/Python/2.7/site-packages/twisted/internet/endpoints.py", line 788, in connect
    EndpointReceiver, self._hostText, portNumber=self._port
  File "/Library/Python/2.7/site-packages/twisted/internet/_resolver.py", line 174, in resolveHostName
    onAddress = self._simpleResolver.getHostByName(hostName)
  File "/Library/Python/2.7/site-packages/scrapy/resolver.py", line 21, in getHostByName
    d = super(CachingThreadedResolver, self).getHostByName(name, timeout)
  File "/Library/Python/2.7/site-packages/twisted/internet/base.py", line 276, in getHostByName
    timeoutDelay = sum(timeout)
TypeError: 'float' object is not iterable
2017-02-13 19:40:32 [scrapy] INFO: Closing spider (finished)
2017-02-13 19:40:32 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 1,
 'downloader/exception_type_count/exceptions.TypeError': 1,
 'downloader/request_bytes': 1003,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2017, 2, 13, 11, 40, 32, 108493),
 'log_count/ERROR': 1,
 'log_count/INFO': 7,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2017, 2, 13, 11, 40, 31, 780254)}
2017-02-13 19:40:32 [scrapy] INFO: Spider closed (finished)

Process finished with exit code 0

以上是运行的时候的错误堆栈，大神快出现。

SinaSpider3 去重应该有bug

self.server.setbit(self.key + str(uid / 4000000000), uid % 4000000000, 1)

key = "dupefilter:%s" % int(time.time())

这里key不唯一，去重应该是失败的。

请楼主确认。

运行Sina_spider1出错

刚开始我运行了D:\SinaSpider-master\Sina_spider1\Begin.py（我的路径）没有任何反应，也没有报错
于是我尝试了在命令行进行运行，代码都没有动因为之前有人分享过将twisted改到16以下，于是我也试了，改成了Twisted-16.5.0，可还是不行，跪求大神指点
却出了错误，报错如下：
D:\SinaSpider-master\Sina_spider1>scrapy crawl sinaSpider -s LOG_LEVEL=ERROR
D:\SinaSpider-master\Sina_spider1\Sina_spider1\spiders\spiders.py:4: ScrapyDeprecationWarning: Module scrapy.spider is deprecated, use scrapy.spiders instead
from scrapy.spider import CrawlSpider
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5235640836/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5235640836/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5235640836>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5235640836/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5676304901/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5676304901/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5676304901>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5676304901/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5871897095/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5871897095/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5871897095>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5871897095/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2139359753/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2139359753/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=2139359753>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2139359753/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5579672076/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5579672076/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5579672076>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5579672076/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2517436943/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2517436943/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=2517436943>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2517436943/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5778999829/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5778999829/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5778999829>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5778999829/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5780802073/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5780802073/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5780802073>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5780802073/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2159807003/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2159807003/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=2159807003>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2159807003/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/3378940452/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/3378940452/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=3378940452>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/3378940452/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/1885080105/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/1885080105/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=1885080105>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/1885080105/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5778836010/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5778836010/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5778836010>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5778836010/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5762793904/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5762793904/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5762793904>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5762793904/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5722737202/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5722737202/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5722737202>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5722737202/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/3105589817/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/3105589817/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/3105589817/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=3105589817>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5882481217/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5882481217/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5882481217>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5882481217/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5831264835/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5831264835/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5831264835/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5831264835>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/1756807885/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/1756807885/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=1756807885>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/1756807885/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/3637185102/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/3637185102/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=3637185102>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/3637185102/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2717354573/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2717354573/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2717354573/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=2717354573>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/1934363217/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/1934363217/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=1934363217>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/1934363217/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5336500817/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5336500817/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5336500817>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5336500817/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/1431308884/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/1431308884/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=1431308884>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/1431308884/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5818747476/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5818747476/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5818747476>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5073111647/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5818747476/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5073111647/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5073111647/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5073111647>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5398825573/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5398825573/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5398825573/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5398825573>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2501511785/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2501511785/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=2501511785>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2501511785/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range

爬虫出现302错误

你好,之前爬虫都没有问题,后来有一次DOWNLOAD_DELAY设为0了,爬了半个小时就出现302错误,错误如下:
Redirecting (302) to GET http://m.weibo.cn/security from GET http://weibo.cn/2139359753/fans
Redirecting (302) to GET http://m.weibo.cn/security from GET http://weibo.cn/attgroup/opening?uid=2139359753
一开始以为是ip被封了,但是好几天都没好,而且直接用浏览器访问
http://weibo.cn/2139359753/fans
又都能打开,请问是什么问题呢?

捣鼓了一天终于能运行，踩过的坑跟大家分享

踩过的坑：
a. scrapy版本不对，安装时不要安装自带的，而是用sudo pip install scrapy==1.0.5。如果已经安装自带的，就用sudo pipi uninstall scrapy卸载后，再用sudo pip install scrapy==1.0.5安装。
b.安装mongo后，安装pymongo，然后就可以运行了，不要管之前报的错误。之前看似跟mongo无关的错误，在安装pymongo包后都会消失！
错误类似于：
twisted] CRITICAL: Unhandled error in Deferred:

Traceback (most recent call last):
File "c:\python27\lib\site-packages\scrapy\cmdline.py", line 150, in _run_comm
and
cmd.run(args, opts)
File "c:\python27\lib\site-packages\scrapy\commands\crawl.py", line 57, in run

self.crawler_process.crawl(spname, **opts.spargs)

等等。
c.mongo的管理工具mongoBooster下载地址：http://mongobooster.com/downloads

求助新浪账号

加了代理池配了分布式但是突然发现淘宝没有卖微博账号了求助那里可以购买账号

单机就能爬取1300万？设置了延迟了没有？有点厉害呀

将新爬下来的关注和粉丝ID加入到待爬队列--这里跑的时候并没有实现啊

最后得到的数据只有原始的start url的数据，没有爬他follower的数据啊。

运行一段时间后，会自动正常运行结束。

话说不是一直在往里add和pop么。为啥会运行结束了，大概跑了不到一小时。

单机版没找到什么时候给FollowsItem['follows']赋值

单机版里面parse3回掉没找到FollowsItem["follows"] FansItem["fans"]的赋值，但是程序运行的结果里面确实是有值，不太理解这块

SinaSpider1自动登录要输入验证码的问题

SinaSpider3的cookies文件已经更新了，请问SinaSpider1的cookies文件什么时候能更新一下？或者请问哪个小伙伴有可以输入验证码的SinaSpider1？

单机跑 Sina_spider2 的时候报错Tried to stop a LoopingCall that was " AssertionError: Tried to stop a LoopingCall that was not running.

我在单机跑 Sina_spider2 的时候报错，已经成功获得Cookie，并加载中间件和pipelines ，但是Spider刚open就立刻close了。Sina_spider2能不能再一台机器上跑？Tried to stop a LoopingCall that was " AssertionError: Tried to stop a LoopingCall that was not running.

settings.py

SCHEDULER = 'scrapy_redis.scheduler.Scheduler'
SCHEDULER_PERSIST = True
SCHEDULER_QUEUE_CLASS = 'scrapy_redis.queue.SpiderPriorityQueue'
REDIE_URL = None
REDIS_HOST = 'localhost'
REDIS_PORT = 6379

报错信息

......
2016-07-02 08:15:15 [scrapy] INFO: Enabled item pipelines:
['Sina_spider2.pipelines.MongoDBPipleline']
2016-07-02 08:15:15 [scrapy] INFO: Spider opened
2016-07-02 08:15:15 [scrapy] INFO: Closing spider (shutdown)
Unhandled error in Deferred:
2016-07-02 08:15:15 [twisted] CRITICAL: Unhandled error in Deferred:


Traceback (most recent call last):
  File "/Users/georgezou/Documents/Coding/github/SinaSpider/Sina_spider2/Sina_spider2/commands/crawlall.py", line 37, in run
    self.crawler_process.crawl(spidername, **opts.spargs)
  File "/Library/Python/2.7/site-packages/scrapy/crawler.py", line 163, in crawl
    return self._crawl(crawler, *args, **kwargs)
  File "/Library/Python/2.7/site-packages/scrapy/crawler.py", line 167, in _crawl
    d = crawler.crawl(*args, **kwargs)
  File "/Library/Python/2.7/site-packages/twisted/internet/defer.py", line 1274, in unwindGenerator
    return _inlineCallbacks(None, gen, Deferred())
--- <exception caught here> ---
  File "/Library/Python/2.7/site-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/Library/Python/2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/Library/Python/2.7/site-packages/scrapy/crawler.py", line 87, in crawl
    yield self.engine.close()
  File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 100, in close
    return self._close_all_spiders()
  File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 340, in _close_all_spiders
    dfds = [self.close_spider(s, reason='shutdown') for s in self.open_spiders]
  File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 298, in close_spider
    dfd = slot.close()
  File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 44, in close
    self._maybe_fire_closing()
  File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 51, in _maybe_fire_closing
    self.heartbeat.stop()
  File "/Library/Python/2.7/site-packages/twisted/internet/task.py", line 202, in stop
    assert self.running, ("Tried to stop a LoopingCall that was "
exceptions.AssertionError: Tried to stop a LoopingCall that was not running.
2016-07-02 08:15:15 [twisted] CRITICAL: 
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/Library/Python/2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/Library/Python/2.7/site-packages/scrapy/crawler.py", line 87, in crawl
    yield self.engine.close()
  File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 100, in close
    return self._close_all_spiders()
  File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 340, in _close_all_spiders
    dfds = [self.close_spider(s, reason='shutdown') for s in self.open_spiders]
  File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 298, in close_spider
    dfd = slot.close()
  File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 44, in close
    self._maybe_fire_closing()
  File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 51, in _maybe_fire_closing
    self.heartbeat.stop()
  File "/Library/Python/2.7/site-packages/twisted/internet/task.py", line 202, in stop
    assert self.running, ("Tried to stop a LoopingCall that was "
AssertionError: Tried to stop a LoopingCall that was not running.

”为了你的安全，请输入验证码“ 应该怎么办

返回错误信息“为了你的安全，请输入验证码“，应该怎么解决？

SinaSpider讨论群537549079

借楼主地用一下，我建一个QQ群537549079，大家可以加入讨论一下这个爬微博数据的问题，我现在用分布式的版本，但是有很多问题，希望大家能够一起交流解决问题。

关于windows 平台下ImportError: No module named win32api

老是出现这个问题，要崩溃了。。 stackoverflow 和网上其他地方的方法都用了。。但是没用啊

获取weibo.cn部分的cookies的一点建议

其实可以不用进行验证码操作，受作者启发，可以先登录weibo.com的无验证码入口（微博账号安全里设为常登陆地点可以免验证码），然后直接在phontomjs模拟打开weibo.cn，weibo.cn会是登录状态，这时候获取cookies便可。

由于我自己实现了，代码如下，仅供参考：

def init_phantomjs_driver():
    headers = {
        'Cookie': 'YF-Ugrow-G0=b02489d329584fca03ad6347fc915997; SUB=_2AkMvgPj2dcPxrAFYnPgWyGvkZYpH-jycVZEAAn7uJhMyOhgv7nBSqSVOKynW2PbhU4768kfRGZgNPwXeRA..; SUBP=0033WrSXqPxfM72wWs9jqgMF55529P9D9WWEFXHsNpvgJdQjr1GM.e765JpVF020SKM7e0571hMc',  # 未登录时weibo.com的cookie
    }
    for key, value in headers.items():
        webdriver.DesiredCapabilities.PHANTOMJS['phantomjs.page.customHeaders.{}'.format(key)] = value
    useragent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36'
    webdriver.DesiredCapabilities.PHANTOMJS['phantomjs.page.settings.userAgent'] = useragent

    #   local path refer phantomjs
    driver = webdriver.PhantomJS(executable_path='xxxxxxxphantomjs路径xxxxxxx')
    driver.set_window_size(1366, 768)
    return driver

browser = weibo_auto_handle.init_phantomjs_driver()
    browser.get("http://weibo.com")
    time.sleep(3)
    failure = 0
    while "微博-随时随地发现新鲜事" == browser.title and failure < 5:
        failure += 1
        username = browser.find_element_by_name("username")
        pwd = browser.find_element_by_name("password")
        login_submit = browser.find_element_by_class_name('W_btn_a')
        username.clear()
        username.send_keys(account['usn'])
        pwd.clear()
        pwd.send_keys(account['pwd'])
        login_submit.click()
        time.sleep(5)

        # if browser.find_element_by_class_name('verify').is_displayed():
        #     logging.error("Verify code is needed! (Account: %s)" % account)

    if "我的首页 微博-随时随地发现新鲜事" in browser.title:
        browser.get('http://weibo.cn/')
        cookie = dict()
        if "我的首页" in browser.title:
            for elem in browser.get_cookies():
                cookie[elem["name"]] = elem["value"]
        # p2 = persist_iics.Persist()
        # p2.save_account_cookies(accounts[0][0], cookie, datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"))
        logging.error('Account cookies updated! (Account_id: %s)' % account['usn'])
        return cookie

from Sina_spider1.items import InformationItem, TweetsItem, FollowsItem, FansItem

Sina_spider1.items这个是怎么来的？

from scrapy.spider import CrawlSpider
from scrapy.selector import Selector
from scrapy.http import Request
from Sina_spider1.items import InformationItem, TweetsItem, FollowsItem, FansItem

Traceback (most recent call last):
File "<pyshell#37>", line 1, in
from Sina_spider1.items import InformationItem, TweetsItem, FollowsItem, FansItem
ImportError: No module named Sina_spider1.items
运行出错是什么情况

跑通了，总结下从零开始的部署步骤

安装完python2.7之后，
sudo pip install scrapy==1.0.5
sudo pip install pymongo
sudo pip install pyasn1
sudo pip install scrapy-redis

这四个库一定要装上，尤其是**# scrapy限定1.05版本**，最新版的是会报错的。

如果在单机模式玩分布式版本的话需要改settings.py：
settings.py中加上
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"

不然会报
Failed to instantiate dupefilter class '%s': %s", 'scrapy.dupefilters.RFPDupeFilter

返回403错误怎么办

爬微博用户名更换频率是？每次访问都是随机换用户的么？

Will the middleware change the cookie per spider? (new)

I wonder if the customized CookiesMiddleware bellow will change the cookie per spider. That is, since normally serveral requests are needed for each spider to crawl the target will it (randomly) change the cookie(we know that we provide many cookies for the project containing several spiders) during the this process for each spider?

class CookiesMiddleware(object):
""" 换Cookie """
def process_request(self, request, spider):
    cookie = random.choice(cookies)
    request.cookies = cookie

I'm tailoring your code for my weibo.com crawler. Weibo.com's anti-crawler is way powerful than that of weibo.cn and m.weibo.cn.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.