Comments (11)
把cola目录下的data文件夹删掉再试试。
from cola.
删除之后还是会这样, 错误代码:
get 1898353550 url: http://weibo.com/1898353550/follow
get 3211200050 url: http://weibo.com/3211200050/follow # 之前都没有出错, 但是好像老师这两个ID在重复出现
Error when fetch url: http://weibo.com/1898353550/follow
Error when get bundle: 1898353550
'NoneType' object has no attribute 'find'
Traceback (most recent call last):
File "/home/jiajun/Code/cola/cola/worker/loader.py", line 229, in _execute_bundle
**options).parse()
File "/home/jiajun/Code/cola/contrib/weibo/parsers.py", line 559, in parse
return self._error(url, e)
File "/home/jiajun/Code/cola/contrib/weibo/parsers.py", line 91, in _error
AttributeError: 'NoneType' object has no attribute 'find'
Finish 1898353550
start to get None
Error when fetch url: http://weibo.com/3211200050/follow
Error when get bundle: 3211200050
'NoneType' object has no attribute 'find'
Traceback (most recent call last):
File "/home/jiajun/Code/cola/cola/worker/loader.py", line 229, in _execute_bundle
**options).parse()
File "/home/jiajun/Code/cola/contrib/weibo/parsers.py", line 559, in parse
return self._error(url, e)
File "/home/jiajun/Code/cola/contrib/weibo/parsers.py", line 91, in _error
raise e
AttributeError: 'NoneType' object has no attribute 'find'
Finish 3211200050
start to get None
start to get None
start to get None
start to get None
start to get None
start to get None
start to get None
Finish visiting pages count: 32 # 这里我停止了
Finish visiting pages count: 32
from cola.
你只访问了用户好友页面,有没有抓微博?
from cola.
我是按照wiki里说的做的, python __init__.py
这只是访问了用户好友页面吗?抓微博需要另外的命令吗?
from cola.
那就是没有其他设置了。
你先运行tests/contrib/test_weibo.py看看单元测试能不能通过。
from cola.
可以的
self.browser.set_handle_gzip(True)
.......
----------------------------------------------------------------------
Ran 7 tests in 10.017s
OK
from cola.
我测试了一下,好像确实有问题,不知道是不是这两个用户的页面出现了变化,今天我会看看是什么问题造成的。
from cola.
好的, thank you ;-)
from cola.
besides,如果只是用单机版本,最好使用develop分支,这个分支在单机上已经测试很久了,效率上还是稳定性上都比master要高得多,近期开发完成后我会把它merge到master分支。
from cola.
这个问题是新版微博导致的,已经在master和develop分支下分别修复了,还是像之前说的,最好是用develop分支。
更新代码再试一下。
from cola.
好的 ;-)
from cola.
Related Issues (20)
- json.loads(br.response().read())["data"] HOT 1
- windows下coca无法启动分布式程序 HOT 1
- 遇到执行weibosearch的时候包不存在包问题 HOT 1
- 在CentOS 6中无法运行
- instances设置为大于core个数时,会出问题,过一段时间就会停止爬取了
- 在parser中获取网页html信息时卡住出不来
- 抓取网页出现HTTP ERROR处理问题
- 在抓取过程中突然卡住三四个小时,ctrl C不会退出。应该是mq处理出现问题了 HOT 1
- Failed to save to db, weakly-referenced object no longer exists HOT 2
- ValueError: No JSON object could be decoded HOT 8
- 怎么设置要爬取的用户 HOT 4
- 爬取follow列表的问题 HOT 2
- 爬取新浪微博出错 HOT 3
- 看了下,和上一个issues的log是一样的,应该是mq没有保护好的问题把
- 分布式爬取中,worker的主备mq同步问题
- 任务现场保存问题,任务现场保存在tmp里面,重启pc tmp会被清空
- 不太明白weibo.yaml里面的部分配置,有详细的一对一解释吗? HOT 2
- 还有更新的打算么? HOT 2
- 任务执行完成后为什么始终不退出 HOT 5
- Fix simple typo: falese -> false
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cola.