wangshunping / weibo_spider Goto Github PK

graduate project, a weibo spider to find some interesting information such as "In social network , people tend to be happy or sad."

Python 100.00%

weibo_spider's People

Contributors

Stargazers

Watchers

Forkers

heathlee baininghan skyleft sophiasunshine alexknight zzl133 toobit zhuhj89 ththanks impluse007 powerchina youknownothingall sunday2333 fistw fifa0329 siruliu jatwaston zrango enda-z hejibo korterling va13n zzhangpurdue daisyzhouch nevin47 bosszhou andy-liyi lcandhq randyqin fishxxx yunlei cbbing deepxkn hust1994 xuelu520 haolanguo yuchendeng xczswt1993 garftalk augustlong cht2014 jichmi lozybean ljshou bigsw arclabs001 jeffershaw oliverkehl kub88 threezhang sandra1993 bageda lixiaosi33 shuaiyin vikinghu dyream duramen monbiler airair damonguo zorsting lmwshow fengyouchao loongago zexiland silentgrape alancheg killgit vshangxiao jinhang spring3th ghwxx lhcoding sjy-nankai satma34 listenleeson llouice dlzzxyz kangkangv5 pois-0n inspiron221 iversonlang qiyeboy veterun jia66wei alan0001 cherish24 zhyjor xiaobulv liuhao-bupt lix930 likantour hb44 zqy1 dragonlin hzlpy cinderella2017 little-bear jijicanyu liusiye

weibo_spider's Issues

python util.py报错

执行：python util.py,报：
BeautifulSoup([your markup], "lxml")

markup_type=markup_type))
Traceback (most recent call last):
File "util.py", line 134, in
main()
File "util.py", line 120, in main
firstInfo = getUniversityStudent(mainid, school)
File "util.py", line 58, in getUniversityStudent
personalInfo = divlabel[0].next_sibling.get_text('|',strip=True)
IndexError: list index out of range

问题

依赖项不全用了一个mysqldb的包在pypi中找不到而且标准库是pymysql
另外还有一个pylab 也找不到

同时在plot.py line119

        plt.text(1500,60,u"安徽医科大学",color='black',size=36, \
                 horizontalalignment='right',verticalalignment='top')
        plt.text(1500,40 ,u'性别与微博数量的比较',color='.75',size=12,\
                 ha='right',va='top')
        plt.savefig('pic/sex2totalcontent.png')
        plt.close()

这里直接就写的安徽医科大学

如何识别用户是否是安徽医科大的学生

您好，我今年也大四，我想问下您的代码中，id与cookie就是单纯是某个用户的登陆id与cookie吗？
另外重要的想问下，您是如何识别你抓取的用户就全都是安徽医科大的学生呢？

出现报错如何解决

Error: None
Traceback (most recent call last):
File "weiboSpider.py", line 37, in get_username
selector = etree.HTML(html)
File "src/lxml/lxml.etree.pyx", line 3161, in lxml.etree.HTML (src/lxml/lxml.etree.c:82353)
File "src/lxml/parser.pxi", line 1819, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:124533)
File "src/lxml/parser.pxi", line 1707, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:123074)
File "src/lxml/parser.pxi", line 1079, in lxml.etree._BaseParser._parseDoc (src/lxml/lxml.etree.c:117114)
File "src/lxml/parser.pxi", line 573, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:110510)
File "src/lxml/parser.pxi", line 683, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:112276)
File "src/lxml/parser.pxi", line 624, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:111367)
XMLSyntaxError: None
Error: None
Traceback (most recent call last):
File "weiboSpider.py", line 51, in get_user_info
selector = etree.HTML(html)
File "src/lxml/lxml.etree.pyx", line 3161, in lxml.etree.HTML (src/lxml/lxml.etree.c:82353)
File "src/lxml/parser.pxi", line 1819, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:124533)
File "src/lxml/parser.pxi", line 1707, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:123074)
File "src/lxml/parser.pxi", line 1079, in lxml.etree._BaseParser._parseDoc (src/lxml/lxml.etree.c:117114)
File "src/lxml/parser.pxi", line 573, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:110510)
File "src/lxml/parser.pxi", line 683, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:112276)
File "src/lxml/parser.pxi", line 624, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:111367)
XMLSyntaxError: None
Error: None
Traceback (most recent call last):
File "weiboSpider.py", line 86, in get_weibo_info
selector = etree.HTML(html)
File "src/lxml/lxml.etree.pyx", line 3161, in lxml.etree.HTML (src/lxml/lxml.etree.c:82353)
File "src/lxml/parser.pxi", line 1819, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:124533)
File "src/lxml/parser.pxi", line 1707, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:123074)
File "src/lxml/parser.pxi", line 1079, in lxml.etree._BaseParser._parseDoc (src/lxml/lxml.etree.c:117114)
File "src/lxml/parser.pxi", line 573, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:110510)
File "src/lxml/parser.pxi", line 683, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:112276)
File "src/lxml/parser.pxi", line 624, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:111367)
XMLSyntaxError: None

豆瓣链接挂掉了

你readme里面的豆瓣链接挂掉了

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.