Coder Social home page Coder Social logo

dis_meituanreptile's Introduction

分布式美团外卖小爬虫 ，利用RabbitMQ消息队列

路漫漫其修远兮，吾将上下而求索

小说明

  • 第一次比较大的在github写一个项目，一切都在摸索中。当然这个项目也是第一次接触分布式爬虫。
  • 整体思路是RabbitMQ+多实例+多线程搞定。
  • selenium + Phantomjs
  • 预期目标是，美团数据在2个小时内完全抓取完（估算了一下，，千万级。。。）,这个时间是我瞎定的，美团对于IP的限制太可怕了。
  • IP这里我用的代理是阿布云
  • 具体实现方式和具体处理，还在摸索中......，如有大神路过还望指点....
  • 项目思路目前不是太明朗，还在思考中。有时间了，我把我的思路好好总结写一写

目前进度：

  • 获取美团全国城市列表，并以JSON格式添加到远程RabbitMQ服务器队列
  • 从队列中获取城市列表，解析该城市美团店家分类
  • 添加网页处理类，用来解析各种分类的店铺
  • 完善各种异常提示， 及其处理
  • 添加数据库操作---留个坑明天完成（端午一个人～～默默留在实验室解决坑～～话说这几天坑有点多）
  • 数据库操作安装MySQLdb有坑！有问题的同学可以看一下我的博客 传送门刘鑫的博客
  • 利用mysql数据库感觉有点问题，利用python的mysqldb包来试了试。感觉这TM有点慢啊，，

dis_meituanreptile's People

Contributors

liuxinxx avatar nixuiledoc avatar

Stargazers

avatar avatar avatar avatar HuanZi avatar avatar avatar FoxTheFire avatar avatar avatar Wenqian Mao avatar xujw avatar StayKind avatar gentcode avatar avatar

Watchers

avatar avatar

Forkers

ssky87 xiaohui2856 maxwellwaka radtek bg1cnw

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.