Coder Social home page Coder Social logo

node-jobs-collect's Introduction

时间：2017-04-12 23:15:36

一、爬虫的目标

  • 爬取市面上常见的招聘网站，然后把职位，薪资，地点做一个报图展示
  • 智联招聘，51job ，还有专注于IT行业的拉勾网，内退网，Boss直招等

二、爬虫流程分析

  1. 请求招聘网站的地址，或者页面的HTML内容
  2. 获取到HTML后，解析出需要的数据（不同的网站，解析方式不一样）
  3. 保存成到数据库

不同招聘网站解析的方式不一样，

时间：2017-04-13 16:09:05

目前已经实现的

  • 统一化各种请求的参数【不同的网站在配置里面做匹配】
  • 动态加载路由，根据配置生成接口

采集结构优化

目前把 采集到页面的html是单独一个模块，解析数据是单独一个模块，但是整块采集解析流程都集成在接口中， 如果需要做一个定时任务，实时去爬起数据，则需要把（采集-》解析-》入库-》封装数据）这一套整合成一个单独的模块，可以在定时任务中直接调用跑起来。

就是入库操作和要单独抽离出来，可以不放在接口里面进行处理。否则耦合度比较高

2017-04-14 00:04:11

靠谱的互联网招聘网站有哪些？

node-jobs-collect's People

Watchers

fuqiang avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.