wapiti08 / crawlerset

The collections for different platforms to apply the python crawler and scrapy to extract information and also present different scraping methods

spider-project

The collections for different platforms to apply the python crawler and scrapy to extract information.

1. Douyu:

Using the scrapy framework to capture pictures from mobile app.

2. acfun spider:

Basic crawler to extract core information or comments about the opera. It provides the function to sort the most hot operas.

3. spider-for-movie:

This project is similar to the "acfun spider", it provides some basic ideas to bypass the block of anti-crawler policy on websites.

4. TenSpider:

The crawler to capture the positions from tencent company, which helps you find the most desired job to apply.

5. Products_Clawer:

The most advanced clawer to grap information for all products in one page to save as json format

6. Google_API_Query:

The script can be used to search with any keywords on google with bypassing ability.

7. Go cocurrency:

go mod tidy
go run go_crawler.go {websites}

Usage about the scrapy framework:

You can check the note.txt for some instructions and attentions.