wcong / ants-go Goto Github PK

View Code? Open in Web Editor NEW

363.0 363.0 124.0 146 KB

open source, distributed, restful crawler engine in golang

License: MIT License

Go 100.00%

ants-go's People

Contributors

Stargazers

Watchers

Forkers

catshitaf sinlone zengnotes rouxyang ninjachen cxl008 unclebig no2key wengang285 taoyonggang jarrodcal tennessine zofuthan comdex lihuawu morephp kerasking zhouyige cc272309126 gitter-badger tomkoy vonwenm baiyunping333 zzqss x140cc oblank fxtentacle magicyuuu zqqq msn217 jockchou vizewang alertisme wim666 androidzhaoxiaogang gezidan nwcs xeddmc qibinghua yyhthu wangfakang irecall gyang2007 ngocthanhit jusonalien guanle mackjoner strongc gqlxj1987 jhfnetboy oneactor a6708051 hsd315 nevercgoodbye jash16 aleda benluo allianzcortex wlads cn-chengjiangang jianyongchen solq360 tom-kail lugy90 yb123speed tggo insionng devopsmi lakevilladom githubchenzy doifusd zhangyicou kjfcpua hunny-lh f0cii blueszeng xfstudio yanghahaha phpstudio tianyayoucao cancheung shengjieli s0x06 136772 kenzhaoyihui wliangde jiangshide impress-semirding daniel-007 hilittlecat ms895 yesiming avaisa qbdsoft flytinge zmf173417897 chubin1989 eudemon000 chenliangyfl guyigood

ants-go's Issues

test and alpha

add test func
send it to alpha

docs and test

Test
Docs

cluster

dump result when job is down

as title

add Persistence of ants-go

what if master down,
restart to do what

cookiejar

record cookie and update by server,
multiply cookie client

transport

we can send message by tcp

begin of project

what i should do

test basic component
http
multicast
rpc
html parse

found packages util (conf.go) and main (skiplist.go)

github.com/wcong/ants-go/ants/util

found packages util (conf.go) and main (skiplist.go)

what if one node is down

what if one node is down
reclaim the request of lost node

add proxy for crawler

add proxy setting for crawler

master node manager

how does nodes elect a master node,and each node share the cluster info

write spider

write as much as possible to test ants

first demo

I should make one demo first
should do some basic thing

restructure

Restructure my code
change the entrance of program from node to all action,for example http server , tcp server

muiltiply_spider runtime error

An error has occurred after I run the muiltiply_spider several times.

2015/04/22 14:54:03 http.go:41: get request:/crawl                                        
2015/04/22 14:54:03 http.go:91: start spider: muiltiply_spider                            
2015/04/22 14:54:03 downloader.go:46: start downloader                                    
2015/04/22 14:54:03 scraper.go:47: start scraper                                          
2015/04/22 14:54:03 distributer.go:74: start distributer                                  
2015/04/22 14:54:03 distributer.go:90: muiltiply_spider :distribute: 192.168.206.128:8300 :request: http://www.baidu.com/s?wd=1                                                     
2015/04/22 14:54:03 report.go:80: start reporter                                          
2015/04/22 14:54:04 downloader.go:96: muiltiply_spider depth: 0 download url: http://www.baidu.com/s?wd=1                                                                           
2015/04/22 14:54:14 downloader.go:103: Get http://www.baidu.com/s?wd=1: read tcp 180.76.3.151:80: use of closed network connection                                                  
2015/04/22 14:54:14 scraper.go:91: muiltiply_spider :start to scrapy: http://www.baidu.com/s?wd=1                                                                                   
2015/04/22 14:54:14 scraper.go:95: muiltiply_spiderruntime error: invalid memory address or nil pointer dereference                                                                 
2015/04/22 14:54:15 report.go:101: muiltiply_spider :report request to master: http://www.baidu.com/s?wd=1                                                                          
2015/04/22 14:54:15 report.go:109: stop reporter                                          
2015/04/22 14:54:16 distributer.go:97: stop distributer

native tcp to rpc

I just found rpc is a very good way for node communicate .
so change native tcp to rpc

change package follow organizing-go-code

change package to go way