kevinlynx / dhtcrawler2 Goto Github PK

Any search request gives "Internal Server Error" message. Stats works fine. Mongodb 2.6.1, Erlang 17, Windows 7 32 bit.

I use win7 x64.After running win_start_crawler.bat, it just create two table in Mongodb.No data put in db.So I just want to confirm whether dhtcrawler2 support crawler in intranet. If not, how can I improve it to be that?

voltagex@devbox:~/src/dhtcrawler2$ erl -pa ebin -noshell -s crawler_app start
load file "/home/voltagex/src/dhtcrawler2/ebin/../priv/dhtcrawler.config" success
dhtcrawler startup 6776, 50, "localhost":27017

=INFO REPORT==== 22-Feb-2018::18:18:15 ===
    application: dhtcrawler
    exited: {{shutdown,
                 {failed_to_start_child,hash_cache_writer,
                     {undef,
                         [{mongo_sup,start_pool,
                              [hash_write_db,5,{"localhost",27017}],
                              []},
                          {hash_cache_writer,init,1,
                              [{file,"src/crawler/hash_cache_writer.erl"},
                               {line,37}]},
                          {gen_server,init_it,2,
                              [{file,"gen_server.erl"},{line,365}]},
                          {gen_server,init_it,6,
                              [{file,"gen_server.erl"},{line,333}]},
                          {proc_lib,init_p_do_apply,3,
                              [{file,"proc_lib.erl"},{line,247}]}]}}},
             {crawler_app,start,[normal,[]]}}
    type: temporary

What order do things need to be started in?

I know that dhtcrawler2 search and crawl from random hash generated, but i think submit hash manually feature for user who newly created torrent should be helpful. Thanks

Hi.

I ran crawling process and no infohashes were to be found, no matter how much time I let him run.
I opened a network analyser and I saw several packets were sent in the first phase, and then no packets were on the traffic. I also disabled the firewall.

Is it still working now?

Thank you.

Hello,

Are there any special network requirements in order to get the DHT crawler running?

I opened the 6776 (UDP and TCP) port on my router, but still - after following the instructions on the README (starting crawler/hash/http) and waiting 10 minutes - the crawler has not found anything yet.

Stats:

total 0 torrents

2013-07-10 RecvQuery 0 ProcessedQuery 0 Updated 0 New 0
2013-07-09 RecvQuery 0 ProcessedQuery 0 Updated 0 New 0
2013-07-08 RecvQuery 0 ProcessedQuery 0 Updated 0 New 0

Any ideas?

After letting it run on my Ubuntu computer (with all required dependencies installed) the output is the following:

    2017-01-31 RecvQ 23198 ProcessQ 0 Updated 0 New 0 UniqueQ 177 Filtered 0
    2017-01-30 RecvQ 0 ProcessQ 0 Updated 0 New 0 UniqueQ 0 Filtered 0
    2017-01-29 RecvQ 0 ProcessQ 0 Updated 0 New 0 UniqueQ 0 Filtered 0

This is with the crawler running, the httpd running, and the hash program running

This is the error message:

==> kdht (compile)
/tmp/dhtcrawler2/deps/kdht/src/msg.erl:54: syntax error before: 'query'
/tmp/dhtcrawler2/deps/kdht/src/msg.erl:66: syntax error before: '('
/tmp/dhtcrawler2/deps/kdht/src/msg.erl:76: syntax error before: '('
/tmp/dhtcrawler2/deps/kdht/src/msg.erl:122: syntax error before: '('
/tmp/dhtcrawler2/deps/kdht/src/msg.erl:150: syntax error before: '('
/tmp/dhtcrawler2/deps/kdht/src/msg.erl:9: function announce_peer/5 undefined
/tmp/dhtcrawler2/deps/kdht/src/msg.erl:9: function find_node/3 undefined
/tmp/dhtcrawler2/deps/kdht/src/msg.erl:9: function get_peer/3 undefined
/tmp/dhtcrawler2/deps/kdht/src/msg.erl:9: function ping/2 undefined
/tmp/dhtcrawler2/deps/kdht/src/msg.erl:26: function ping/2 undefined
/tmp/dhtcrawler2/deps/kdht/src/msg.erl:29: function find_node/3 undefined
/tmp/dhtcrawler2/deps/kdht/src/msg.erl:32: function get_peer/3 undefined
/tmp/dhtcrawler2/deps/kdht/src/msg.erl:35: function announce_peer/5 undefined

Great project by the way!

Hello, are there any instructions on how to run this on a linux based OS?

Hello,
This is not really a bug/issue, but there it goes:
What can be done to increase the torrent indexing rate? I am running the latest version of the crawler for 8 hours now and it has only indexed 380 torrents.
The machine is pretty good with a 40 Mbps network connection.
Anything I can do to improve the indexing rate?

I'm trying out DHTcrawler2 and it performs really well - it discovers around 400 thousand unique torrents per day. However, it does not index all of them (merely around 400). Upon examining the logs, I discovered the issue is related to a timed out request to torcache, which I guess is because of a rate limit.

Are you doing anything special on your instance (bt.com) to overcome this limit?

/root/dhtcrawler2/ebin/../priv/dhtcrawler.config" success
dhtcrawler startup 26776, 50, "localhost":27017

=INFO REPORT==== 30-Nov-2016::13:07:07 ===
application: dhtcrawler
exited: {{shutdown,
{failed_to_start_child,hash_cache_writer,
{undef,
[{mongo_sup,start_pool,
[hash_write_db,5,{"localhost",27017}],
[]},
{hash_cache_writer,init,1,
[{file,"src/crawler/hash_cache_writer.erl"},
{line,37}]},
{gen_server,init_it,6,
[{file,"gen_server.erl"},{line,304}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,239}]}]}}},
{crawler_app,start,[normal,[]]}}
type: temporary

Which versions of dependencies are needed to compile & run ?