The dnsseeder from gcash

Service bit filtering

Ideally we'd like to support service bit filtering. The way the dns seeders are supposed to handle this is through subdomains. For example x1.seeder.example.com should return only NODE_NETWORK nodes.

Technically I believe you can do this with this application if you run multiple seeders on the same machine. The app supports doing so in a single instance by just passing in multiple configs via the command line at startup.

The downside to doing it this way is it would require a new crawler and memory cache for each seeder (though I suppose the combined memory might not be much higher since the lists are filtered).

The question is do we consider this sufficient for filtering? Or should the app supporting filtering without starting multiple crawlers? If the latter, there's a decent amount of refactoring needed to make that happen. If we don't think the overhead of multiple crawlers is that big of a deal, then I guess we're is fine as is.

Memory Leak

Hey,

i am not sure where it comes from. I ran the seeder for 4 coins it used up 1.2GB ram on windows after 1H13Minutes runtime. Not sure if it is the matter of running it on windows.

I:\Development\go-dnsseeder>go tool pprof ./Analyse/allocs
Type: alloc_space
Time: Sep 10, 2019 at 12:58pm (EDT)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top 20
Showing nodes accounting for 1232.91MB, 93.16% of 1323.44MB total
Dropped 157 nodes (cum <= 6.62MB)
Showing top 20 nodes out of 68
      flat  flat%   sum%        cum   cum%
  377.28MB 28.51% 28.51%   463.28MB 35.01%  github.com/gcash/bchd/wire.(*MsgAddr).BchDecode
  169.01MB 12.77% 41.28%   169.01MB 12.77%  net.JoinHostPort
  165.45MB 12.50% 53.78%   661.37MB 49.97%  github.com/gcash/bchd/wire.ReadMessageWithEncodingN
  120.32MB  9.09% 62.87%   120.32MB  9.09%  bytes.makeSlice
      99MB  7.48% 70.35%       99MB  7.48%  net.IP.String
   86.50MB  6.54% 76.89%    86.50MB  6.54%  github.com/gcash/bchd/wire.readNetAddress
   79.08MB  5.98% 82.86%    80.58MB  6.09%  github.com/gcash/bchd/peer.newPeerBase
   28.50MB  2.15% 85.02%    28.50MB  2.15%  strconv.formatBits
   15.64MB  1.18% 86.20%    15.64MB  1.18%  github.com/gcash/bchd/wire.discardInput
   15.45MB  1.17% 87.36%    77.82MB  5.88%  encoding/json.Marshal
   11.51MB  0.87% 88.23%    11.51MB  0.87%  net.newFD
   10.50MB  0.79% 89.03%   122.09MB  9.23%  main.crawlIP
    9.67MB  0.73% 89.76%     9.67MB  0.73%  encoding/json.(*Decoder).refill
       9MB  0.68% 90.44%    17.32MB  1.31%  encoding/json.compact
    7.50MB  0.57% 91.01%     7.50MB  0.57%  unicode/utf16.Decode
    7.50MB  0.57% 91.57%     7.50MB  0.57%  time.Time.MarshalJSON
       7MB  0.53% 92.10%   668.37MB 50.50%  github.com/gcash/bchd/peer.(*Peer).readMessage
       5MB  0.38% 92.48%    12.50MB  0.94%  github.com/gcash/bchd/wire.WriteMessageWithEncodingN
       5MB  0.38% 92.86%        8MB   0.6%  github.com/gcash/bchd/wire.readMessageHeader
       4MB   0.3% 93.16%    15.01MB  1.13%  net.(*OpError).Error
(pprof)

Total: 1.29GB
ROUTINE ======================== github.com/gcash/bchd/wire.(*MsgAddr).BchDecode in C:\Users\aviat\go\pkg\mod\github.com\gcash\[email protected]\wire\msgaddr.go
  377.28MB   463.28MB (flat, cum) 35.01% of Total
         .          .     68:
         .          .     69:   // Limit to max addresses per message.
         .          .     70:   if count > MaxAddrPerMsg {
         .          .     71:           str := fmt.Sprintf("too many addresses for message "+
         .          .     72:                   "[count %v, max %v]", count, MaxAddrPerMsg)
  336.46MB   336.46MB     73:           return messageError("MsgAddr.BchDecode", str)
   40.81MB    40.81MB     74:   }
         .          .     75:
         .          .     76:   addrList := make([]NetAddress, count)
         .       86MB     77:   //msg.AddrList = make([]*NetAddress, 0, count)
         .          .     78:   // for _, v := range addrList {
         .          .     79:   //      err := readNetAddress(r, pver, &v, true)
         .          .     80:   //      if err != nil {
         .          .     81:   //              return err
         .          .     82:   //      }

Total: 1.29GB
ROUTINE ======================== net.JoinHostPort in c:\go\src\net\ipsock.go
  169.01MB   169.01MB (flat, cum) 12.77% of Total
         .          .    226:// See func Dial for a description of the host and port parameters.
         .          .    227:func JoinHostPort(host, port string) string {
         .          .    228:   // We assume that host is a literal IPv6 address if host has
         .          .    229:   // colons.
         .          .    230:   if bytealg.IndexByteString(host, ':') >= 0 {
      34MB       34MB    231:           return "[" + host + "]:" + port
         .          .    232:   }
     135MB      135MB    233:   return host + ":" + port
         .          .    234:}
         .          .    235:
         .          .    236:// internetAddrList resolves addr, which may be a literal IP
         .          .    237:// address or a DNS name, and returns a list of internet protocol
         .          .    238:// family addresses. The result contains at least one address when
(pprof)

Not sure where it Comes From i am new to go and just started learning might be something simple.

Here are the pprof saves:
memory.zip

Make user agent filter handle version comparisons.

We should be able to put in the filter: >= bchd:0.15.0 so we don't need to continually upgrade the filter every release.

gcash / dnsseeder Goto Github PK

dnsseeder's People

Contributors

Stargazers

Watchers

Forkers

dnsseeder's Issues

Service bit filtering

Memory Leak

Make user agent filter handle version comparisons.

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent