phillipmartin / gopassivedns Goto Github PK

review and potentially revamp error handling.

http://stackoverflow.com/questions/24125652/golang-auto-including-extensible-application has some good thoughts.

Avro is a much more compact output format than JSON, whilst being straightforward to convert to/from JSON. It is natively supported in Kafka and has good support for schema evolution.

gopacket's DNS support is thin, and does not include String() methods that cover many common record types. Fix that.

vendor the Go deps.

Due to the log volume from gopassivedns, we'd like to configure it to log to a specific syslog facility to further separate it out from the rest of the system logs. However it seems that,

JSON.

break up main.go, probably into a logging.go, a main.go, a and dns.go.

I'd like to be able to configure the tag that gopassivedns passes to syslog. It appears it's currently hardcoded to /usr/sbin/gopassivedns[12406]:. Ideally, I'd be able to simply have them set as gopassivedns.

Generate docs that work with man, PDF and readthedocs (at least)

Stack Trace:

panic: runtime error: index out of range

goroutine 20 [running]:
panic(0x5ba220, 0xc4200120a0)
        /usr/lib/golang/src/runtime/panic.go:500 +0x1a1
main.initLogEntry(0xc4236eb2f0, 0x4, 0x10, 0xc4236eb310, 0x4, 0x10, 0x0, 0x0, 0x0, 0x0, ...)
        /builddir/build/BUILD/gopassivedns-aa047e156339bc54f46cf6ccc7fa99f24d94c9d2/gocode/src/github.com/phillipmartin/gopassivedns/main.go:157 +0x541
main.handleDns(0xc421450000, 0xc423560180, 0xc42000c1e0, 0xc4236eb2f0, 0x4, 0x10, 0xc4236eb310, 0x4, 0x10)
        /builddir/build/BUILD/gopassivedns-aa047e156339bc54f46cf6ccc7fa99f24d94c9d2/gocode/src/github.com/phillipmartin/gopassivedns/main.go:224 +0x59e
main.handlePacket(0xc42000c300, 0xc42000c1e0, 0x29e8d60800, 0xfffffff207b8a800, 0x1, 0x0)
        /builddir/build/BUILD/gopassivedns-aa047e156339bc54f46cf6ccc7fa99f24d94c9d2/gocode/src/github.com/phillipmartin/gopassivedns/main.go:291 +0x423
created by main.doCapture
        /builddir/build/BUILD/gopassivedns-aa047e156339bc54f46cf6ccc7fa99f24d94c9d2/gocode/src/github.com/phillipmartin/gopassivedns/main.go:383 +0x278

We have a few DNS resolvers that handle between 2800-3800 requests per minute (according to the logs gopassivedns generates). It's much more frequent on resolves seeing >7500 requests per minute. It seems that after a while and not consistently the process will die and dump the above stack trace. This is not an issue at all on resolves seeing between 200-700 requests per minute. Happy to provide any other information that might be useful!

Recently had another security tool freaking out and it filled up disk on the machines. But I noticed later that after freeing up space, gopassivedns was running, but not logging anything. If I bounced the service it was fine. Is there a sane way for the logger to recover from not being able to write to the disk?

There are no tests. There should be some tests.

DNS over TCP includes a length header that we should use to ensure we have all the data packets before we parse and log the entries.

logrus includes native output for Kafka (among many others). leverage that to implement the kafka output.

ffjson is good, but JSON marshaling is still the single most expensive operation we deal with.

build in an optional capture stats output stream. something that works with statsd.

use logrus syslog logging.

Handle log file rotation in a sane way (HUP handler perhaps?)

push production build artifacts to...somewhere?

Hello,

just tried using gopassivedns on a Ubuntu 16.04 box. Reading from PCAP works fine, but when I start it on eth0, nothing happens, no output, no errors. gopassivedns version is e879ce4.

./gopassivedns -dev eth0 -debug DEBU[0000] STDOUT logging enabled

Also the interface does not switch into promiscuous mode.

Let me know what I should do to debug this. FWIW, the interface has a second IPv4 address via ip addr I can provide an strace to you privately.

Hi, I just did a fresh install on Ubuntu 16.04.1 LTS on i686 architecture and am getting an error when running. I have 2 monitor/SPAN interfaces, enp5s0f1 and enp9s2. Both interfaces show the same error when running gopassivedns with only the -dev option specifid: FATA[0000] string '' did not parse as a facility

Sometimes it decodes 1 query, but usually it just fails with the error as seen below.

root@ns-mon-1:/gopassivedns# ./gopassivedns -dev enp5s0f1
FATA[0000] string '' did not parse as a facility
root@ns-mon-1:/gopassivedns# ./gopassivedns -dev enp5s0f1
FATA[0000] string '' did not parse as a facility
root@ns-mon-1:/gopassivedns# ./gopassivedns -dev enp5s0f1
FATA[0000] string '' did not parse as a facility
root@ns-mon-1:/gopassivedns# ./gopassivedns -dev enp5s0f1
{"query_id":41507,"response_code":0,"question":"<DOMAIN_REMOVED>","question_type":"A","answer":"<IP_REMOVED>","answer_type":"A","ttl":195,"server":"<IP_REMOVED>","client":"<IP_REMOVED>","timestamp":"2016-12-01T22:43:23Z"}
FATA[0000] string '' did not parse as a facility

root@ns-mon-1:/gopassivedns# ./gopassivedns -dev enp9s2
FATA[0000] string '' did not parse as a facility
root@ns-mon-1:/gopassivedns# ./gopassivedns -dev enp9s2
{"query_id":53889,"response_code":0,"question":"<DOMAIN_REMOVED>","question_type":"A","answer":"<IP_REMOVED>","answer_type":"A","ttl":24,"server":"<IP_REMOVED>","client":"<IP_REMOVED>","timestamp":"2016-12-01T22:49:35Z"}
FATA[0000] string '' did not parse as a facility
root@ns-mon-1:/gopassivedns# ./gopassivedns -dev enp9s2
FATA[0000] string '' did not parse as a facility
root@ns-mon-1:/gopassivedns# ./gopassivedns -dev enp9s2
FATA[0000] string '' did not parse as a facility

root@ns-mon-1:~# uname -a
Linux ns-mon-1 4.4.0-51-generic #72-Ubuntu SMP Thu Nov 24 18:29:17 UTC 2016 i686 i686 i686 GNU/Linux

root@ns-mon-1:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.1 LTS
Release: 16.04
Codename: xenial

Please let me know if there is any other info required to help understand why it's not working as expected.

Thanks

DNS ID is a 16 bit value. In large environments this may not be enough space to prevent collisions. We split the traffic up over an arbitrary number of goroutines, so a simple fix is to spin up more goroutines such that each routine gets fewer values and lowers the chances of collision. A long term solution probably involves adding some other items to the map key.