mechpen / sockdump

Dump unix domain socket traffic with bpf

sockdump

Dump unix domain socket traffic.

Supports STREAM and DGRAM types.

Requirement

• bcc
• python bcc bindings

Example

string output

$ sudo ./sockdump.py --format string /var/run/docker.sock
waiting for data
19:23:06.633 >>> process docker [31042 -> 13710] len 81(81)
HEAD /_ping HTTP/1.1
Host: docker
User-Agent: Docker-Client/19.03.5 (linux)

19:23:06.633 >>> process dockerd [13710 -> 31042] len 280(280)
HTTP/1.1 200 OK
Api-Version: 1.40
Cache-Control: no-cache, no-store, must-revalidate
Content-Length: 0
Content-Type: text/plain; charset=utf-8
Docker-Experimental: false
Ostype: linux
Pragma: no-cache
Server: Docker/19.03.5 (linux)
Date: Fri, 20 Dec 2019 03:23:06 GMT

19:23:06.633 >>> process docker [31042 -> 13710] len 96(96)
GET /v1.40/containers/json HTTP/1.1
Host: docker
User-Agent: Docker-Client/19.03.5 (linux)

19:23:06.633 >>> process dockerd [13710 -> 31042] len 204(204)
HTTP/1.1 200 OK
Api-Version: 1.40
Content-Type: application/json
Docker-Experimental: false
Ostype: linux
Server: Docker/19.03.5 (linux)
Date: Fri, 20 Dec 2019 03:23:06 GMT
Content-Length: 3

[]
^C
4 packets captured

hexstring output with prefix-based matching

ptp4l + pmc used to show that sockdump can capture from all sockets starting with '/var/run/p', as pmc uses '/var/run/pmc.$PID' socket and talks to ptp4l that listens on '/var/run/ptp4l'.

$ sudo ./sockdump.py '/var/run/p*' --format hexstring
waiting for data
10:11:28.968 >>> process pmc [1108317 -> 0] len 74(74)
\x0d\x12\x00\x4a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xe9\x5d\x00\x00\x04\x7f\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x00\x00\x01\x00\x16\x20\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
10:11:28.968 >>> process ptp4l [896569 -> 0] len 74(74)
\x0d\x12\x00\x4a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x48\x57\xdd\xff\xfe\x07\x93\x21\x00\x00\x00\x00\x04\x7f\x00\x00\x00\x00\x00\x00\x00\x00\xe9\x5d\x00\x00\x02\x00\x00\x01\x00\x16\x20\x00\x01\x00\x00\x01\x80\xf8\xfe\xff\xff\x80\x48\x57\xdd\xff\xfe\x07\x93\x21\x00\x00
^C
2 packets captured

pcap output

$ sudo ./sockdump.py /var/run/docker.sock --format pcap --output dump
waiting for data
^C
8 packets captured
$ wireshark -X lua_script:wireshark/dummy.lua dump

sockdump's Issues

PCAP files have no source and destination

The pcap file output doesn't write a source and destination.

How to distinguish traffic on two different sockets with the same path?

The problem

When using containers and isolated filesystems, two containers may create a socket with the same path, e.g., /var/run/php-fpm.sock. To capture traffic on these sockets, one has to use their paths as seen from container's perspective, i.e., /var/run/php-fpm.sock.

Is it possible to filter for traffic on a specific socket?

Snap

Support full path for socket file name

See details here.

unable to run when --output is connected to syslog

I'm running socklog in a systemd service. By default, systemd connects this to the systemd journal, which works for most services.

I assumed it'd work with hex format too.

However, it fails with a "OSError: [Errno 6] No such device or address: '/dev/stdout'".

Same applies if I explicitly use /proc/self/fd/1.

bpf: Argument list too long. Program too large

Arch Linux
linux 5.17.9.arch1-1
bcc 0.24.0-2
python-bcc 0.24.0-2
clang 14.0.6-1
libbpf 0.8.0-1

$ sudo ./sockdump.py --format string /run/opendkim/opendkim.sock
bpf: Argument list too long. Program  too large (1155 insns), at most 4096 insns

Traceback (most recent call last):
  File "/local/./sockdump.py", line 348, in <module>
    main(args)
  File "/local/./sockdump.py", line 292, in main
    b.attach_kprobe(
  File "/usr/lib/python3.10/site-packages/bcc/__init__.py", line 836, in attach_kprobe
    fn = self.load_func(fn_name, BPF.KPROBE)
  File "/usr/lib/python3.10/site-packages/bcc/__init__.py", line 522, in load_func
    raise Exception("Failed to load BPF program %s: %s" %
Exception: Failed to load BPF program b'probe_unix_socket_sendmsg': Argument list too long

https://stackoverflow.com/questions/70147464/program-too-large-threshold-greater-than-actual-instruction-count
"Program too large" threshold greater than actual instruction count

On Linux 5.15 with a privileged user, the verifier gives up after reading 1 million instructions (the complexity limit). Since it has to analyze all paths through the program, a program with a small number of instructions can have a very high complexity.
...
Since the bpf(2) syscall returns E2BIG both when the program is too large and when its complexity is too high, libbpf prints the same error message for both cases, always with at most 4096 instructions.

The bpf Verifier is a moving target...

Tool fails to run on Kali

Running on Kali 2020.4, I get:

modprobe: FATAL: Module kheaders not found in directory /lib/modules/5.9.0-kali1-amd64
Unable to find kernel headers. Try rebuilding kernel with CONFIG_IKHEADERS=m (module) or installing the kernel development package for your running kernel version.
chdir(/lib/modules/5.9.0-kali1-amd64/build): No such file or directory
Traceback (most recent call last):
  File "./sockdump.py", line 313, in <module>
    main(args)
  File "./sockdump.py", line 261, in main
    b = BPF(text=text)
  File "/usr/lib/python3/dist-packages/bcc/__init__.py", line 364, in __init__
    raise Exception("Failed to compile BPF module %s" % (src_file or "<text>"))
Exception: Failed to compile BPF module <text>

I ran the tool with: sudo ./sockdump.py /path/to/sockfile.sock
Python3 version: Python 3.8.6 (default, Sep 25 2020, 09:36:53) [GCC 10.2.0]

no data captured for kernel 6.0

Add information about what "socket path" is

The problem

usage: sockdump.py [-h] [--seg-size SEG_SIZE] [--segs-per-msg SEGS_PER_MSG] [--segs-in-buffer SEGS_IN_BUFFER] [--format {hex,hexstring,string,pcap}] [--output OUTPUT] sock

The <sock> description is not detailed enough. It can't be any path leading to the socket, it seems to must be the path the process which created the socket used. This confused me for a while, as I'm using Docker & bind mounts to expose socket files to the host system from containers.

Soft links (and probably hard links) wouldn't work either.

Example

The socket created in the container has the path /var/run/php-fpm.sock, which is mapped to the path /opt/app/var/run/php-fpm.sock in the host system. When sniffing for traffic using sockdump.py /opt/app/var/run/php-fpm.sock no traffic is reported (i.e. only waiting for data is shown). sockdump.py /var/run/php-fpm.sock works fine though, even though the /var/run/php-fpm.sock socket doesn't exist from the host system perspective.

Error: cannot take the address of an rvalue of type

/virtual/main.c:57:12: error: cannot take the address of an rvalue of type
      'typeof(struct iov_iter)' (aka 'struct iov_iter')
  ...&({ typeof(struct iov_iter) _val; memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (u64)&msg->msg_iter); _val; });
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
Traceback (most recent call last):
  File "/root/bin/sockdump.py", line 306, in <module>
    main(args)
  File "/root/bin/sockdump.py", line 254, in main
    b = BPF(text=text)
  File "/usr/lib/python3/dist-packages/bcc/__init__.py", line 301, in __init__
    raise Exception("Failed to compile BPF module %s" % src_file)
Exception: Failed to compile BPF module 

Ubuntu 18.04.4 LTS
Python 3.6.9
libbpfcc 0.5.0-5ubuntu1
python3-bpfcc 0.5.0-5ubuntu1
libcc1-0:amd64 8.4.0-1ubuntu1~18.04

/virtual/main.c:67:16: error: no member named 'type' in 'struct iov_iter'

I tried this with bcc from debian testing as well as removing it and then compiling and installing bcc from the most current repo state:

# ./sockdump.py --format string /var/run/docker.sock
/virtual/main.c:67:16: error: no member named 'type' in 'struct iov_iter'
    if ((iter->type & WRITE) == 0 || iter->iov_offset != 0) {
         ~~~~  ^
1 error generated.
Traceback (most recent call last):
  File "/prg/tmp/sockdump/./sockdump.py", line 335, in <module>
    main(args)
  File "/prg/tmp/sockdump/./sockdump.py", line 281, in main
    b = BPF(text=text)
  File "/usr/lib/python3/dist-packages/bcc/__init__.py", line 479, in __init__
    raise Exception("Failed to compile BPF module %s" % (src_file or "<text>"))
Exception: Failed to compile BPF module <text>

kernel is 5.14.0

No data reported at all by sockdump.py

I start like in the README:

sander@witte2004:~/git/sockdump$ sudo ./sockdump.py --format string /var/run/docker.sock
waiting for data

and then I do a lot of docker interaction (docker ps, docker run ..., access the webinterface in that container), but ... nothing appears on the output sockdump.py

Help appreciated!

"--format string" has no leading header delimiters - some output can be difficult to read - and non-printing characters

This may - or may not - be a "no brainer" issue for sockdump.py users. Nonetheless -

The output from sockdump.py --format string ... has no leading delimiters to set apart the descriptive header from the actual data into and out of the socket. A trailing delimiter is already included with the python3 print() command by default.

The output format is not a problem with, for instance, an http interaction, which is profuse with "CR/LF" line endings. However, when viewing output from interaction with an smtp "milter" socket, many interactions involve short commands with no such "CR/LF" line endings. The result is that a single line displays data from an interaction in front of the header for the following interaction. This can be very confusing, especially the first time it is seen.

Of course, the user can easily modify the sockdump.py script to add leading delimiters to the header output, which then act as effective delimiters to those, in this example, short milter commands which have no "LF" terminators. Still, the question arises, would we like sockdump.py to provide, perhaps, an alternative format option which adds a line separator before the header?

To avoid interfering with the way sockdump.py currently functions, there might be an additional - rather trivial - option, perhaps "--format delimited", which is the same as "--format string" except with a simple "LF" preceding the header, so that the header is always set apart, on a line by itself, regardless of the form of the data being displayed.

Or, maybe sockdump.py should just include a leading "LF" in the header, by default? That's the simple ask.

And then, the user has to keep in mind that this "LF" at the end of the data is also not itself part of that data, but was a "LF" added by sockdump.py. But, I expect that that is easier than trying to read "run together" output on a single line.

In my case, I've added an additional leading "LF LF TAB" to the header, just to leave an empty line between data exchanges, and to set off the header itself from those short milter command lines. But that's just my preference.

And, something related, the display of non-printing characters in "--format string" output is an issue. This is something that may not be immediately apparent to the user, especially the new user, when data containing non-printing characters is displayed on a terminal, where those characters will be "invisible". Interestingly, when using the "--output" option, and viewing the output file with, say, less, then otherwise non-printing characters will be apparent, especially shown with reversed colors. But on a terminal - not so much.

Another question then, would we like a sockdump.py option which automatically displays non-printing characters to a terminal?

Of course, the user can easily use sockdump.py --format string ... | cat -e, but they would have to know to do this. Should this functionality be included, either automatically or as an option, in sockdump.py itself?

Alternatively, there might be a "--format raw" option, in which the invisible "LF" is removed from the header itself - print('...', end='') - some unusual, easily parsed, visible leading and trailing delimiters included with the sockdump.py header, such as "#$#", and the actual data is set between the headers, unmodified, as is.

Again, instead, the user could just customize the sockdump.py header themselves. But the current header format, as displayed on a terminal, with only the trailing "LF" and no leading "LF", can be confusing with some output.

Some things for consideration...

Failed to compile BPF module <text>

Traceback (most recent call last):
File "./sockdump.py", line 412, in
main(args)
File "./sockdump.py", line 353, in main
b = BPF(text=text)
File "/usr/lib/python3/dist-packages/bcc/init.py", line 347, in init
raise Exception("Failed to compile BPF module %s" % (src_file or ""))
Exception: Failed to compile BPF module

[Question] How to configure protocol?

Hello everybody 😉

I want to use sockdump for analyzing traffic between a client and server communicating using the Some/IP protocol. My command is:

sudo ./sockdump.py "/tmp/vsomeip*" --format pcap --output someip

Then I am importing the file someip via wireshark. But it is not showing the expected traffic.

Do I need to write an extra lua script like this one: wireshark/dummy.lua ?

Help is appreciated ☺️

No LICENSE

First of all, thanks for the helpful tool.

Unfortunately as the project specifies no license, it's dangerous from legal standpoint to even use it, and definitely not safe to contribute to it.

Can you please consider specifying any open-source license? https://choosealicense.com/ is a good starting point.

Or otherwise explicitly mention that the code is copyrighted and should only be viewed.

Compilation error: cannot take the address of an rvalue of type 'typeof(struct iov_iter)'

OS: Linux
Distribution: Ubuntu 18.04
Kernel version: 4.15.0-48-generic

1. Executed sudo apt install bpfcc-tools python3-bpfcc.
2. Cloned this repository.
3. Executed

/usr/bin/python3 sockdump.py /proc/28937/cwd/socket --format pcap --output dump

which output:

/virtual/main.c:54:12: error: cannot take the address of an rvalue of type 'typeof(struct iov_iter)'
      (aka 'struct iov_iter')
  ...&({ typeof(struct iov_iter) _val; memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (u64)&msg->msg_iter); _val; });
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
Traceback (most recent call last):
  File "sockdump.py", line 303, in <module>
    main(args)
  File "sockdump.py", line 252, in main
    b = BPF(text=text)
  File "/usr/lib/python3/dist-packages/bcc/__init__.py", line 301, in __init__
    raise Exception("Failed to compile BPF module %s" % src_file)
Exception: Failed to compile BPF module

Linux 5.7: "use of undeclared identifier 'KBUILD_MODNAME'"

On Ubuntu 20.04 with Linux 5.7.0, I get the errors below.

Might be related to

• bpftrace/bpftrace#715
• falcosecurity/falco#1334
• foniod/redbpf#71

It might also be a PEBKAC, or wrong setup on my side. I'll try to boot an older kernel from GRUB.

sander@witte2004:~/git/sockdump$ ll /var/run/docker.sock
srw-rw---- 1 root docker 0 dec 22 17:42 /var/run/docker.sock=
sander@witte2004:~/git/sockdump$ sudo ./sockdump.py --format string /var/run/docker.sock
In file included from /virtual/main.c:5:
In file included from include/net/af_unix.h:9:
In file included from include/net/sock.h:59:
In file included from include/linux/filter.h:25:
In file included from include/net/sch_generic.h:21:
include/net/flow_offload.h:304:4: error: use of undeclared identifier 'KBUILD_MODNAME'
                        NL_SET_ERR_MSG_MOD(extack, "Mixing HW stats types for actions is not supported");
                        ^
include/linux/netlink.h:96:27: note: expanded from macro 'NL_SET_ERR_MSG_MOD'
        NL_SET_ERR_MSG((extack), KBUILD_MODNAME ": " msg)
                                 ^
In file included from /virtual/main.c:5:
In file included from include/net/af_unix.h:9:
In file included from include/net/sock.h:59:
In file included from include/linux/filter.h:25:
In file included from include/net/sch_generic.h:21:
include/net/flow_offload.h:304:4: error: expected ';' at end of declaration
include/linux/netlink.h:96:42: note: expanded from macro 'NL_SET_ERR_MSG_MOD'
        NL_SET_ERR_MSG((extack), KBUILD_MODNAME ": " msg)
                                                ^
In file included from /virtual/main.c:5:
In file included from include/net/af_unix.h:9:
In file included from include/net/sock.h:59:
In file included from include/linux/filter.h:25:
In file included from include/net/sch_generic.h:21:
include/net/flow_offload.h:338:3: error: use of undeclared identifier 'KBUILD_MODNAME'
                NL_SET_ERR_MSG_MOD(extack, "Driver supports only default HW stats type \"any\"");
                ^
include/linux/netlink.h:96:27: note: expanded from macro 'NL_SET_ERR_MSG_MOD'
        NL_SET_ERR_MSG((extack), KBUILD_MODNAME ": " msg)
                                 ^
In file included from /virtual/main.c:5:
In file included from include/net/af_unix.h:9:
In file included from include/net/sock.h:59:
In file included from include/linux/filter.h:25:
In file included from include/net/sch_generic.h:21:
include/net/flow_offload.h:338:3: error: expected ';' at end of declaration
include/linux/netlink.h:96:42: note: expanded from macro 'NL_SET_ERR_MSG_MOD'
        NL_SET_ERR_MSG((extack), KBUILD_MODNAME ": " msg)
                                                ^
In file included from /virtual/main.c:5:
In file included from include/net/af_unix.h:9:
In file included from include/net/sock.h:59:
In file included from include/linux/filter.h:25:
In file included from include/net/sch_generic.h:21:
include/net/flow_offload.h:342:3: error: use of undeclared identifier 'KBUILD_MODNAME'
                NL_SET_ERR_MSG_MOD(extack, "Driver does not support selected HW stats type");
                ^
include/linux/netlink.h:96:27: note: expanded from macro 'NL_SET_ERR_MSG_MOD'
        NL_SET_ERR_MSG((extack), KBUILD_MODNAME ": " msg)
                                 ^
In file included from /virtual/main.c:5:
In file included from include/net/af_unix.h:9:
In file included from include/net/sock.h:59:
In file included from include/linux/filter.h:25:
In file included from include/net/sch_generic.h:21:
include/net/flow_offload.h:342:3: error: expected ';' at end of declaration
include/linux/netlink.h:96:42: note: expanded from macro 'NL_SET_ERR_MSG_MOD'
        NL_SET_ERR_MSG((extack), KBUILD_MODNAME ": " msg)
                                                ^
6 errors generated.
Traceback (most recent call last):
  File "./sockdump.py", line 306, in <module>
    main(args)
  File "./sockdump.py", line 254, in main
    b = BPF(text=text)
  File "/usr/lib/python3/dist-packages/bcc/__init__.py", line 347, in __init__
    raise Exception("Failed to compile BPF module %s" % (src_file or "<text>"))
Exception: Failed to compile BPF module <text>
sander@witte2004:~/git/sockdump$ 

no member named 'iov' in 'struct iov_iter'

In file included from arch/x86/include/asm/linkage.h:6:
arch/x86/include/asm/ibt.h:77:8: warning: 'nocf_check' attribute ignored; use -fcf-protection to enable the attribute [-Wignored-attributes]
extern __noendbr u64 ibt_save(bool disable);
       ^
arch/x86/include/asm/ibt.h:32:34: note: expanded from macro '__noendbr'
#define __noendbr       __attribute__((nocf_check))
                                       ^
arch/x86/include/asm/ibt.h:78:8: warning: 'nocf_check' attribute ignored; use -fcf-protection to enable the attribute [-Wignored-attributes]
extern __noendbr void ibt_restore(u64 save);
       ^
arch/x86/include/asm/ibt.h:32:34: note: expanded from macro '__noendbr'
#define __noendbr       __attribute__((nocf_check))
                                       ^
/virtual/main.c:122:17: error: no member named 'iov' in 'struct iov_iter'
    iov = iter->iov;
          ~~~~  ^
2 warnings and 1 error generated.

uname --kernel-release: 6.5.7-zen1-1-zen

That's not a missing dependency, isn't it?