[Doc] Getting Started > File Access Monitoring, is the doc correct ? about tetragon

[mtardy]

Hello, thanks for taking the time to write this issue. From what I can read I can see you are running Docker Desktop on macOS arm64 and it might be that the Linux kernel provided by Docker is missing the security_path_truncate function for some reason.

Could you try running it in a "proper" distro VM, like Ubuntu using lima? If Docker Desktop kernel was the problem, we can reach out to the Docker devs to tune the config, we already did in the past and that can be useful for them.

[zdk]

@mtardy Thanks for pointing out. It could be the Docker Desktop kernel.
I will try to test on an actual Linux machine.

But, for now, I have just quickly tested with lima Ubuntu and got the following details:

~/D/t/test-tetragon ❯❯❯ lima nerdctl run --name tetragon-container --rm --pull always \
  --pid=host --cgroupns=host --privileged               \
  -v ${PWD}/file_monitoring.yaml:/etc/tetragon/tetragon.tp.d/file_monitoring.yaml \
  -v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf      \
  quay.io/cilium/tetragon-ci:latest
quay.io/cilium/tetragon-ci:latest:                                                resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:fbe23d3fb0aec315a0c1e5bff55adb0bf19fdc99b9b7341d1c15b84567d2e23a:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:f0c6b66e38674a5067397d6550607128d59edd7cca67d1ac588e7165ab382d0c: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:6dec64b4fe0ac92e8e81b604bd11bcafc23dc9394b761536b26b1e6b1d619fb9:   done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 2.3 s                                                                    total:   0.0 B (0.0 B/s)
time="2024-07-02T15:21:59Z" level=info msg="Starting tetragon" version=v1.2.0-pre.0-241-ge44f7dfc0
time="2024-07-02T15:21:59Z" level=info msg="config settings" config="map[bpf-lib:/var/lib/tetragon/ btf: cgroup-rate: config-dir: cpuprofile: data-cache-size:1024 debug:false disable-kprobe-multi:false enable-export-aggregation:false enable-k8s-api:false enable-msg-handling-latency:false enable-pid-set-filter:false enable-pod-info:false enable-policy-filter:false enable-policy-filter-debug:false enable-process-ancestors:true enable-process-cred:false enable-process-ns:false enable-tracing-policy-crd:true event-queue-size:10000 export-aggregation-buffer-size:10000 export-aggregation-window-size:15s export-allowlist: export-denylist: export-file-compress:false export-file-max-backups:5 export-file-max-size-mb:10 export-file-perm:600 export-file-rotation-interval:0s export-filename: export-rate-limit:-1 expose-kernel-addresses:false expose-stack-addresses:false field-filters: force-large-progs:false force-small-progs:false generate-docs:false gops-address: health-server-address::6789 health-server-interval:10 k8s-kubeconfig-path: kernel: kmods:[] log-format:text log-level:info memprofile: metrics-label-filter:namespace,workload,pod,binary metrics-server: netns-dir:/var/run/docker/netns/ pprof-addr: process-cache-size:65536 procfs:/proc/ rb-queue-size:65535 rb-size:0 rb-size-total:0 redaction-filters: release-pinned-bpf:true server-address:localhost:54321 tracing-policy: tracing-policy-dir:/etc/tetragon/tetragon.tp.d username-metadata:disabled verbose:0]"
time="2024-07-02T15:21:59Z" level=info msg="Tetragon current security context" AppArmor=unconfined Lockdown= SELinux=unconfined Smack=
time="2024-07-02T15:21:59Z" level=info msg="Tetragon pid file creation succeeded" pid=6856 pidfile=/var/run/tetragon/tetragon.pid
time="2024-07-02T15:21:59Z" level=error msg="detect modify return syscall" error="failed to load: load program: operation not permitted (MEMLOCK may be too low, consider rlimit.RemoveMemlock)"
time="2024-07-02T15:21:59Z" level=info msg="BPF detected features: override_return: false, buildid: false, kprobe_multi: false, uprobe_multi false, fmodret: false, fmodret_syscall: false, signal: false, large: false"
time="2024-07-02T15:21:59Z" level=info msg="Kernel does not support time namespaces" error="stat /proc/1/ns/time: permission denied"
time="2024-07-02T15:21:59Z" level=fatal msg="Failed to initialize host namespaces" error="namespace '/proc/1/ns/uts' readlink /proc/1/ns/uts: permission denied" procfs=/proc/
~/D/t/test-tetragon ❯❯❯ limactl shell default                                                                                                                                ✘ 1
zdk@lima-default:/Users/zdk/Developer/tmp/test-tetragon$ sudo bpftrace --info
System
  OS: Linux 5.15.0-113-generic #123-Ubuntu SMP Mon Jun 10 08:16:46 UTC 2024
  Arch: aarch64

Build
  version: v0.14.0
  LLVM: 11.1.0
  ORC: v2
  foreach_sym: yes
  unsafe uprobe: no
  bfd: no
  bpf_attach_kfunc: yes
  bcc_usdt_addsem: yes
  bcc bpf_attach_uprobe refcount: yes
  bcc library path resolution: yes
  libbpf: yes
  libbpf btf dump: yes
  libbpf btf dump type decl: yes
  libdw (DWARF support): no

Kernel helpers
  probe_read: yes
  probe_read_str: yes
  probe_read_user: yes
  probe_read_user_str: yes
  probe_read_kernel: yes
  probe_read_kernel_str: yes
  get_current_cgroup_id: yes
  send_signal: yes
  override_return: yes
  get_boot_ns: yes
  dpath: yes

Kernel features
  Instruction limit: 1000000
  Loop support: yes
  btf (depends on Build:libbpf): yes
  map batch (depends on Build:libbpf): yes
  uprobe refcount (depends on Build:bcc bpf_attach_uprobe refcount): yes

Map types
  hash: yes
  percpu hash: yes
  array: yes
  percpu array: yes
  stack_trace: yes
  perf_event_array: yes

Probe types
  kprobe: yes
  tracepoint: yes
  perf_event: yes
  kfunc: yes
  iter:task: yes
  iter:task_file: yes

---

Notes:

The lima Ubuntu vm is configured to start with the following config:

~/D/t/test-lima-ebpf ❯❯❯ cat ubuntu-vm.yml
images:
  # Try to use release-yyyyMMdd image if available. Note that release-yyyyMMdd will be removed after several months.
  - location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.img"
    arch: "x86_64"
  - location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-arm64.img"
    arch: "aarch64"

mounts:
  - location: "~"
    writable: true
  - location: "/tmp/lima"
    writable: true
provision:
  - mode: system
    script: |
      apt-get update
      apt-get install -y apt-transport-https ca-certificates curl clang llvm jq
      apt-get install -y libelf-dev libpcap-dev libbfd-dev binutils-dev build-essential make
      apt-get install -y linux-tools-common linux-tools-5.15.0-41-generic bpfcc-tools
      apt-get install -y python3-pip
      apt-get install --yes bsdutils
      apt-get install --yes build-essential
      apt-get install --yes pkgconf
      apt-get install --yes llvm-12 clang-12
      apt-get install --yes clang-format-12
      apt-get install --yes zlib1g-dev libelf-dev
      apt-get install --yes protobuf-compiler

      sudo apt-get install bpfcc-tools linux-headers-$(uname -r)
      sudo snap install --devmode bpftrace

      # it downloads binaries with version appended
      # like llvm-strip-12, clang-12 etc
      # bpf stuff uses plain names like llvm-strip, clang and fails
      # to make them use this creating soft links with plain names
      for tool in "clang" "llc" "llvm-strip"
      do
        path=$(which $tool-12)
        sudo ln -s $path ${path%-*}
      done

      # uname -r returns kernel version
      # need linux-tools for kernel specific
      apt-get install --yes linux-tools-$(uname -r)

      # keep gp off, self signed cert issue else it'll fail to download
      # or add --no-check-certificate
      wget --quiet https://golang.org/dl/go1.20.1.linux-arm64.tar.gz
      tar -C /usr/local -xzf go1.20.1.linux-arm64.tar.gz
      echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.profile

which works ok with this hello-world program:

zdk@lima-default:/Users/zdk/Developer/tmp/test-lima-ebpf$ cat hello-world.py
#!/usr/bin/python3
from bcc import BPF
program = """
int hello(void *ctx) {
    bpf_trace_printk("Hello World!\\n");
return 0; }
"""
b = BPF(text=program)
syscall = b.get_syscall_fnname("execve")
b.attach_kprobe(event=syscall, fn_name="hello")
b.trace_print()

Output:

zdk@lima-default:/Users/zdk/Developer/tmp/test-lima-ebpf$ sudo python3 hello-world.py
b'           <...>-7291    [000] d...1   963.817171: bpf_trace_printk: Hello World!'
b''
b'            bash-7291    [000] d...1   963.819399: bpf_trace_printk: Hello World!'
b''
b'           <...>-7292    [001] d...1   963.820278: bpf_trace_printk: Hello World!'
b''
b'           <...>-7293    [002] d...1   963.821031: bpf_trace_printk: Hello World!'
b''
b'           <...>-7294    [003] d...1   963.821753: bpf_trace_printk: Hello World!'
b''
b'           <...>-7297    [003] d...1   963.827897: bpf_trace_printk: Hello World!'
b''
b'           <...>-7298    [002] d...1   963.828236: bpf_trace_printk: Hello World!'
b''
b'           <...>-7300    [002] d...1   963.828866: bpf_trace_printk: Hello World!'
b''
b'           <...>-7301    [002] d...1   963.829558: bpf_trace_printk: Hello World!'
b''
b'           <...>-7303    [002] d...1   968.753268: bpf_trace_printk: Hello World!'
b''
^CTraceback (most recent call last):
  File "/Users/zdk/Developer/tmp/test-lima-ebpf/hello-world.py", line 11, in <module>
    b.trace_print()
  File "/usr/lib/python3/dist-packages/bcc/__init__.py", line 1332, in trace_print
    line = self.trace_readline(nonblocking=False)
  File "/usr/lib/python3/dist-packages/bcc/__init__.py", line 1312, in trace_readline
    line = trace.readline(1024).rstrip()
KeyboardInterrupt

zdk@lima-default:/Users/zdk/Developer/tmp/test-lima-ebpf$ uname -r
5.15.0-113-generic
zdk@lima-default:/Users/zdk/Developer/tmp/test-lima-ebpf$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
zdk@lima-default:/Users/zdk/Developer/tmp/test-lima-ebpf$ cat /proc/meminfo
MemTotal:        4004432 kB
MemFree:          226380 kB
MemAvailable:    3548912 kB
Buffers:           69692 kB
Cached:          3275476 kB
SwapCached:            0 kB
Active:           861368 kB
Inactive:        2585592 kB
Active(anon):       1300 kB
Inactive(anon):   113192 kB
Active(file):     860068 kB
Inactive(file):  2472400 kB
Unevictable:       30188 kB
Mlocked:           26188 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:        132004 kB
Mapped:           171648 kB
Shmem:              5316 kB
KReclaimable:     173896 kB
Slab:             234624 kB
SReclaimable:     173896 kB
SUnreclaim:        60728 kB
KernelStack:        3360 kB
PageTables:         3248 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     2002216 kB
Committed_AS:     736204 kB
VmallocTotal:   133143592960 kB
VmallocUsed:       17956 kB
VmallocChunk:          0 kB
Percpu:             2032 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
CmaTotal:          32768 kB
CmaFree:           13740 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB

[zdk]

@mtardy
Just did the test on Ubuntu 22.04.4 LTS in AWS.
It seems working fine.

I'm closing the issue due to it's something more on container challenges on MacOS.