libbpf / libbpf-bootstrap

Scaffolding for BPF application development with libbpf and BPF CO-RE

libbpf-bootstrap: demo BPF applications

minimal

minimal is just that – a minimal practical BPF application example. It doesn't use or require BPF CO-RE, so should run on quite old kernels. It installs a tracepoint handler which is triggered once every second. It uses bpf_printk() BPF helper to communicate with the world. To see it's output, read /sys/kernel/debug/tracing/trace_pipe file as a root:

$ cd examples/c
$ make minimal
$ sudo ./minimal
$ sudo cat /sys/kernel/debug/tracing/trace_pipe
           <...>-3840345 [010] d... 3220701.101143: bpf_trace_printk: BPF triggered from PID 3840345.
           <...>-3840345 [010] d... 3220702.101265: bpf_trace_printk: BPF triggered from PID 3840345.

minimal is great as a bare-bones experimental playground to quickly try out new ideas or BPF features.

minimal_ns

minimal_ns is as same as minimal but for namespaced environments. minimal would not work in environments that have namespace, like containers, or WSL2, because the perceived pid of the process in the namespace is not the actual pid of the process. For executing minimal in namespaced environments you need to use minimal_ns instead.

$ cd examples/c
$ make minimal_ns
$ sudo ./minimal_ns
$ sudo cat /sys/kernel/debug/tracing/trace_pipe
           <...>-3840345 [022] d...1  8804.331204: bpf_trace_printk: BPF triggered from PID 9087.
           <...>-3840345 [022] d...1  8804.331215: bpf_trace_printk: BPF triggered from PID 9087.

minimal_Legacy

This version of minimal is modified to allow running on even older kernels that do not allow global variables. bpf_printk uses global variables unless BPF_NO_GLOBAL_DATA is defined before including bpf_helpers.h. Additionally, the global variable my_pid has been replaced with an array of one element to hold the process pid.

$ cd examples/c
$ make minimal_legacy
$ sudo ./minimal_legacy
$ sudo cat /sys/kernel/debug/tracing/trace_pipe
  minimal_legacy-52030 [001] .... 491227.784078: 0x00000001: BPF triggered from PID 52030.
  minimal_legacy-52030 [001] .... 491228.840571: 0x00000001: BPF triggered from PID 52030.
  minimal_legacy-52030 [001] .... 491229.841643: 0x00000001: BPF triggered from PID 52030.
  minimal_legacy-52030 [001] .... 491230.842432: 0x00000001: BPF triggered from PID 52030.

bootstrap

bootstrap is an example of a simple (but realistic) BPF application. It tracks process starts (exec() family of syscalls, to be precise) and exits and emits data about filename, PID and parent PID, as well as exit status and duration of the process life. With -d <min-duration-ms> you can specify minimum duration of the process to log. In such mode process start (technically, exec()) events are not output (see example output below).

bootstrap was created in the similar spirit as libbpf-tools from BCC package, but is designed to be more stand-alone and with simpler Makefile to simplify adoption to user's particular needs. It demonstrates the use of typical BPF features:

• cooperating BPF programs (tracepoint handlers for process exec and exit events, in this particular case);
• BPF map for maintaining the state;
• BPF ring buffer for sending data to user-space;
• global variables for application behavior parameterization.
• it utilizes BPF CO-RE and vmlinux.h to read extra process information from kernel's struct task_struct.

bootstrap is intended to be the starting point for your own BPF application, with things like BPF CO-RE and vmlinux.h, consuming BPF ring buffer data, command line arguments parsing, graceful Ctrl-C handling, etc. all taken care of for you, which are crucial but mundane tasks that are no fun, but necessary to be able to do anything useful. Just copy/paste and do simple renaming to get yourself started.

Here's an example output in minimum process duration mode:

$ sudo ./bootstrap -d 50
TIME     EVENT COMM             PID     PPID    FILENAME/EXIT CODE
19:18:32 EXIT  timeout          3817109 402466  [0] (126ms)
19:18:32 EXIT  sudo             3817117 3817111 [0] (259ms)
19:18:32 EXIT  timeout          3817110 402466  [0] (264ms)
19:18:33 EXIT  python3.7        3817083 1       [0] (1026ms)
19:18:38 EXIT  python3          3817429 3817424 [1] (60ms)
19:18:38 EXIT  sh               3817424 3817420 [0] (79ms)
19:18:38 EXIT  timeout          3817420 402466  [0] (80ms)
19:18:43 EXIT  timeout          3817610 402466  [0] (70ms)
19:18:43 EXIT  grep             3817619 3817617 [1] (271ms)
19:18:43 EXIT  timeout          3817609 402466  [0] (321ms)
19:18:44 EXIT  iostat           3817585 3817531 [0] (3006ms)
19:18:44 EXIT  tee              3817587 3817531 [0] (3005ms)
...

uprobe

uprobe is an example of dealing with user-space entry and exit (return) probes, uprobe and uretprobe in libbpf lingo. It attached uprobe and uretprobe BPF programs to its own functions (uprobed_add() and uprobed_sub()) and logs input arguments and return result, respectively, using bpf_printk() macro. The user-space function is triggered once every second:

$ sudo ./uprobe
libbpf: loading object 'uprobe_bpf' from buffer
...
Successfully started!
...........

You can see uprobe demo output in /sys/kernel/debug/tracing/trace_pipe:

$ sudo cat /sys/kernel/debug/tracing/trace_pipe
          uprobe-1809291 [007] .... 4017233.106596: 0: uprobed_add ENTRY: a = 0, b = 1
          uprobe-1809291 [007] .... 4017233.106605: 0: uprobed_add EXIT: return = 1
          uprobe-1809291 [007] .... 4017233.106606: 0: uprobed_sub ENTRY: a = 0, b = 0
          uprobe-1809291 [007] .... 4017233.106607: 0: uprobed_sub EXIT: return = 0
          uprobe-1809291 [007] .... 4017234.106694: 0: uprobed_add ENTRY: a = 1, b = 2
          uprobe-1809291 [007] .... 4017234.106697: 0: uprobed_add EXIT: return = 3
          uprobe-1809291 [007] .... 4017234.106700: 0: uprobed_sub ENTRY: a = 1, b = 1
          uprobe-1809291 [007] .... 4017234.106701: 0: uprobed_sub EXIT: return = 0

usdt

usdt is an example of dealing with USDT probe. It attaches USDT BPF programs to the libc:setjmp probe, which is triggered by calling setjmp in user-space program once per second and logs USDT arguments using bpf_printk() macro:

$ sudo ./usdt
libbpf: loading object 'usdt_bpf' from buffer
...
Successfully started!
...........

You can see usdt demo output in /sys/kernel/debug/tracing/trace_pipe:

$ sudo cat /sys/kernel/debug/tracing/trace_pipe
            usdt-1919077 [005] d..21 537310.886092: bpf_trace_printk: USDT auto attach to libc:setjmp: arg1 = 55d03d6a42a0, arg2 = 0, arg3 = 55d03d65e54e
            usdt-1919077 [005] d..21 537310.886105: bpf_trace_printk: USDT manual attach to libc:setjmp: arg1 = 55d03d6a42a0, arg2 = 0, arg3 = 55d03d65e54e
            usdt-1919077 [005] d..21 537311.886214: bpf_trace_printk: USDT auto attach to libc:setjmp: arg1 = 55d03d6a42a0, arg2 = 0, arg3 = 55d03d65e54e
            usdt-1919077 [005] d..21 537311.886227: bpf_trace_printk: USDT manual attach to libc:setjmp: arg1 = 55d03d6a42a0, arg2 = 0, arg3 = 55d03d65e54e

fentry

fentry is an example that uses fentry and fexit BPF programs for tracing. It attaches fentry and fexit traces to do_unlinkat() which is called when a file is deleted and logs the return value, PID, and filename to the trace pipe.

Important differences, compared to kprobes, are improved performance and usability. In this example, better usability is shown with the ability to directly dereference pointer arguments, like in normal C, instead of using various read helpers. The big distinction between fexit and kretprobe programs is that fexit one has access to both input arguments and returned result, while kretprobe can only access the result.

fentry and fexit programs are available starting from 5.5 kernels.

$ sudo ./fentry
libbpf: loading object 'fentry_bpf' from buffer
...
Successfully started!
..........

The fentry output in /sys/kernel/debug/tracing/trace_pipe should look something like this:

$ sudo cat /sys/kernel/debug/tracing/trace_pipe
              rm-9290    [004] d..2  4637.798698: bpf_trace_printk: fentry: pid = 9290, filename = test_file
              rm-9290    [004] d..2  4637.798843: bpf_trace_printk: fexit: pid = 9290, filename = test_file, ret = 0
              rm-9290    [004] d..2  4637.798698: bpf_trace_printk: fentry: pid = 9290, filename = test_file2
              rm-9290    [004] d..2  4637.798843: bpf_trace_printk: fexit: pid = 9290, filename = test_file2, ret = 0

kprobe

kprobe is an example of dealing with kernel-space entry and exit (return) probes, kprobe and kretprobe in libbpf lingo. It attaches kprobe and kretprobe BPF programs to the do_unlinkat() function and logs the PID, filename, and return result, respectively, using bpf_printk() macro.

$ sudo ./kprobe
libbpf: loading object 'kprobe_bpf' from buffer
...
Successfully started!
...........

The kprobe demo output in /sys/kernel/debug/tracing/trace_pipe should look something like this:

$ sudo cat /sys/kernel/debug/tracing/trace_pipe
              rm-9346    [005] d..3  4710.951696: bpf_trace_printk: KPROBE ENTRY pid = 9346, filename = test1
              rm-9346    [005] d..4  4710.951819: bpf_trace_printk: KPROBE EXIT: ret = 0
              rm-9346    [005] d..3  4710.951852: bpf_trace_printk: KPROBE ENTRY pid = 9346, filename = test2
              rm-9346    [005] d..4  4710.951895: bpf_trace_printk: KPROBE EXIT: ret = 0

xdp

xdp is an example written in Rust (using libbpf-rs). It attaches to the ingress path of networking device and logs the size of each packet, returning XDP_PASS to allow the packet to be passed up to the kernel’s networking stack.

$ sudo ./target/release/xdp 1
..........

The xdp output in /sys/kernel/debug/tracing/trace_pipe should look something like this:

$ sudo cat /sys/kernel/debug/tracing/trace_pipe
           <...>-823887  [000] d.s1 602386.079100: bpf_trace_printk: packet size: 75
           <...>-823887  [000] d.s1 602386.079141: bpf_trace_printk: packet size: 66
           <...>-2813507 [000] d.s1 602386.696702: bpf_trace_printk: packet size: 77
           <...>-2813507 [000] d.s1 602386.696735: bpf_trace_printk: packet size: 66

tc

tc (short for Traffic Control) is an example of handling ingress network traffics. It creates a qdisc on the lo interface and attaches the tc_ingress BPF program to it. It reports the metadata of the IP packets that coming into the lo interface.

$ sudo ./tc
...
Successfully started! Please run `sudo cat /sys/kernel/debug/tracing/trace_pipe` to see output of the BPF program.
......

The tc output in /sys/kernel/debug/tracing/trace_pipe should look something like this:

$ sudo cat /sys/kernel/debug/tracing/trace_pipe
            node-1254811 [007] ..s1 8737831.671074: 0: Got IP packet: tot_len: 79, ttl: 64
            sshd-1254728 [006] ..s1 8737831.674334: 0: Got IP packet: tot_len: 79, ttl: 64
            sshd-1254728 [006] ..s1 8737831.674349: 0: Got IP packet: tot_len: 72, ttl: 64
            node-1254811 [007] ..s1 8737831.674550: 0: Got IP packet: tot_len: 71, ttl: 64

profile

profile is an example written in Rust and C using the blazesym symbolization library. It attaches to perf events, sampling on every processor periodically. It shows addresses, symbols, file names, and line numbers of stacktraces (if available).

$ sudo ./target/release/profile
COMM: swapper/2 (pid=0) @ CPU 2
Kernel:
0xffffffffb59141f8: mwait_idle_with_hints.constprop.0 @ 0xffffffffb59141b0+0x48
0xffffffffb5f731ce: intel_idle @ 0xffffffffb5f731b0+0x1e
0xffffffffb5c7bf09: cpuidle_enter_state @ 0xffffffffb5c7be80+0x89
0xffffffffb5c7c309: cpuidle_enter @ 0xffffffffb5c7c2e0+0x29
0xffffffffb516f57c: do_idle @ 0xffffffffb516f370+0x20c
0xffffffffb516f829: cpu_startup_entry @ 0xffffffffb516f810+0x19
0xffffffffb5075bfa: start_secondary @ 0xffffffffb5075ae0+0x11a
0xffffffffb500015a: secondary_startup_64_no_verify @ 0xffffffffb5000075+0xe5
No Userspace Stack

C version and Rust version show the same content. Both of them use blazesym to symbolize stacktraces.

sockfilter

sockfilter is an example of monitoring packet and dealing with __sk_buff structure. It attaches socket BPF program to sock_queue_rcv_skb() function and retrieve information from BPF_MAP_TYPE_RINGBUF, then print protocol, src IP, src port, dst IP, dst port in standard output. Currently, most of the IPv4 protocols defined in uapi/linux/in.h are included, please check ipproto_mapping of examples/c/sockfilter.c for the supported protocols.

$ sudo ./sockfilter -i <interface>
interface:lo    protocol: UDP   127.0.0.1:51845(src) -> 127.0.0.1:53(dst)
interface:lo    protocol: UDP   127.0.0.1:41552(src) -> 127.0.0.1:53(dst)

task_iter

task_iter is an example of using BPF Iterators. This example iterates over all tasks on the host and gets their pid, process name, kernel stack, and their state. Note: you can use BlazeSym to symbolize the kernel stacktraces (like in profile) but that code is omitted for simplicity.

$ sudo ./task_iter
Task Info. Pid: 3647645. Process Name: TTLSFWorker59. Kernel Stack Len: 3. State: INTERRUPTIBLE
Task Info. Pid: 1600495. Process Name: tmux: client. Kernel Stack Len: 6. State: INTERRUPTIBLE
Task Info. Pid: 1600497. Process Name: tmux: server. Kernel Stack Len: 0. State: RUNNING
Task Info. Pid: 1600498. Process Name: bash. Kernel Stack Len: 5. State: INTERRUPTIBLE

lsm

lsm serves as an illustrative example of utilizing LSM BPF. In this example, the bpf() system call is effectively blocked. Once the lsm program is operational, its successful execution can be confirmed by using the bpftool prog list command.

$ sudo ./lsm
libbpf: loading object 'lsm_bpf' from buffer
...
Successfully started! Please run `sudo cat /sys/kernel/debug/tracing/trace_pipe` to see output of the BPF programs.
..........

The output from lsm in /sys/kernel/debug/tracing/trace_pipe is expected to resemble the following:

$ sudo cat /sys/kernel/debug/tracing/trace_pipe
         bpftool-70646   [002] ...11 279318.416393: bpf_trace_printk: LSM: block bpf() worked
         bpftool-70646   [002] ...11 279318.416532: bpf_trace_printk: LSM: block bpf() worked
         bpftool-70646   [002] ...11 279318.416533: bpf_trace_printk: LSM: block bpf() worked

When the bpf() system call gets blocked, the bpftool prog list command yields the following output:

$ sudo bpftool prog list
Error: can't get next program: Operation not permitted

Building

libbpf-bootstrap supports multiple build systems that do the same thing. This serves as a cross reference for folks coming from different backgrounds.

Install Dependencies

You will need clang (at least v11 or later), libelf and zlib to build the examples, package names may vary across distros.

On Ubuntu/Debian, you need:

$ apt install clang libelf1 libelf-dev zlib1g-dev

On CentOS/Fedora, you need:

$ dnf install clang elfutils-libelf elfutils-libelf-devel zlib-devel

Getting the source code

Download the git repository and check out submodules:

$ git clone --recurse-submodules https://github.com/libbpf/libbpf-bootstrap

C Examples

Makefile build:

$ git submodule update --init --recursive       # check out libbpf
$ cd examples/c
$ make
$ sudo ./bootstrap
TIME     EVENT COMM             PID     PPID    FILENAME/EXIT CODE
00:21:22 EXIT  python3.8        4032353 4032352 [0] (123ms)
00:21:22 EXEC  mkdir            4032379 4032337 /usr/bin/mkdir
00:21:22 EXIT  mkdir            4032379 4032337 [0] (1ms)
00:21:22 EXEC  basename         4032382 4032381 /usr/bin/basename
00:21:22 EXIT  basename         4032382 4032381 [0] (0ms)
00:21:22 EXEC  sh               4032381 4032380 /bin/sh
00:21:22 EXEC  dirname          4032384 4032381 /usr/bin/dirname
00:21:22 EXIT  dirname          4032384 4032381 [0] (1ms)
00:21:22 EXEC  readlink         4032387 4032386 /usr/bin/readlink
^C

CMake build:

$ git submodule update --init --recursive       # check out libbpf
$ mkdir build && cd build
$ cmake ../examples/c
$ make
$ sudo ./bootstrap
<...>

XMake build (Linux):

$ git submodule update --init --recursive       # check out libbpf
$ cd examples/c
$ xmake
$ xmake run bootstrap

XMake build (Android):

$ git submodule update --init --recursive       # check out libbpf
$ cd examples/c
$ xmake f -p android
$ xmake

Install Xmake

$ bash <(wget https://xmake.io/shget.text -O -)
$ source ~/.xmake/profile

Rust Examples

Install libbpf-cargo:

$ cargo install libbpf-cargo

Build using cargo:

$ cd examples/rust
$ cargo build --release
$ sudo ./target/release/xdp 1
<...>

Troubleshooting

Libbpf debug logs are quire helpful to pinpoint the exact source of problems, so it's usually a good idea to look at them before starting to debug or posting question online.

./minimal is always running with libbpf debug logs turned on.

For ./bootstrap, run it in verbose mode (-v) to see libbpf debug logs:

$ sudo ./bootstrap -v
libbpf: loading object 'bootstrap_bpf' from buffer
libbpf: elf: section(2) tp/sched/sched_process_exec, size 384, link 0, flags 6, type=1
libbpf: sec 'tp/sched/sched_process_exec': found program 'handle_exec' at insn offset 0 (0 bytes), code size 48 insns (384 bytes)
libbpf: elf: section(3) tp/sched/sched_process_exit, size 432, link 0, flags 6, type=1
libbpf: sec 'tp/sched/sched_process_exit': found program 'handle_exit' at insn offset 0 (0 bytes), code size 54 insns (432 bytes)
libbpf: elf: section(4) license, size 13, link 0, flags 3, type=1
libbpf: license of bootstrap_bpf is Dual BSD/GPL
...

libbpf-bootstrap's Issues

unable to run in aarch64

Hello,

I was trying to prepare libbpf-bootstrap on aarch64 using BTF and CORE. When I cloned the GIT, I realized that the bbpftool bin was built for x86. Hence I rebilt the “bpftool” from the kernel tree (v 5.9.0) on aarch64 base board. When I try to move the bpftool to my final board, it missed the shared library (libbfd-2.30-system.so & libopcodes-2.30-system.so), Which I copied from the base board to make the build through. Also I update the ARCH in the libbpf-bootstrap-master/src/Makefile to ARCH := $(shell uname -m | sed 's/x86_64/x86/aarch64') to consider aarch64

When I try to run, I get the following errors. I am not sure the steps I follow are right or am I missing something.

Any hint would be of great help.
Thankyou for your support.

Tools used:
Clang version: clang version 10.0.1 Target: aarch64-fsl-linux, Thread model: posix, InstalledDir: /usr/bin
Kernel version: 5.4.69
Pahole version used at kernel build: V1.9
.config of the target: attached
config.txt
cc –version : cc (GCC) 9.2.0

Error Message:

~/libbpf-bootstrap-master/src# ./minimal
libbpf: loading object 'minimal_bpf' from buffer
libbpf: elf: section(2) tp/syscalls/sys_enter_write, size 192, link 0, flags 6, type=1
libbpf: sec 'tp/syscalls/sys_enter_write': found program 'handle_tp' at insn offset 0 (0 bytes), code size 24 insns (192 bytes)
libbpf: elf: section(3) license, size 13, link 0, flags 3, type=1
libbpf: license of minimal_bpf is Dual BSD/GPL
libbpf: elf: section(4) .bss, size 4, link 0, flags 3, type=8
libbpf: elf: section(5) .rodata.str1.1, size 28, link 0, flags 32, type=1
libbpf: elf: skipping unrecognized data section(5) .rodata.str1.1
libbpf: elf: section(6) .BTF, size 490, link 0, flags 0, type=1
libbpf: elf: section(7) .BTF.ext, size 160, link 0, flags 0, type=1
libbpf: elf: section(8) .symtab, size 168, link 13, flags 0, type=2
libbpf: elf: section(9) .reltp/syscalls/sys_enter_write, size 16, link 8, flags 0, type=9
libbpf: looking for externs among 7 symbols...
libbpf: collected 0 externs total
libbpf: map 'minimal_.bss' (global data): at sec_idx 4, offset 0, flags 400.
libbpf: map 0 is "minimal_.bss"
libbpf: sec '.reltp/syscalls/sys_enter_write': collecting relocation for section(2) 'tp/syscalls/sys_enter_write'
libbpf: sec '.reltp/syscalls/sys_enter_write': relo #0: insn #2 against 'my_pid'
libbpf: prog 'handle_tp': found data map 0 (minimal_.bss, sec 4, off 0) for insn 2
libbpf: map 'minimal_.bss': created successfully, fd=4
Segmentation fault
~/libbpf-bootstrap-master/src#
~/libbpf-bootstrap-master/src# ./bootstrap -v
libbpf: loading object 'bootstrap_bpf' from buffer
libbpf: elf: section(2) tp/sched/sched_process_exec, size 504, link 0, flags 6, type=1
libbpf: sec 'tp/sched/sched_process_exec': found program 'handle_exec' at insn offset 0 (0 bytes), code size 63 insns (504 bytes)
libbpf: elf: section(3) tp/sched/sched_process_exit, size 680, link 0, flags 6, type=1
libbpf: sec 'tp/sched/sched_process_exit': found program 'handle_exit' at insn offset 0 (0 bytes), code size 85 insns (680 bytes)
libbpf: elf: section(4) license, size 13, link 0, flags 3, type=1
libbpf: license of bootstrap_bpf is Dual BSD/GPL
libbpf: elf: section(5) .rodata, size 8, link 0, flags 2, type=1
libbpf: elf: section(6) .maps, size 48, link 0, flags 3, type=1
libbpf: elf: section(7) .BTF, size 24513, link 0, flags 0, type=1
libbpf: elf: section(8) .BTF.ext, size 1356, link 0, flags 0, type=1
libbpf: elf: section(9) .symtab, size 360, link 15, flags 0, type=2
libbpf: elf: section(10) .reltp/sched/sched_process_exec, size 48, link 9, flags 0, type=9
libbpf: elf: section(11) .reltp/sched/sched_process_exit, size 80, link 9, flags 0, type=9
libbpf: looking for externs among 15 symbols...
libbpf: collected 0 externs total
libbpf: map 'exec_start': at sec_idx 6, offset 0.
libbpf: map 'exec_start': found type = 1.
libbpf: map 'exec_start': found max_entries = 8192.
libbpf: map 'exec_start': found key [9], sz = 4.
libbpf: map 'exec_start': found value [12], sz = 8.
libbpf: map 'rb': at sec_idx 6, offset 32.
libbpf: map 'rb': found type = 27.
libbpf: map 'rb': found max_entries = 262144.
libbpf: map 'bootstra.rodata' (global data): at sec_idx 5, offset 0, flags 480.
libbpf: map 2 is "bootstra.rodata"
libbpf: sec '.reltp/sched/sched_process_exec': collecting relocation for section(2) 'tp/sched/sched_process_exec'
libbpf: sec '.reltp/sched/sched_process_exec': relo #0: insn #10 against 'exec_start'
libbpf: prog 'handle_exec': found map 0 (exec_start, sec 6, off 0) for insn #10
libbpf: sec '.reltp/sched/sched_process_exec': relo #1: insn #14 against 'min_duration_ns'
libbpf: prog 'handle_exec': found data map 2 (bootstra.rodata, sec 5, off 0) for insn 14
libbpf: sec '.reltp/sched/sched_process_exec': relo #2: insn #19 against 'rb'
libbpf: prog 'handle_exec': found map 1 (rb, sec 6, off 32) for insn #19
libbpf: sec '.reltp/sched/sched_process_exit': collecting relocation for section(3) 'tp/sched/sched_process_exit'
libbpf: sec '.reltp/sched/sched_process_exit': relo #0: insn #9 against 'exec_start'
libbpf: prog 'handle_exit': found map 0 (exec_start, sec 6, off 0) for insn #9
libbpf: sec '.reltp/sched/sched_process_exit': relo #1: insn #20 against 'min_duration_ns'
libbpf: prog 'handle_exit': found data map 2 (bootstra.rodata, sec 5, off 0) for insn 20
libbpf: sec '.reltp/sched/sched_process_exit': relo #2: insn #26 against 'exec_start'
libbpf: prog 'handle_exit': found map 0 (exec_start, sec 6, off 0) for insn #26
libbpf: sec '.reltp/sched/sched_process_exit': relo #3: insn #29 against 'min_duration_ns'
libbpf: prog 'handle_exit': found data map 2 (bootstra.rodata, sec 5, off 0) for insn 29
libbpf: sec '.reltp/sched/sched_process_exit': relo #4: insn #35 against 'rb'
libbpf: prog 'handle_exit': found map 1 (rb, sec 6, off 32) for insn #35
libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': 0
libbpf: map 'exec_start': created successfully, fd=4
libbpf: map 'rb': failed to create: Invalid argument(-22)
libbpf: failed to load object 'bootstrap_bpf'
libbpf: failed to load BPF skeleton 'bootstrap_bpf': -22
Failed to load and verify BPF skeleton
~/libbpf-bootstrap-master/src#
~/libbpf-bootstrap-master/src#

While generation the “bpftool”, clang-bpf-co-re: [ OFF ] was there, I am not sure either it could cause the problem

BPFTOOL Log:

~/linux-5.9/tools$ make bpf
  DESCEND  bpf

Auto-detecting system features:
...                        libbfd: [ on  ]
...        disassembler-four-args: [ on  ]

  CC       bpf_jit_disasm.o
  LINK     bpf_jit_disasm
  CC       bpf_dbg.o
  LINK     bpf_dbg
  CC       bpf_asm.o
  BISON    bpf_exp.yacc.c
  CC       bpf_exp.yacc.o
  FLEX     bpf_exp.lex.c
  CC       bpf_exp.lex.o
  LINK     bpf_asm
  DESCEND  bpftool

Auto-detecting system features:
...                        libbfd: [ on  ]
...        disassembler-four-args: [ on  ]
...                          zlib: [ on  ]
...                        libcap: [ OFF ]
...               clang-bpf-co-re: [ OFF ]

  CC       map_perf_ring.o
  CC       xlated_dumper.o
  CC       iter.o
  CC       btf.o
  CC       tracelog.o
  CC       link.o
  CC       perf.o
  CC       prog.o
  CC       btf_dumper.o
  CC       net.o
  CC       struct_ops.o
  CC       netlink_dumper.o
  CC       common.o
  CC       cgroup.o
  CC       gen.o
  CC       main.o
  CC       json_writer.o
  CC       cfg.o
  CC       map.o
  CC       pids.o
  CC       feature.o
  CC       jit_disasm.o
  CC       disasm.o

Auto-detecting system features:
...                        libelf: [ on  ]
...                          zlib: [ on  ]
...                           bpf: [ on  ]

  GEN      bpf_helper_defs.h
  MKDIR    staticobjs/
  CC       staticobjs/libbpf.o
  CC       staticobjs/bpf.o
  CC       staticobjs/nlattr.o
  CC       staticobjs/btf.o
  CC       staticobjs/libbpf_errno.o
  CC       staticobjs/str_error.o
  CC       staticobjs/netlink.o
  CC       staticobjs/bpf_prog_linfo.o
  CC       staticobjs/libbpf_probes.o
  CC       staticobjs/xsk.o
  CC       staticobjs/hashmap.o
  CC       staticobjs/btf_dump.o
  CC       staticobjs/ringbuf.o
  LD       staticobjs/libbpf-in.o
  LINK     libbpf.a
  LINK     bpftool
  DESCEND  runqslower
  MKDIR    .output

libbpf: sec 'tp/syscalls/sys_enter_write': failed to find program symbol at offset 0

hi

I had some problems compiling minimal. I hope to get help

[root@localhost src]# make clean minimal
  CLEAN
  MKDIR    .output
  MKDIR    .output/libbpf
  LIB      libbpf.a
  MKDIR    staticobjs
  CC       bpf.o
  CC       btf.o
  CC       libbpf.o
  CC       libbpf_errno.o
  CC       netlink.o
  CC       nlattr.o
  CC       str_error.o
  CC       libbpf_probes.o
  CC       bpf_prog_linfo.o
  CC       xsk.o
  CC       btf_dump.o
  CC       hashmap.o
  CC       ringbuf.o
  AR       libbpf.a
  INSTALL  bpf.h libbpf.h btf.h xsk.h libbpf_util.h bpf_helpers.h bpf_helper_defs.h bpf_tracing.h bpf_endian.h bpf_core_read.h libbpf_common.h
  INSTALL  libbpf.pc
  INSTALL  libbpf.a
  BPF      .output/minimal.bpf.o
  GEN-SKEL .output/minimal.skel.h
libbpf: sec 'tp/syscalls/sys_enter_write': failed to find program symbol at offset 0
Error: failed to open BPF object file: BPF object format invalid
make: *** [.output/minimal.skel.h] 错误 255
make: *** 正在删除文件“.output/minimal.skel.h”

env

[root@localhost src]# uname -a
Linux localhost.localdomain 5.4.108-1.el7.elrepo.x86_64 #1 SMP Mon Mar 22 18:37:08 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost src]# rpm -qa | grep elf
elfutils-libs-0.176-5.el7.x86_64
devtoolset-7-elfutils-libs-0.170-5.el7.x86_64
elfutils-libelf-0.176-5.el7.x86_64
elfutils-libelf-devel-0.176-5.el7.x86_64
elfutils-default-yama-scope-0.172-2.el7.noarch
devtoolset-7-elfutils-libelf-0.170-5.el7.x86_64
devtoolset-7-elfutils-0.170-5.el7.x86_64
[root@localhost src]# rpm -qa | grep zlib
zlib-devel-1.2.7-19.el7_9.x86_64
zlib-1.2.7-19.el7_9.x86_64
[root@localhost src]#

[root@localhost src]# ../tools/bpftool version
../tools/bpftool v5.10.0-rc3
features: libbfd, skeletons

Thanks.

Kernel freeze when instrument `sys_enter_execve` and `mm_page_alloc_zone_locked` simultaneously

My system is:

$ uname -a
Linux localhost.localdomain 5.11.12-300.fc34.x86_64 #1 SMP Wed Apr 7 16:31:13 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

The minimal example:

#include "vmlinux.h"
#include <bpf/bpf_helpers.h>

char LICENSE[] SEC("license") = "Dual BSD/GPL";

SEC("tracepoint/syscalls/sys_enter_execve")
int execve(void *ctx)
{
	u32 value = 0;
	bpf_probe_read(&value, sizeof(value), ctx);
	return 0;
}

SEC("tracepoint/kmem/mm_page_alloc_zone_locked")
int page_alloc(void *ctx)
{
	u32 value = 0;
	bpf_probe_read(&value, sizeof(value), ctx);
	return 0;
}

It is a *.bpf.c file. In userspace do open, load and attach as usual. And entire system is freeze.

Obviously it is a kernel bug.

Unable to run minimal example

Similarly i can not run minimal example in Centos7，kernel 4.18.9.
But I am able to run the minimal compiled in kernel 5.4（Centos7）
can anybody help me？

`[root@localhost c]# ./minimal

libbpf: loading object 'minimal_bpf' from buffer
libbpf: elf: section(2) tp/syscalls/sys_enter_write, size 104, link 0, flags 6, type=1
libbpf: sec 'tp/syscalls/sys_enter_write': found program 'handle_tp' at insn offset 0 (0 bytes), code size 13 insns (104 bytes)
libbpf: elf: section(3) license, size 13, link 0, flags 3, type=1
libbpf: license of minimal_bpf is Dual BSD/GPL
libbpf: elf: section(4) .bss, size 4, link 0, flags 3, type=8
libbpf: elf: section(5) .rodata, size 28, link 0, flags 2, type=1
libbpf: elf: section(6) .BTF, size 591, link 0, flags 0, type=1
libbpf: elf: section(7) .BTF.ext, size 160, link 0, flags 0, type=1
libbpf: elf: section(8) .symtab, size 192, link 13, flags 0, type=2
libbpf: elf: section(9) .reltp/syscalls/sys_enter_write, size 32, link 8, flags 0, type=9
libbpf: looking for externs among 8 symbols...
libbpf: collected 0 externs total
libbpf: map 'minimal_.bss' (global data): at sec_idx 4, offset 0, flags 400.
libbpf: map 0 is "minimal_.bss"
libbpf: map 'minimal_.rodata' (global data): at sec_idx 5, offset 0, flags 480.
libbpf: map 1 is "minimal_.rodata"
libbpf: sec '.reltp/syscalls/sys_enter_write': collecting relocation for section(2) 'tp/syscalls/sys_enter_write'
libbpf: sec '.reltp/syscalls/sys_enter_write': relo #0: insn #2 against 'my_pid'
libbpf: prog 'handle_tp': found data map 0 (minimal_.bss, sec 4, off 0) for insn 2
libbpf: sec '.reltp/syscalls/sys_enter_write': relo #1: insn #6 against '.rodata'
libbpf: prog 'handle_tp': found data map 1 (minimal_.rodata, sec 5, off 0) for insn 6
libbpf: map 'minimal_.bss': skipped auto-creating...
libbpf: map 'minimal_.rodata': skipped auto-creating...
libbpf: prog 'handle_tp': relo #0: poisoning insn #2 that loads map #0 'minimal_.bss'
libbpf: prog 'handle_tp': relo #1: poisoning insn #6 that loads map #1 'minimal_.rodata'
libbpf: prog 'handle_tp': BPF program load failed: Invalid argument
libbpf: prog 'handle_tp': -- BEGIN PROG LOAD LOG --
0: (85) call bpf_get_current_pid_tgid#14
1: (77) r0 >>= 32
2:
BPF map 'minimal_.bss' is referenced but wasn't created
-- END PROG LOAD LOG --
libbpf: failed to load program 'handle_tp'
libbpf: failed to load object 'minimal_bpf'
libbpf: failed to load BPF skeleton 'minimal_bpf': -22
Failed to load and verify BPF skeleton`

CPU arch for tools/bpftool

Currently the executable tools/bpftool is a x86_64 ELF.

How am I suppose to compile the examples/c on an aarch64 computer?

Could somebody help me please? Thanks.

Relocation error on 32 bit systems

This program will compile and run fine on my 64 bit system, but it will fail with a relocation error on 32 bit systems:

SEC("tp/raw_syscalls/sys_enter")
int sys_enter(struct trace_event_raw_sys_enter *ctx) {
        long int n = ctx->id;
        bpf_printk("hello world %d", n);
        return 0;
}

libbpf: prog 'sys_enter': relo #0: insn #0 (LDX/ST/STX) accesses field incorrectly. Make sure you are accessing pointers, unsigned integers, or fields of matching type and size.
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
R1 type=ctx expected=fp
; long int n = ctx->id;
0: (85) call unknown#195896080
invalid func unknown#195896080
processed 1 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
-- END PROG LOAD LOG --
libbpf: failed to load program 'sys_enter'
libbpf: failed to load object 'bootstrap_bpf'
libbpf: failed to load BPF skeleton 'bootstrap_bpf': -22
Failed to load and verify BPF skeleton

I'm cross-compiling using a Yocto build. I've reproduced this both with arm and x86.

From my understanding, the issue comes from the long int in trace_event_raw_sys_enter, which is 64 bit in the compiled eBPF program, but 32 bit in the target kernel.

struct trace_event_raw_sys_enter {        
        struct trace_entry ent;
        long int id;
        long unsigned int args[6];
        char __data[0];
} __attribute__((preserve_access_index));

Indeed, manually changing the id definition in vmlinux.h will fix the relocation error:

struct trace_event_raw_sys_enter {
        u32 id;
} __attribute__((preserve_access_index));

Q: clang flag for target bpf? hints that using a native target could help, but I guess that would completely break CORE relocations since preserve_access_index is a -target bpf-specific attribute, right?

Am I missing something? If I had to fix the issue right now I would replace all long definitions in vmlinux.h to u32 when targeting 32 bit systems. Shouldn't bpftool btf dump handle this?
We're using eBPF on embedded systems, where 32 bit is still fairly common.

Thanks.

Building using cmake is not working

When trying to build the project using cmake, I've got the following error:

bash: line 1: /home/mariusz/git/libbpf-bootstrap/examples/c/../../tools/bpftool: No such file or directory
make[2]: *** [CMakeFiles/bootstrap.dir/build.make:74: bootstrap.skel.h] Error 127
make[2]: *** Deleting file 'bootstrap.skel.h'
make[1]: *** [CMakeFiles/Makefile2:145: CMakeFiles/bootstrap.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

minimal example can't run

After compile and build the minimal example application, I try to run it but with the following error message:

libbpf: loading object 'minimal_bpf' from buffer
libbpf: elf: section(2) tp/syscalls/sys_enter_write, size 104, link 0, flags 6, type=1
libbpf: sec 'tp/syscalls/sys_enter_write': found program 'handle_tp' at insn offset 0 (0 bytes), code size 13 insns (104 bytes)
libbpf: elf: section(3) license, size 13, link 0, flags 3, type=1
libbpf: license of minimal_bpf is Dual BSD/GPL
libbpf: elf: section(4) .bss, size 4, link 0, flags 3, type=8
libbpf: elf: section(5) .rodata, size 28, link 0, flags 2, type=1
libbpf: elf: section(6) .BTF, size 617, link 0, flags 0, type=1
libbpf: elf: section(7) .BTF.ext, size 160, link 0, flags 0, type=1
libbpf: elf: section(8) .symtab, size 216, link 13, flags 0, type=2
libbpf: elf: section(9) .reltp/syscalls/sys_enter_write, size 32, link 8, flags 0, type=9
libbpf: looking for externs among 9 symbols...
libbpf: collected 0 externs total
libbpf: map 'minimal_.bss' (global data): at sec_idx 4, offset 0, flags 400.
libbpf: map 0 is "minimal_.bss"
libbpf: map 'minimal_.rodata' (global data): at sec_idx 5, offset 0, flags 480.
libbpf: map 1 is "minimal_.rodata"
libbpf: sec '.reltp/syscalls/sys_enter_write': collecting relocation for section(2) 'tp/syscalls/sys_enter_write'
libbpf: sec '.reltp/syscalls/sys_enter_write': relo #0: insn #2 against 'my_pid'
libbpf: prog 'handle_tp': found data map 0 (minimal_.bss, sec 4, off 0) for insn 2
libbpf: sec '.reltp/syscalls/sys_enter_write': relo #1: insn #6 against '.rodata'
libbpf: prog 'handle_tp': found data map 1 (minimal_.rodata, sec 5, off 0) for insn 6
libbpf: Kernel doesn't support BTF, skipping uploading it.
libbpf: prog 'handle_tp': relo #0: kernel doesn't support global data
libbpf: prog 'handle_tp': failed to relocate data references: -95
libbpf: failed to load object 'minimal_bpf'
libbpf: failed to load BPF skeleton 'minimal_bpf': -95
Failed to load and verify BPF skeleton

My kernel version is: 4.15.0-96-generic. I run the example on Ubuntu-18.04 system.

Make Kprobe fails on arm64

I am running an arm64 ubuntu VM on my M1 macbook and make kprobe does not compile. Other examples except fentry work as expected.

$ uname -a
Linux devvm 5.13.0-21-generic #21-Ubuntu SMP Tue Oct 19 09:01:50 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

$ clang -v
Ubuntu clang version 13.0.0-2
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/aarch64-linux-gnu/11
Selected GCC installation: /usr/bin/../lib/gcc/aarch64-linux-gnu/11
Candidate multilib: .;@m64
Selected multilib: .;@m64

Error:

$ make kprobe
  BPF      .output/kprobe.bpf.o
kprobe.bpf.c:11:5: error: incomplete definition of type 'struct user_pt_regs'
int BPF_KPROBE(do_unlinkat, int dfd, struct filename *name)
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.output/bpf/bpf_tracing.h:429:20: note: expanded from macro 'BPF_KPROBE'
        return ____##name(___bpf_kprobe_args(args));                        \
                          ^~~~~~~~~~~~~~~~~~~~~~~~
.output/bpf/bpf_tracing.h:409:2: note: expanded from macro '___bpf_kprobe_args'
        ___bpf_apply(___bpf_kprobe_args, ___bpf_narg(args))(args)
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.output/bpf/bpf_helpers.h:165:29: note: expanded from macro '___bpf_apply'
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
                            ^
note: (skipping 2 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
.output/bpf/bpf_tracing.h:401:2: note: expanded from macro '___bpf_kprobe_args2'
        ___bpf_kprobe_args1(args), (void *)PT_REGS_PARM2(ctx)
        ^~~~~~~~~~~~~~~~~~~~~~~~~
.output/bpf/bpf_tracing.h:399:33: note: expanded from macro '___bpf_kprobe_args1'
        ___bpf_kprobe_args0(), (void *)PT_REGS_PARM1(ctx)
                                       ^~~~~~~~~~~~~~~~~~
.output/bpf/bpf_tracing.h:195:49: note: expanded from macro 'PT_REGS_PARM1'
#define PT_REGS_PARM1(x) (((PT_REGS_ARM64 *)(x))->regs[0])
                          ~~~~~~~~~~~~~~~~~~~~~~^
kprobe.bpf.c:11:5: note: forward declaration of 'struct user_pt_regs'
.output/bpf/bpf_tracing.h:429:20: note: expanded from macro 'BPF_KPROBE'
        return ____##name(___bpf_kprobe_args(args));                        \
                          ^
.output/bpf/bpf_tracing.h:409:2: note: expanded from macro '___bpf_kprobe_args'
        ___bpf_apply(___bpf_kprobe_args, ___bpf_narg(args))(args)
        ^
.output/bpf/bpf_helpers.h:165:29: note: expanded from macro '___bpf_apply'
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
                            ^
note: (skipping 3 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
.output/bpf/bpf_tracing.h:399:33: note: expanded from macro '___bpf_kprobe_args1'
        ___bpf_kprobe_args0(), (void *)PT_REGS_PARM1(ctx)
                                       ^
.output/bpf/bpf_tracing.h:195:29: note: expanded from macro 'PT_REGS_PARM1'
#define PT_REGS_PARM1(x) (((PT_REGS_ARM64 *)(x))->regs[0])
                            ^
.output/bpf/bpf_tracing.h:194:45: note: expanded from macro 'PT_REGS_ARM64'
#define PT_REGS_ARM64 const volatile struct user_pt_regs
                                            ^

Any ideas what may be going wrong here?

BPF_KPROBE macro provides unexpected value of function argument

Hello! While playing with libbpf-bootstrap I'm getting unexpected (and strange) function argument for kprobe syscalls. For example for kprobe on close syscall with int close(inf fd) signature, I got enormous fd values like fd=15761240 while expected small int like fd=4. Reproduced this on Debian 11 x64 (kernel 5.10.0-7-amd64) and Ubuntu 21.10 x64 (kernel ~5.13).

Debug code:

#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>

char LICENSE[] SEC("license") = "Dual BSD/GPL";

// close accept4 syscall
// int accept4(int sockfd, struct sockaddr *restrict addr, socklen_t *restrict addrlen);
SEC("kretprobe/__x64_sys_accept4")
int BPF_KRETPROBE(accept, int ret) {
	u64 id = bpf_get_current_pid_tgid();
	u32 pid = id >> 32;

	// filter specific pid for simplicity
	if (pid != 31114 || ret < 0) {
		return 0;
	}

	// debug returned file descriptor
	bpf_printk("opened pid=%d fd=%d", pid, ret);
	return 0;
}

// close syscall
// int close(int fd);
SEC("kprobe/__x64_sys_close")
int BPF_KPROBE(close, int fd) {
	u64 id = bpf_get_current_pid_tgid();
	u32 pid = id >> 32;
	// filter specific pid for simplicity
	if (pid != 31114) {
		return 0;
	}

	// debug fd arg (expected to be equal to fd returned on accept4)
	bpf_printk("closed pid=%d fd=%d", pid, fd);
    return 0;
}

Results

$ cat /sys/kernel/debug/tracing/trace_pipe
            main-31114   [001] d...  9069.254408: bpf_trace_printk: opened pid=31114 fd=4
            main-31114   [001] d...  9069.321946: bpf_trace_printk: closed pid=31114 fd=15761240

I tried to alter vmlinux.h: at first with vmlinux.h delivered by libbbpf-bootstrap and then with "native" vmlinux.h from the instance OS kernel and on both ways I got the issue above.

Also tried to run the same bpf program in BCC way (compiled with bcc at run-time) with kprobes declared without BPF_KPROBE macro, like that:

int syscall__probe_close_entry(struct pt_regs *ctx, int fd) { ... }

and it worked as expected: fd=4 at all the debug points.
Is it a BPF_KPROBE macro bug/incompatibility with the kernel or I'm missing something?

build error compiling C minimal example - 'make minimal' in examples/c

When trying to run make minimal in the examples/c directory, getting the following errors - the full output in gist
https://gist.github.com/dmitris/697849eb3c0b80e6f2ca50430cae499b:

skeleton/pid_iter.bpf.c:47:14: error: incomplete definition of type 'struct bpf_perf_link'
[...]
skeleton/pid_iter.bpf.c:44:9: note: forward declaration of 'struct bpf_perf_link'
        struct bpf_perf_link *perf_link;
               ^
skeleton/pid_iter.bpf.c:48:10: error: incomplete definition of type 'struct bpf_perf_link'
        event = BPF_CORE_READ(perf_link, perf_file, private_data);
[...]
skeleton/pid_iter.bpf.c:49:30: error: no member named 'bpf_cookie' in 'struct perf_event'
        return BPF_CORE_READ(event, bpf_cookie);

Is it related to (or even fixed in) https://lore.kernel.org/bpf/[email protected]/ and
https://lore.kernel.org/bpf/[email protected]/ ?

I got that error with the current rhel8 and Fedora 35 - Linux mybox.example.com 5.14.10-300.fc35.x86_64 #1 SMP Thu Oct 7 20:48:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

compile error while running make in example/c

git clone and git submodule update --init --recursive
run make in example/c dir

BPF      .output/usdt.bpf.o
In file included from usdt.bpf.c:6:
.output/bpf/usdt.bpf.h:94:7: error: use of unknown builtin '__builtin_preserve_enum_value' [-Wimplicit-function-declaration]
        if (!BPF_USDT_HAS_BPF_COOKIE) {
             ^
.output/bpf/usdt.bpf.h:39:2: note: expanded from macro 'BPF_USDT_HAS_BPF_COOKIE'
        bpf_core_enum_value_exists(enum bpf_func_id___usdt, BPF_FUNC_get_attach_cookie___usdt)
        ^
.output/bpf/bpf_core_read.h:205:2: note: expanded from macro 'bpf_core_enum_value_exists'
        __builtin_preserve_enum_value(*(typeof(enum_type) *)enum_value, BPF_ENUMVAL_EXISTS)
        ^
1 error generated.
make: *** [Makefile:106: .output/usdt.bpf.o] Error 1

ubuntu 20.04 with 5.13.0-52-generic kernel

make failed

[root@x src]# yum install clang

[root@x src]# make
MKDIR .output
MKDIR .output/libbpf
LIB libbpf.a
MKDIR staticobjs
CC bpf.o
CC btf.o
CC libbpf.o
CC libbpf_errno.o
CC netlink.o
CC nlattr.o
CC str_error.o
CC libbpf_probes.o
CC bpf_prog_linfo.o
CC xsk.o
CC btf_dump.o
CC hashmap.o
CC ringbuf.o
AR libbpf.a
INSTALL bpf.h libbpf.h btf.h xsk.h libbpf_util.h bpf_helpers.h bpf_helper_defs.h bpf_tracing.h bpf_endian.h bpf_core_read.h libbpf_common.h
INSTALL libbpf.pc
INSTALL libbpf.a
BPF .output/minimal.bpf.o
error: unknown target triple 'bpf', please use -triple or -arch
make: *** [.output/minimal.bpf.o]

Unable to build minimal example with Makefile

I pulled down the repo and init'd the submodules

mike@framework:~/Development/libbpf-bootstrap$ git submodule init
mike@framework:~/Development/libbpf-bootstrap$ git submodule update
Cloning into '/home/mike/Development/libbpf-bootstrap/bpftool'...
Cloning into '/home/mike/Development/libbpf-bootstrap/libbpf'...
Submodule path 'bpftool': checked out 'cdd0b425fce958e3d02424c9d0296e6d0f7e323f'
Submodule path 'libbpf': checked out '9c44c8a8e01cf86bc801c3b72324358d5ea99e50'

mike@framework:~/Development/libbpf-bootstrap$ cd examples/c/
mike@framework:~/Development/libbpf-bootstrap/examples/c$ make minimal
  MKDIR    .output
  MKDIR    .output/libbpf

... (omitting a bunch of build output)

 INSTALL  /home/mike/Development/libbpf-bootstrap/examples/c/.output//libbpf/libbpf.a 
  BPF      .output/minimal.bpf.o
  MKDIR    bpftool/
  BPFTOOL  bpftool/bpftool
...                        libbfd: [ OFF ]
...        disassembler-four-args: [ OFF ]
...                          zlib: [ on  ]
...                        libcap: [ OFF ]
...               clang-bpf-co-re: [ on  ]
  MKDIR    /home/mike/Development/libbpf-bootstrap/examples/c/.output/bpftool//libbpf/
make[2]: *** /home/mike/Development/libbpf-bootstrap/bpftool/libbpf/src: No such file or directory.  Stop.
make[1]: *** [Makefile:44: /home/mike/Development/libbpf-bootstrap/examples/c/.output/bpftool//libbpf/libbpf.a] Error 2
make: *** [Makefile:65: /home/mike/Development/libbpf-bootstrap/examples/c/.output/bpftool/bpftool] Error 2

bpftool takes libbpf as a submodule but never inits it so there's nothing in that dir. Is there any (easy) way to get it so that bpftool just uses the libbpf from this repo or can we just have it init its own submodule in this makefile?

Loading of bpf fails with invalid argument

Latest (c126589) libbpf-bootstrap gives errors when loading on Debian testing x86_64 with kernel 5.10.17. Actually, already the compilation step is a bit fishy, as it says:

libbpf-bootstrap/src(master)$ make
  MKDIR    .output
  MKDIR    .output/libbpf
  LIB      libbpf.a
  MKDIR    staticobjs
  CC       bpf.o
  CC       btf.o
  CC       libbpf.o
  CC       libbpf_errno.o
  CC       netlink.o
  CC       nlattr.o
  CC       str_error.o
  CC       libbpf_probes.o
  CC       bpf_prog_linfo.o
  CC       xsk.o
  CC       btf_dump.o
  CC       hashmap.o
  CC       ringbuf.o
  AR       libbpf.a
  INSTALL  bpf.h libbpf.h btf.h xsk.h libbpf_util.h bpf_helpers.h bpf_helper_defs.h bpf_tracing.h bpf_endian.h bpf_core_read.h libbpf_common.h
  INSTALL  libbpf.pc
  INSTALL  libbpf.a
  BPF      .output/minimal.bpf.o
  GEN-SKEL .output/minimal.skel.h
libbpf: elf: skipping unrecognized data section(5) .rodata.str1.1
  CC       .output/minimal.o
  BINARY   minimal
  BPF      .output/bootstrap.bpf.o
  GEN-SKEL .output/bootstrap.skel.h
  CC       .output/bootstrap.o
  BINARY   bootstrap
  BPF      .output/uprobe.bpf.o
  GEN-SKEL .output/uprobe.skel.h
libbpf: elf: skipping unrecognized data section(5) .rodata.str1.1
  CC       .output/uprobe.o
  BINARY   uprobe
  BPF      .output/kprobe.bpf.o
  GEN-SKEL .output/kprobe.skel.h
libbpf: elf: skipping unrecognized data section(5) .rodata.str1.1
  CC       .output/kprobe.o
  BINARY   kprobe
  BPF      .output/fentry.bpf.o
  GEN-SKEL .output/fentry.skel.h
libbpf: elf: skipping unrecognized data section(5) .rodata.str1.1
  CC       .output/fentry.o
  BINARY   fentry

In all three cases of GEN-SKEL, the output happens in response to

...
llvm-strip -g .output/minimal.bpf.o # strip useless DWARF info
libbpf-bootstrap/tools/bpftool gen skeleton .output/minimal.bpf.o > .output/minimal.skel.h
libbpf: elf: skipping unrecognized data section(5) .rodata.str1.1
...

But then running 'minimal' fails with

libbpf-bootstrap/src(master)# ./minimal 
libbpf: loading object 'minimal_bpf' from buffer
libbpf: elf: section(2) tp/syscalls/sys_enter_write, size 192, link 0, flags 6, type=1
libbpf: sec 'tp/syscalls/sys_enter_write': found program 'handle_tp' at insn offset 0 (0 bytes), code size 24 insns (192 bytes)
libbpf: elf: section(3) license, size 13, link 0, flags 3, type=1
libbpf: license of minimal_bpf is Dual BSD/GPL
libbpf: elf: section(4) .bss, size 4, link 0, flags 3, type=8
libbpf: elf: section(5) .rodata.str1.1, size 28, link 0, flags 32, type=1
libbpf: elf: skipping unrecognized data section(5) .rodata.str1.1
libbpf: elf: section(6) .BTF, size 494, link 0, flags 0, type=1
libbpf: elf: section(7) .BTF.ext, size 160, link 0, flags 0, type=1
libbpf: elf: section(8) .symtab, size 144, link 13, flags 0, type=2
libbpf: elf: section(9) .reltp/syscalls/sys_enter_write, size 16, link 8, flags 0, type=9
libbpf: looking for externs among 6 symbols...
libbpf: collected 0 externs total
libbpf: map 'minimal_.bss' (global data): at sec_idx 4, offset 0, flags 400.
libbpf: map 0 is "minimal_.bss"
libbpf: sec '.reltp/syscalls/sys_enter_write': collecting relocation for section(2) 'tp/syscalls/sys_enter_write'
libbpf: sec '.reltp/syscalls/sys_enter_write': relo #0: insn #2 against 'my_pid'
libbpf: prog 'handle_tp': found data map 0 (minimal_.bss, sec 4, off 0) for insn 2
libbpf: map 'minimal_.bss': created successfully, fd=4
libbpf: load bpf program failed: Invalid argument
libbpf: failed to load program 'handle_tp'
libbpf: failed to load object 'minimal_bpf'
libbpf: failed to load BPF skeleton 'minimal_bpf': -22
Failed to load and verify BPF skeleton

The other programs also fail at the exact same point when loading eBPF programs and with the same error. Might this be an issue with the kernel (v5.10.17, compile by me), clang/llvm (Debian testing versions 11.0.1-2 and 11.0.1), Debian (testing), libbpf-bootstrap (c126589) or something else?

Where do we get vmlinux.h?

Hi team,

I'm learning ebpf, and would like to build an app from scratch. So I'd like to know where to get vmlinux.h?

BTW, do we have slack?

-- K

Discussion about socket examples

Greeting,

Thanks for the hard work and great examples.

Since there are more and more SEC types now supported by the kernel:
https://github.com/libbpf/libbpf/blob/4eb6485c08867edaa5a0a81c64ddb23580420340/src/libbpf.c#L9002-L9081

I think it would be great if we could cover more SEC types.

I wonder if I can add an example about SEC(socket), which can get an insight about how to retrieve data from __sk_buff, maybe start from most used UDP and TCP on top of IP protocol.

Do you think it's a good idea?

Question regarding the Rust Examples' Readme

Hello!
I was wondering why you build & generate the skel manually (using cargo libbpf <...>) rather than using cargo build like you did at the end.
These commands generate another .skel.rs and .mod files plus keeping the .bpf.o file which cargo build discards automatically.

$ cd examples/rust
$ cargo libbpf build
$ cargo libbpf gen
$ cargo build --release
$ sudo ./target/release/xdp 1
<...>

Thanks!

example/c/bootstrap.c cleanup code path is not suitable?

	/* Load & verify BPF programs */
	err = bootstrap_bpf__load(skel);
	if (err) {
		fprintf(stderr, "Failed to load and verify BPF skeleton\n");
                // No need to cleanup, just return, bootstrap_bpf__destroy may work after bootstrap_bpf__load successfully?
		goto cleanup;
	}
...
cleanup:
	/* Clean up */
        /*some code path go here, rb may not be ring_buffer__new yet */
	ring_buffer__free(rb);
	bootstrap_bpf__destroy(skel);

build error for android

$ xmake f -p android -a arm64-v8a
$ xmake 

[ 11%]: compiling.bpf fentry.bpf.c
[ 11%]: ccache compiling.release ../../libbpf/src/bpf_prog_linfo.c
[ 11%]: ccache compiling.release ../../libbpf/src/netlink.c
[ 11%]: ccache compiling.release ../../libbpf/src/bpf.c
[ 11%]: ccache compiling.release ../../libbpf/src/hashmap.c
[ 11%]: ccache compiling.release ../../libbpf/src/ringbuf.c
[ 11%]: ccache compiling.release ../../libbpf/src/btf_dump.c
[ 11%]: ccache compiling.release ../../libbpf/src/str_error.c
[ 11%]: ccache compiling.release ../../libbpf/src/libbpf_probes.c
[ 11%]: compiling.bpf minimal.bpf.c
[ 11%]: ccache compiling.release ../../libbpf/src/linker.c
[ 11%]: ccache compiling.release ../../libbpf/src/btf.c
[ 11%]: ccache compiling.release ../../libbpf/src/libbpf.c
[ 11%]: ccache compiling.release ../../libbpf/src/nlattr.c
[ 11%]: compiling.bpf uprobe.bpf.c
[ 11%]: ccache compiling.release ../../libbpf/src/xsk.c
[ 11%]: compiling.bpf bootstrap.bpf.c
[ 11%]: ccache compiling.release ../../libbpf/src/strset.c
[ 13%]: ccache compiling.release ../../libbpf/src/libbpf_errno.c
libbpf: elf: skipping unrecognized data section(6) .rodata.str1.1
libbpf: elf: skipping unrecognized data section(7) .rodata.str1.1
[ 18%]: ccache compiling.release ../../libbpf/src/gen_loader.c
libbpf: elf: skipping unrecognized data section(6) .rodata.str1.1
[ 58%]: archiving.release libbpf.a
[ 60%]: ccache compiling.release fentry.c
[ 60%]: ccache compiling.release fentry.bpf.c
[ 60%]: ccache compiling.release bootstrap.bpf.c
[ 60%]: ccache compiling.release bootstrap.c
[ 60%]: ccache compiling.release minimal.bpf.c
[ 60%]: ccache compiling.release minimal.c
[ 60%]: ccache compiling.release uprobe.bpf.c
[ 60%]: ccache compiling.release uprobe.c
error: bootstrap.bpf.c:53:12: error: using builtin_preserve_access_index() without -g
        e->ppid = BPF_CORE_READ(task, real_parent, tgid);
                  ^
build/bpf/bpf_core_read.h:404:2: note: expanded from macro 'BPF_CORE_READ'
        BPF_CORE_READ_INTO(&__r, (src), a, ##__VA_ARGS__);                  \
        ^
build/bpf/bpf_core_read.h:311:30: note: expanded from macro 'BPF_CORE_READ_INTO'
        ___core_read(bpf_core_read, bpf_core_read,                          \
                                    ^
bootstrap.bpf.c:53:12: error: using builtin_preserve_access_index() without -g
build/bpf/bpf_core_read.h:404:2: note: expanded from macro 'BPF_CORE_READ'
        BPF_CORE_READ_INTO(&__r, (src), a, ##__VA_ARGS__);                  \
        ^
build/bpf/bpf_core_read.h:311:15: note: expanded from macro 'BPF_CORE_READ_INTO'
        ___core_read(bpf_core_read, bpf_core_read,                          \
                     ^
11692 warnings and 2 errors generated.

Compilation erros on ubuntu x86_64

I try build it on my ubuntu x86_64, but it fails.

libbpf-bootstrap/src$ make
  MKDIR    .output
  MKDIR    .output/libbpf
  LIB      libbpf.a
  MKDIR    staticobjs
  CC       bpf.o
  CC       btf.o
  CC       libbpf.o
  CC       libbpf_errno.o
  CC       netlink.o
  CC       nlattr.o
  CC       str_error.o
  CC       libbpf_probes.o
  CC       bpf_prog_linfo.o
  CC       xsk.o
  CC       btf_dump.o
  CC       hashmap.o
  CC       ringbuf.o
  AR       libbpf.a
  INSTALL  bpf.h libbpf.h btf.h xsk.h libbpf_util.h bpf_helpers.h bpf_helper_defs.h bpf_tracing.h bpf_endian.h bpf_core_read.h libbpf_common.h
  INSTALL  libbpf.pc
  INSTALL  libbpf.a
  BPF      .output/minimal.bpf.o
In file included from minimal.bpf.c:4:
.output/bpf/bpf_helpers.h:99:10: error: unknown register name 'r0' in asm
                     : "r0", "r1", "r2", "r3", "r4", "r5");
                       ^
1 error generated.
Makefile:59: recipe for target '.output/minimal.bpf.o' failed
make: *** [.output/minimal.bpf.o] Error 1

r0? Why does it generate header files for arm architecture?

failed to run kprobe example on arm64

Hi team,

I failed to run kprobe example n arm64 as below,

root@tegra-ubuntu:/ota# ./kprobe -v
libbpf: loading object 'kprobe_bpf' from buffer
libbpf: elf: section(2) kprobe/do_unlinkat, size 152, link 0, flags 6, type=1
libbpf: sec 'kprobe/do_unlinkat': found program 'do_unlinkat' at insn offset 0 (0 bytes), code size 19 insns (152 bytes)
libbpf: elf: section(3) kretprobe/do_unlinkat, size 88, link 0, flags 6, type=1
libbpf: sec 'kretprobe/do_unlinkat': found program 'do_unlinkat_exit' at insn offset 0 (0 bytes), code size 11 insns (88 bytes)
libbpf: elf: section(4) license, size 13, link 0, flags 3, type=1
libbpf: license of kprobe_bpf is Dual BSD/GPL
libbpf: elf: section(5) .rodata, size 72, link 0, flags 2, type=1
libbpf: elf: section(6) .BTF, size 1467, link 0, flags 0, type=1
libbpf: elf: section(7) .BTF.ext, size 364, link 0, flags 0, type=1
libbpf: elf: section(8) .symtab, size 240, link 14, flags 0, type=2
libbpf: elf: section(9) .relkprobe/do_unlinkat, size 16, link 8, flags 0, type=9
libbpf: elf: section(10) .relkretprobe/do_unlinkat, size 16, link 8, flags 0, type=9
libbpf: looking for externs among 10 symbols...
libbpf: collected 0 externs total
libbpf: map 'kprobe_b.rodata' (global data): at sec_idx 5, offset 0, flags 480.
libbpf: map 0 is "kprobe_b.rodata"
libbpf: sec '.relkprobe/do_unlinkat': collecting relocation for section(2) 'kprobe/do_unlinkat'
libbpf: sec '.relkprobe/do_unlinkat': relo #0: insn #12 against '.rodata'
libbpf: prog 'do_unlinkat': found data map 0 (kprobe_b.rodata, sec 5, off 0) for insn 12
libbpf: sec '.relkretprobe/do_unlinkat': collecting relocation for section(3) 'kretprobe/do_unlinkat'
libbpf: sec '.relkretprobe/do_unlinkat': relo #0: insn #3 against '.rodata'
libbpf: prog 'do_unlinkat_exit': found data map 0 (kprobe_b.rodata, sec 5, off 0) for insn 3
libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': 0
libbpf: map 'kprobe_b.rodata': created successfully, fd=4
libbpf: sec 'kprobe/do_unlinkat': found 2 CO-RE relocations
libbpf: CO-RE relocating [2] struct pt_regs: found target candidate [248] struct pt_regs in [vmlinux]
libbpf: prog 'do_unlinkat': relo #0: <byte_off> [2] struct pt_regs.si (0:13 @ offset 104)
libbpf: prog 'do_unlinkat': relo #0: non-matching candidate #0 <byte_off> [248] struct pt_regs (0 @ offset 0)
libbpf: prog 'do_unlinkat': relo #0: no matching targets found
libbpf: prog 'do_unlinkat': relo #0: substituting insn #0 w/ invalid insn
libbpf: CO-RE relocating [7] struct filename: found target candidate [1173] struct filename in [vmlinux]
libbpf: prog 'do_unlinkat': relo #1: <byte_off> [7] struct filename.name (0:0 @ offset 0)
libbpf: prog 'do_unlinkat': relo #1: matching candidate #0 <byte_off> [1173] struct filename.name (0:0 @ offset 0)
libbpf: prog 'do_unlinkat': relo #1: patched insn #3 (ALU/ALU64) imm 0 -> 0
libbpf: sec 'kretprobe/do_unlinkat': found 1 CO-RE relocations
libbpf: prog 'do_unlinkat_exit': relo #0: <byte_off> [2] struct pt_regs.ax (0:10 @ offset 80)
libbpf: prog 'do_unlinkat_exit': relo #0: non-matching candidate #0 <byte_off> [248] struct pt_regs (0 @ offset 0)
libbpf: prog 'do_unlinkat_exit': relo #0: no matching targets found
libbpf: prog 'do_unlinkat_exit': relo #0: substituting insn #0 w/ invalid insn
libbpf: prog 'do_unlinkat': BPF program load failed: Invalid argument
libbpf: prog 'do_unlinkat': -- BEGIN PROG LOAD LOG --
Unrecognized arg#0 type PTR
; int BPF_KPROBE(do_unlinkat, int dfd, struct filename *name)
0: <invalid CO-RE relocation>
failed to resolve CO-RE relocation <byte_off> [2] struct pt_regs.si (0:13 @ offset 104)
processed 1 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
-- END PROG LOAD LOG --
libbpf: failed to load program 'do_unlinkat'
libbpf: failed to load object 'kprobe_bpf'
libbpf: failed to load BPF skeleton 'kprobe_bpf': -22
Failed to open BPF skeleton

Cound someone point out what's wrong here? Thanks.

"cargo build" errors: 'bpf/bpf_helpers.h file not found' or 'Failed to generate skeleton ... Failed to spawn rustfmt '

when "cargo build --release", this error (screenshot error1) came at first, I found the "bpf/bpf_helpers.h" was generated in 2 pathes ( find /home/vagrant/libbpf-bootstrap/ -name "bpf_helpers.h" got "screenshot find"), so cop bpf directorys one to /usr/include (in fact I fixed "vmlinux.h not found" error in the same way after being generated by bpftool ), but got another error: screenshot error2, I got no way to continue... BTW. it works well for the C version in /home/vagrant/libbpf-bootstrap/examples/rust/

screenshot error1

text:

   Compiling xdp v0.1.0 (/home/vagrant/libbpf-bootstrap/examples/rust/xdp)
error: failed to run custom build command for `xdp v0.1.0 (/home/vagrant/libbpf-bootstrap/examples/rust/xdp)`

Caused by:
  process didn't exit successfully: `/home/vagrant/libbpf-bootstrap/examples/rust/target/release/build/xdp-ea3972c5b26f2632/build-script-build` (exit code: 101)
  --- stderr
  thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Build("Failed to compile obj=/tmp/.tmpIAhV9L/xdppass.o with status=exit code: 1\n stdout=\n \n stderr=\n ./src/bpf/xdppass.bpf.c:2:10: fatal error: \'bpf/bpf_helpers.h\' file not found\n#include <bpf/bpf_helpers.h>\n
         ^~~~~~~~~~~~~~~~~~~\n1 error generated.\n\n")', xdp/build.rs:19:47
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: build failed

screenshot find

screenshot error2

text

~/libbpf-bootstrap/examples/rust$ sudo cargo build --release
   Compiling tracecon v0.1.0 (/home/vagrant/libbpf-bootstrap/examples/rust/tracecon)
   Compiling xdp v0.1.0 (/home/vagrant/libbpf-bootstrap/examples/rust/xdp)
error: failed to run custom build command for `xdp v0.1.0 (/home/vagrant/libbpf-bootstrap/examples/rust/xdp)`

Caused by:
  process didn't exit successfully: `/home/vagrant/libbpf-bootstrap/examples/rust/target/release/build/xdp-ea3972c5b26f2632/build-script-build` (exit code: 101)
  --- stderr
  libbpf: elf: skipping unrecognized data section(4) .rodata.str1.1
  libbpf: elf: skipping unrecognized data section(4) .rodata.str1.1
  Warning: unrecognized map: license
  thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Generate("Failed to generate skeleton for /tmp/.tmpPpu8kb/xdppass.o: Failed to spawn rustfmt")', xdp/build.rs:19:47
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: build failed

libbpf-rs XDP example attach error

I'm using Vagrant (box: bento/ubuntu-20.10) to compile and run the xdp example. Compilation works fine, but when I run the app I get this error (I added the debug option):

vagrant@vagrant:/vagrant$ sudo ./target/debug/xdppass
libbpf: loading object 'xdppass_bpf' from buffer
libbpf: elf: section(3) xdp, size 120, link 0, flags 6, type=1
libbpf: sec 'xdp': found program 'xdp_pass' at insn offset 0 (0 bytes), code size 15 insns (120 bytes)
libbpf: elf: section(4) .rodata.str1.1, size 16, link 0, flags 32, type=1
libbpf: elf: skipping unrecognized data section(4) .rodata.str1.1
libbpf: elf: section(5) license, size 4, link 0, flags 3, type=1
libbpf: license of xdppass_bpf is GPL
libbpf: elf: section(12) .BTF, size 669, link 0, flags 0, type=1
libbpf: elf: section(14) .BTF.ext, size 220, link 0, flags 0, type=1
libbpf: elf: section(21) .symtab, size 912, link 1, flags 0, type=2
libbpf: looking for externs among 38 symbols...
libbpf: collected 0 externs total
libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': 0
libbpf: sec 'xdp': found 2 CO-RE relocations
libbpf: prog 'xdp_pass': relo #0: kind <byte_off> (0), spec is [2] struct xdp_md.data_end (0:1 @ offset 4)
libbpf: CO-RE relocating [0] struct xdp_md: found target candidate [20548] struct xdp_md in [vmlinux]
libbpf: prog 'xdp_pass': relo #0: matching candidate #0 [20548] struct xdp_md.data_end (0:1 @ offset 4)
libbpf: prog 'xdp_pass': relo #0: patched insn #0 (LDX/ST/STX) off 4 -> 4
libbpf: prog 'xdp_pass': relo #1: kind <byte_off> (0), spec is [2] struct xdp_md.data (0:0 @ offset 0)
libbpf: prog 'xdp_pass': relo #1: matching candidate #0 [20548] struct xdp_md.data (0:0 @ offset 0)
libbpf: prog 'xdp_pass': relo #1: patched insn #1 (LDX/ST/STX) off 0 -> 0
libbpf: prog 'xdp_pass': failed to attach to xdp: Invalid argument
Error: System error, errno: -22

I believe this may be because libbpf needs to use SKB_MODE when running in a virtualized environment, but I don't believe libbpf-rs allows you to set that flag.

why bpf ringbuf can not be used in uprobe of libbpf?

Recently, I am trying to use bpf ringbuf in uprobe example of libbpf. But when running, error occurred which is "libbpf: load bpf program failed: Invalid argument". I have no idea why this happened. Could anyone help? Below is my test code.
Kernel space code: uprobe.bpf.c, define a rb struct, and use bpf_ringbuf_reserve in uprobe code block.

#include <linux/bpf.h>
#include <linux/ptrace.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

char LICENSE[] SEC("license") = "Dual BSD/GPL";

struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} rb SEC(".maps");
SEC("uprobe/func")
int BPF_KPROBE(uprobe, int a, int b)
{
    __u64* e = bpf_ringbuf_reserve(&rb, sizeof(__u64), 0);
    if (!e)
        return 0;
    bpf_printk("UPROBE ENTRY: a = %d, b = %d\n", a, b);
    return 0;
}

SEC("uretprobe/func")
int BPF_KRETPROBE(uretprobe, int ret)
{
    bpf_printk("UPROBE EXIT: return = %d\n", ret);
    return 0;
}

User space code: uprobe.c

#include <errno.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/resource.h>
#include <bpf/libbpf.h>
#include "uprobe.skel.h"

static int libbpf_print_fn(enum libbpf_print_level level, const char *format, va_list args)
{
    return vfprintf(stderr, format, args);
}

static void bump_memlock_rlimit(void)
{
    struct rlimit rlim_new = {
        .rlim_cur   = RLIM_INFINITY,
        .rlim_max   = RLIM_INFINITY,
    };

    if (setrlimit(RLIMIT_MEMLOCK, &rlim_new)) {
        fprintf(stderr, "Failed to increase RLIMIT_MEMLOCK limit!\n");
        exit(1);
    }
}

/* Find process's base load address. We use /proc/self/maps for that,
 * searching for the first executable (r-xp) memory mapping:
 *
 * 5574fd254000-5574fd258000 r-xp 00002000 fd:01 668759                     /usr/bin/cat
 * ^^^^^^^^^^^^                   ^^^^^^^^
 *
 * Subtracting that region's offset (4th column) from its absolute start
 * memory address (1st column) gives us the process's base load address.
 */
static long get_base_addr() {
    size_t start, offset;
    char buf[256];
    FILE *f;

    f = fopen("/proc/self/maps", "r");
    if (!f)
        return -errno;

    while (fscanf(f, "%zx-%*x %s %zx %*[^\n]\n", &start, buf, &offset) == 3) {
        if (strcmp(buf, "r-xp") == 0) {
            fclose(f);
            return start - offset;
        }
    }

    fclose(f);
    return -1;
}

static int handle_event(void *ctx, void *data, size_t data_sz)
{
    return 0;
}

/* It's a global function to make sure compiler doesn't inline it. */
int uprobed_function(int a, int b)
{
    return a + b;
}

int main(int argc, char **argv)
{
    struct ring_buffer *rb = NULL;
    struct uprobe_bpf *skel;
    long base_addr, uprobe_offset;
    int err, i;

    /* Set up libbpf errors and debug info callback */
    libbpf_set_print(libbpf_print_fn);

    /* Bump RLIMIT_MEMLOCK to allow BPF sub-system to do anything */
    bump_memlock_rlimit();

    /* Load and verify BPF application */
    skel = uprobe_bpf__open_and_load();
    if (!skel) {
        fprintf(stderr, "Failed to open and load BPF skeleton\n");
        return 1;
    }

    base_addr = get_base_addr();
    if (base_addr < 0) {
        fprintf(stderr, "Failed to determine process's load address\n");
        err = base_addr;
        goto cleanup;
    }

    /* uprobe/uretprobe expects relative offset of the function to attach
     * to. This offset is relateve to the process's base load address. So
     * easy way to do this is to take an absolute address of the desired
     * function and substract base load address from it.  If we were to
     * parse ELF to calculate this function, we'd need to add .text
     * section offset and function's offset within .text ELF section.
     */
    uprobe_offset = (long)&uprobed_function - base_addr;

    /* Attach tracepoint handler */
    skel->links.uprobe = bpf_program__attach_uprobe(skel->progs.uprobe,
                            false /* not uretprobe */,
                            0 /* self pid */,
                            "/proc/self/exe",
                            uprobe_offset);
    err = libbpf_get_error(skel->links.uprobe);
    if (err) {
        fprintf(stderr, "Failed to attach uprobe: %d\n", err);
        goto cleanup;
    }

    /* we can also attach uprobe/uretprobe to any existing or future
     * processes that use the same binary executable; to do that we need
     * to specify -1 as PID, as we do here
     */
    skel->links.uretprobe = bpf_program__attach_uprobe(skel->progs.uretprobe,
                               true /* uretprobe */,
                               -1 /* any pid */,
                               "/proc/self/exe",
                               uprobe_offset);
    err = libbpf_get_error(skel->links.uretprobe);
    if (err) {
        fprintf(stderr, "Failed to attach uprobe: %d\n", err);
        goto cleanup;
    }

    /* Set up ring buffer polling */
    rb = ring_buffer__new(bpf_map__fd(skel->maps.rb), handle_event, NULL, NULL);
    if (!rb) {
        err = -1;
        fprintf(stderr, "Failed to create ring buffer\n");
        goto cleanup;
    }

    printf("Successfully started! Please run `sudo cat /sys/kernel/debug/tracing/trace_pipe` "
           "to see output of the BPF programs.\n");

    for (i = 0; ; i++) {
        err = ring_buffer__poll(rb, 100 /* timeout, ms */);
        /* trigger our BPF programs */
        fprintf(stderr, ".");
        uprobed_function(i, i + 1);
        sleep(1);
    }

cleanup:
    ring_buffer__free(rb);
    uprobe_bpf__destroy(skel);
    return -err;
}

xdp example

hi
I am currently trying to write an xdp program by libbpf-bootstrap. But get an error

libbpf: loading object 'http_bpf' from buffer
libbpf: elf: section(2) xdp, size 120, link 0, flags 6, type=1
libbpf: sec 'xdp': found program 'xdp_pass' at insn offset 0 (0 bytes), code size 15 insns (120 bytes)
libbpf: elf: section(3) .rodata.str1.1, size 16, link 0, flags 32, type=1
libbpf: elf: skipping unrecognized data section(3) .rodata.str1.1
libbpf: elf: section(4) license, size 4, link 0, flags 3, type=1
libbpf: license of http_bpf is GPL
libbpf: elf: section(5) .BTF, size 706, link 0, flags 0, type=1
libbpf: elf: section(6) .BTF.ext, size 220, link 0, flags 0, type=1
libbpf: elf: section(7) .symtab, size 120, link 11, flags 0, type=2
libbpf: looking for externs among 5 symbols...
libbpf: collected 0 externs total
libbpf: failed to find valid kernel BTF
libbpf: Error loading vmlinux BTF: -3
libbpf: failed to load object 'http_bpf'
libbpf: failed to load BPF skeleton 'http_bpf': -3
Failed to open BPF skeleton

my http.bpf.c

// SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
/* Copyright (c) 2020 Facebook */
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>

SEC("xdp")
int xdp_pass(struct xdp_md *ctx)
{
    void *data = (void *)(long)ctx->data;
    void *data_end = (void *)(long)ctx->data_end;
    int pkt_sz = data_end - data;

    bpf_printk("packet size: %d", pkt_sz);
    return XDP_PASS;
}

char __license[] SEC("license") = "GPL";

my http.c

#include "http.skel.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <getopt.h>

#include <bpf/bpf.h>

#include <net/if.h>
#include <linux/if_link.h> /* depend on kernel-headers installed */

static int libbpf_print_fn(enum libbpf_print_level level, const char *format, va_list args)
{
    return vfprintf(stderr, format, args);
}

int main(int argc, char **argv){
    struct http_bpf *skel;
    int err;
    libbpf_set_print(libbpf_print_fn);

    /* Open load and verify BPF application */
    skel = http_bpf__open_and_load();
    if (!skel) {
        fprintf(stderr, "Failed to open BPF skeleton\n");
        return 1;
    }

    /* Attach xdp handler */
    err = http_bpf__attach(skel);
    if (err) {
        fprintf(stderr, "Failed to attach BPF skeleton\n");
        goto cleanup;
    }

    while (1) {
        fprintf(stderr, ".");
        sleep(1);
    }

cleanup:
    http_bpf__destroy(skel);
    return -err;
}

I think that's error in SEC(xdp)

thanks

uprobe example

Would it be possible to add an example of how work with uprobes?

bootstrap c example not working

Hi there the c bootstrap example does not run. I'm running it on ubuntu 20.04

Error is

teroz@ubuntulsm:~/libbpf-bootstrap/examples/c$ sudo ./bootstrap 
[sudo] password for teroz: 
libbpf: map 'rb': failed to create: Invalid argument(-22)
libbpf: failed to load object 'bootstrap_bpf'
libbpf: failed to load BPF skeleton 'bootstrap_bpf': -22
Failed to load and verify BPF skeleton

How to generate `bpftool`?

I would like to know how bpftool and vmlinux.h is generated on this repository to it make cross compatible between different kernel versions that have /sys/kernel/btf/vmlinux exposed.

I can generate vmlinux.h for kernel 5.15 as follows,

$ uname -r
5.15.0-22-generic
$ git clone --depth 1 git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git -b v5.15 linux_v5.15
$ cd linux_v5.15/tools/bpf/bpftool/
$ make
$ ./bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

Then I can use vmlinux.h to compile libbpf programs.

However, if I download kernel source with tag v5.8 I am not able to generate vmlinux.h on the same machine,

$ git clone --depth 1 git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git -b v5.8 linux_v5.8
$ cd cd linux_v5.8/tools/bpf/bpftool
$ make
$ ./bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h
Error: failed to load BTF from /sys/kernel/btf/vmlinux: Invalid argument

Does it mean that bpftool must be of the same version (at least major and minor version matching) when generating vmlinux.h? So, this would explain why bpftool and vmlinux.h are included in the version control because, for example, if these files are generated on the fly in a container with mismatched host kernel and bpftool versions, the build can fail?

If yes, how do you go about picking which kernel version to use to compile libbpf program?

compilation of minimal failing because kernel header path not in includes

On Ubuntu 20.10 this happens to me:

$ V=1 make minimal
clang -g -O2 -target bpf -D__TARGET_ARCH_x86 -I.output -c minimal.bpf.c -o .output/minimal.bpf.o
In file included from minimal.bpf.c:3:
In file included from /usr/include/linux/bpf.h:11:
/usr/include/linux/types.h:5:10: fatal error: 'asm/types.h' file not found
#include <asm/types.h>

The rub seems to be that /usr/include/linux/types.h wants asm/types.h which, on my system, I believe lives in /usr/include/x86_64-linux-gnu/asm/types.h (which file just reads #include <asm-generic/types.h>), and /usr/include/x86_64-linux-gnu does not seem to be among clang's include paths, as demonstrated by the clang -v output below.

clang -v -g -O2 -target bpf -D__TARGET_ARCH_x86 -I.output -c minimal.bpf.c -o .output/minimal.bpf.o
Ubuntu clang version 11.0.0-2
...
#include "..." search starts here:
#include <...> search starts here:
 .output
 /usr/local/include`
 /usr/lib/llvm-11/lib/clang/11.0.0/include
 /usr/include
End of search list.

FWIW, on the gcc side, the preprocessor includes /usr/include/x86_64-linux-gnu in the cpp -v output.

I'm not sure who's at fault here, or what the correct fix is, but adding -I/usr/include/x86_64-linux-gnu makes it work. Changing the source to use vmlinux.h instead of <linux/bpf.h> also makes it work.

Although I don't believe it to be the case, it is possible that my system got screwed up by me manually installing kernel headers through the Debian package created by the kernel's deb-pkg make target. I don't think it's the case because I've verified that /usr/include/linux/types.h corresponds to the file provided by the Ubuntu's linux-libc-dev package.

Use xmake to port libbpf-bootstrap and support android bpf program

libbpf-bootstrap is a great project that allows beginners to quickly get started with bpf program development.

Now I used xmake to port the libbpf-bootstrap program and compiled it to the android platform.

xmake.lua

add_rules("mode.release", "mode.debug")
add_rules("platform.linux.bpf")

add_requires("linux-tools", {configs = {bpftool = true}})
add_requires("libbpf")
if is_plat("android") then
    add_requires("ndk >=22.x")
    set_toolchains("@ndk", {sdkver = "23"})
else
    add_requires("llvm >=10.x")
    set_toolchains("@llvm")
    add_requires("linux-headers")
end

target("minimal")
    set_kind("binary")
    add_files("src/*.c")
    add_packages("linux-tools", "linux-headers", "libbpf")
    set_license("GPL-2.0")

In addition, I ported the libbpf library and other dependent libraries to the android platform. All dependent packages and tool chains can be automatically downloaded and integrated by xmake.

We only need to execute xmake to compile linux bpf program, or use xmake f -p android; xmake to compile android bpf program.

If the user has installed the new version of llvm/clang and ndk r22, then xmake will also give priority to using them and will not download them automatically.

build on linux

It will download llvm toolchain and use it automatically if not found.

we can also install llvm manually. sudo apt install llvm and xmake will also use it.

$ xmake

build android on linux

It will download ndk toolchain and use it automatically

$ xmake f -p android
$ xmake

We can also set ndk toolchain manually (need ndk >= r22)

$ xmake f -p android --ndk=~/file/android-ndk-r22
$ xmake

Here is a libbpf-bootstrap project that I use xmake to ported. The project directory is very clean. We only need to keep the src source file and the xmake.lua project configuration file.

https://github.com/hack0z/libbpf-bootstrap

And I added libbpf package into our official repository, https://github.com/xmake-io/xmake-repo/blob/master/packages/l/libbpf/xmake.lua

If you are interested, I can open a pr and just add xmake.lua to your project, but I will not delete any other files, otherwise please ignore this issue.

Related issue: xmake-io/xmake#1274

Thanks!

git clone does not include libbpf

I was running git clone https://github.com/libbpf/libbpf-bootstrap.git but it looks like it missed libbpf for obvious reason. I guess documentation should mention that. That would be helpful.

Unable to run minimal example

Hello!

I have a docker container running Ubuntu 22.04 LTS, kernel version is 5.10.47.

I am able to build the minimal example from the cloned repo perfectly: make minimal.
But when I try to run the binary ./minimal, I get the following error:

libbpf: Failed to bump RLIMIT_MEMLOCK (err = -1), you might need to do it explicitly!
libbpf: Error in bpf_object__probe_loading():Operation not permitted(1). Couldn't load trivial BPF program. Make sure your kernel supports BPF (CONFIG_BPF_SYSCALL=y) and/or that RLIMIT_MEMLOCK is set to big enough value.
libbpf: failed to load object 'minimal_bpf'
libbpf: failed to load BPF skeleton 'minimal_bpf': -1
Failed to load and verify BPF skeleton

I can see the kernel is compiled with the flags correctly by checking the boot file:

cat /boot/config-5.15.0-30-generic:

CONFIG_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y

#
# BPF subsystem
#
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_JIT_DEFAULT_ON=y
CONFIG_BPF_UNPRIV_DEFAULT_OFF=y
CONFIG_USERMODE_DRIVER=y

I can also see that my RLIMIT_MEMLOCK is kinda big enough:

cat /etc/security/limits.conf

#*               soft    core               1000000
#root            hard    core            1000000
#*               hard    rss             1000000

# End of file

Am I missing something here?

Appreciate any help!

Xmake + Android - /examples/c build error

• host: Ubuntu 20.04.4 LTS / Kernel version 5.18-051800-generic
• guest: Android 11 x64 (with arm64 emulation by default) - vanilla stock Google Pixel 4 emulator

$ xmake f -p android && xmake
[ 34%]: ccache compiling.release ../../libbpf/src/libbpf.c
[ 43%]: ccache compiling.release ../../libbpf/src/usdt.c
[ 43%]: ccache compiling.release ../../libbpf/src/hashmap.c
[ 46%]: ccache compiling.release ../../libbpf/src/strset.c
error: ../../libbpf/src/usdt.c:562:7: error: unknown type name 'GElf_Nhdr'
      GElf_Nhdr *nhdr, const char *data, size_t name_off, size_t desc_off,
      ^
../../libbpf/src/usdt.c:578:11: error: expected ';' after expression
 GElf_Nhdr nhdr;
          ^
          ;
../../libbpf/src/usdt.c:578:2: error: use of undeclared identifier 'GElf_Nhdr'
 GElf_Nhdr nhdr;
 ^
../../libbpf/src/usdt.c:578:12: error: use of undeclared identifier 'nhdr'; did you mean 'ehdr'?
 GElf_Nhdr nhdr;
           ^~~~
           ehdr
../../libbpf/src/usdt.c:577:12: note: 'ehdr' declared here
 GElf_Ehdr ehdr;
           ^
  > in ../../libbpf/src/usdt.c

warning: "Attempt to use kernel headers from user space

I imitated Minimal Makefile and wrote an out-of-kernel tree BPF program which is not CO-RE.
Hence, I include some kernel header which comes form kernel-devel in BPF program. Then I got some error:

/usr/src/kernels/4.19.90-2106.3.0.0095.oe1.x86_64/include/uapi/linux/types.h:10:2: warning: "Attempt to use kernel headers from user space, see http://kernelnewbies.org/KernelHeaders" [-W#warnings]
#warning "Attempt to use kernel headers from user space, see http://kernelnewbies.org/KernelHeaders"
 ^
readahead_tune.bpf.c:64:24: error: unknown type name 'bool'
static __always_inline bool is_expected_file(void *name)
                       ^
readahead_tune.bpf.c:71:16: error: use of undeclared identifier 'false'; did you mean 'else'?
        return false;
               ^~~~~
               else
readahead_tune.bpf.c:71:16: error: expected expression
readahead_tune.bpf.c:85:30: error: use of undeclared identifier 'FMODE_WILLNEED'
        rd_ctx->set_f_mode = FMODE_WILLNEED;
                             ^
readahead_tune.bpf.c:91:5: error: use of undeclared identifier 'bool'
    bool first = false;
    ^
readahead_tune.bpf.c:97:9: error: use of undeclared identifier 'first'
        first = true;
...

And the Makefile:

OUTPUT := .output
CLANG ?= clang -v
BPFTOOL ?= bpftool
LINUX_HEADER ?= /usr/src/kernels/`uname -r` 
LINUX_INCLUDE = -I $(LINUX_HEADER)/include/uapi \
                -I $(LINUX_HEADER)/include
...
# Build BPF code
$(OUTPUT)/%.bpf.o: %.bpf.c $(wildcard %.h) | $(OUTPUT)
    $(call msg,BPF,$@)
    $(Q)$(CLANG) -g -O2 -target bpf \
    -D__TARGET_ARCH_$(ARCH) \
    $(LINUX_INCLUDE) $(CLANG_BPF_SYS_INCLUDES) -c $(filter %.c,$^) -o $@

The BPF program:

#include <linux/stddef.h> // for true/false
#include <linux/types.h> // for bool
#include <linux/fs.h> // for FMODE_WILLNEED
...

So what do we need something else to make out-of-kernel tree Non-CO-RE BPF program build successfully?

vmlinux.h question

In the vmlinux.h on line 950 appears:
struct hlist_node pid_links[4];
shouldn't it be
struct pid_link pid_links[4];
instead?

XDP rust example

I compiled the program but when I try to attach it to an interface with sudo ./target/release/xdp 1 I get this message
libbpf: elf: skipping unrecognized data section(4) .rodata.str1.1
...........
I know this is a warning message but how to fix it?

bpf_printk doesn't create data on trace_pipe

My minimal app executes correctly:

But when I cat the "/sys/kernel/debug/tracing/trace_pipe" file it looks empty:

I'm running a Ubuntu 22.04 LTS docker container on privileged mode.

This is how I have manually mounted my debugfs:
mount -t debugfs none /sys/kernel/debug

Am I missing anything? Appreciate your help.

Is it possible to run minimal with BTF?

libbpf: loading object 'minimal_bpf' from buffer
libbpf: elf: section(2) tp/syscalls/sys_enter_write, size 24, link 0, flags 6, type=1
libbpf: sec 'tp/syscalls/sys_enter_write': found program 'handle_tp' at insn offset 0 (0 bytes), code size 3 insns (24 bytes)
libbpf: elf: section(3) license, size 13, link 0, flags 3, type=1
libbpf: license of minimal_bpf is Dual BSD/GPL
libbpf: elf: section(4) .bss, size 4, link 0, flags 3, type=8
libbpf: elf: section(5) .BTF, size 446, link 0, flags 0, type=1
libbpf: elf: section(6) .BTF.ext, size 96, link 0, flags 0, type=1
libbpf: elf: section(7) .symtab, size 144, link 11, flags 0, type=2
libbpf: looking for externs among 6 symbols...
libbpf: collected 0 externs total
libbpf: map 'minimal_.bss' (global data): at sec_idx 4, offset 0, flags 400.
libbpf: map 0 is "minimal_.bss"
libbpf: Kernel doesn't support BTF, skipping uploading it.
libbpf: kernel doesn't support global data
libbpf: failed to load object 'minimal_bpf'
libbpf: failed to load BPF skeleton 'minimal_bpf': -95
Failed to load and verify BPF skeleton

• Kernel Version: Linux VM-0-13-ubuntu 4.15.0-118-generic #119-Ubuntu SMP Tue Sep 8 12:30:01 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
• OS Version:

Distributor ID: Ubuntu
Description:    Ubuntu 18.04.5 LTS
Release:        18.04
Codename:       bionic

make: *** [.output/minimal.skel.h] Error 139

Hi,

When running make minimal from examples/c folder, I am getting the following error :

GEN-SKEL .output/minimal.skel.h
Segmentation fault (core dumped)
Makefile:110: recipe for target '.output/minimal.skel.h' failed
make: *** [.output/minimal.skel.h] Error 139
make: *** Deleting file '.output/minimal.skel.h'

The steps I have followed for installation are :

git clone https://github.com/libbpf/libbpf-bootstrap.git
git submodule update --init --recursive
cd examples/c
make minimal

I have also googled for Error 139 and have also tried the solution in the closed issues #69 #70 #72 #15 but couldn't find a solution that works for me.
I and gone through the Makefile line 109 - 110. I tried removing | $(OUTPUT) $(BPFTOOL) from line 109 $(OUTPUT)/%.skel.h: $(OUTPUT)/%.bpf.o | $(OUTPUT) $(BPFTOOL) but that too didn't work.

The current version of clang and kernel installed on Ubuntu 18.04:

I am not sure what's missing.

kprobe not appearing in output, only kretprobe (in prospective kprobe_netlink example)

I created a fork and new feature branch at https://github.com/chrispsommers/libbpf-bootstrap/tree/kprobe_netlink-example#kprobe_netlink. I added example kprobe_netlink to trace netlink messages. The tracing output only shows the kretprobe messages for some reason. Any advice? You can clone this and build using the normal recipes if you care to try it. See below, thanks:

sudo ./kprobe_netlink 
libbpf: loading object 'kprobe_netlink_bpf' from buffer
libbpf: elf: section(2) kprobe/netlink_unicast, size 240, link 0, flags 6, type=1
libbpf: sec 'kprobe/netlink_unicast': found program 'netlink_unicast' at insn offset 0 (0 bytes), code size 30 insns (240 bytes)
libbpf: elf: section(3) kretprobe/netlink_unicast, size 280, link 0, flags 6, type=1
libbpf: sec 'kretprobe/netlink_unicast': found program 'netlink_unicast_exit' at insn offset 0 (0 bytes), code size 35 insns (280 bytes)
libbpf: elf: section(4) kprobe/netlink_broadcast, size 232, link 0, flags 6, type=1
libbpf: sec 'kprobe/netlink_broadcast': found program 'netlink_broadcast' at insn offset 0 (0 bytes), code size 29 insns (232 bytes)
libbpf: elf: section(5) kretprobe/netlink_broadcast, size 248, link 0, flags 6, type=1
libbpf: sec 'kretprobe/netlink_broadcast': found program 'netlink_broadcast_exit' at insn offset 0 (0 bytes), code size 31 insns (248 bytes)
libbpf: elf: section(6) license, size 13, link 0, flags 3, type=1
libbpf: license of kprobe_netlink_bpf is Dual BSD/GPL
libbpf: elf: section(7) .rodata.str1.1, size 201, link 0, flags 32, type=1
libbpf: elf: skipping unrecognized data section(7) .rodata.str1.1
libbpf: elf: section(8) .BTF, size 1641, link 0, flags 0, type=1
libbpf: elf: section(9) .BTF.ext, size 684, link 0, flags 0, type=1
libbpf: elf: section(10) .symtab, size 240, link 14, flags 0, type=2
libbpf: looking for externs among 10 symbols...
libbpf: collected 0 externs total
libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': 0
libbpf: sec 'kprobe/netlink_unicast': found 1 CO-RE relocations
libbpf: prog 'netlink_unicast': relo #0: kind <byte_off> (0), spec is [2] struct pt_regs.dx (0:12 @ offset 96)
libbpf: CO-RE relocating [0] struct pt_regs: found target candidate [221] struct pt_regs in [vmlinux]
libbpf: prog 'netlink_unicast': relo #0: matching candidate #0 [221] struct pt_regs.dx (0:12 @ offset 96)
libbpf: prog 'netlink_unicast': relo #0: patched insn #0 (LDX/ST/STX) off 96 -> 96
libbpf: sec 'kretprobe/netlink_unicast': found 1 CO-RE relocations
libbpf: prog 'netlink_unicast_exit': relo #0: kind <byte_off> (0), spec is [2] struct pt_regs.ax (0:10 @ offset 80)
libbpf: prog 'netlink_unicast_exit': relo #0: matching candidate #0 [221] struct pt_regs.ax (0:10 @ offset 80)
libbpf: prog 'netlink_unicast_exit': relo #0: patched insn #0 (LDX/ST/STX) off 80 -> 80
libbpf: sec 'kprobe/netlink_broadcast': found 1 CO-RE relocations
libbpf: prog 'netlink_broadcast': relo #0: kind <byte_off> (0), spec is [2] struct pt_regs.dx (0:12 @ offset 96)
libbpf: prog 'netlink_broadcast': relo #0: matching candidate #0 [221] struct pt_regs.dx (0:12 @ offset 96)
libbpf: prog 'netlink_broadcast': relo #0: patched insn #0 (LDX/ST/STX) off 96 -> 96
libbpf: sec 'kretprobe/netlink_broadcast': found 1 CO-RE relocations
libbpf: prog 'netlink_broadcast_exit': relo #0: kind <byte_off> (0), spec is [2] struct pt_regs.ax (0:10 @ offset 80)
libbpf: prog 'netlink_broadcast_exit': relo #0: matching candidate #0 [221] struct pt_regs.ax (0:10 @ offset 80)
libbpf: prog 'netlink_broadcast_exit': relo #0: patched insn #0 (LDX/ST/STX) off 80 -> 80
Successfully started! Please run `sudo cat /sys/kernel/debug/tracing/trace_pipe` to see output of the BPF programs.
........

Trace output:
$ sudo cat /sys/kernel/debug/tracing/trace_pipe
sudo-31063 [010] d... 90928.229245: bpf_trace_printk: KPROBE netlink_unicast_exit EXIT: pid = 31063, ret = 36

        sudo-31063   [010] d... 90928.229252: bpf_trace_printk: KPROBE netlink_unicast_exit EXIT: pid = 31063, ret = 176

        sudo-31063   [010] d... 90928.229352: bpf_trace_printk: KPROBE netlink_unicast_exit EXIT: pid = 31063, ret = 36

        sudo-31063   [010] d... 90928.229354: bpf_trace_printk: KPROBE netlink_unicast_exit EXIT: pid = 31063, ret = 136

XDP example in C

Would it be possible to provide a C version of the XDP example? I'm struggling to figure out the syntax for attaching a program to a particular network interface.

I can compile and run the example posted in issue #35 without errors after including unistd.h, but the kernel code never triggers no matter which interface I send traffic to. I would expect to be able to replace http_bpf__attach(skel) with something like http_bpf__attach_xdp(skel, ifindex), but no such function gets generated in the skel.h file and the standard attach function only accepts a single argument. Or is the interface supposed to be set as an attribute of the skeleton before attaching somehow?

Thanks in advance for any help.

Fentry failing to load on arm64

I am successfully building bootstrap, minimal, uprobe examples and they all work on M1 Macbook running an arm64 ubuntu VM but Fentry fails for some odd reason I can't figure out. Any tips on how to debug such issues:

$ uname -a
Linux devvm 5.13.0-21-generic #21-Ubuntu SMP Tue Oct 19 09:01:50 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

$ sudo ./fentry
[sudo] password for sasan:
libbpf: loading object 'fentry_bpf' from buffer
libbpf: elf: section(2) fentry/do_unlinkat, size 200, link 0, flags 6, type=1
libbpf: sec 'fentry/do_unlinkat': found program 'do_unlinkat' at insn offset 0 (0 bytes), code size 25 insns (200 bytes)
libbpf: elf: section(3) fexit/do_unlinkat, size 256, link 0, flags 6, type=1
libbpf: sec 'fexit/do_unlinkat': found program 'do_unlinkat_exit' at insn offset 0 (0 bytes), code size 32 insns (256 bytes)
libbpf: elf: section(4) license, size 13, link 0, flags 3, type=1
libbpf: license of fentry_bpf is Dual BSD/GPL
libbpf: elf: section(5) .rodata.str1.1, size 76, link 0, flags 32, type=1
libbpf: elf: skipping unrecognized data section(5) .rodata.str1.1
libbpf: elf: section(6) .BTF, size 945, link 0, flags 0, type=1
libbpf: elf: section(7) .BTF.ext, size 364, link 0, flags 0, type=1
libbpf: elf: section(8) .symtab, size 144, link 12, flags 0, type=2
libbpf: looking for externs among 6 symbols...
libbpf: collected 0 externs total
libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': 0
libbpf: sec 'fentry/do_unlinkat': found 1 CO-RE relocations
libbpf: prog 'do_unlinkat': relo #0: kind <byte_off> (0), spec is [6] struct filename.name (0:0 @ offset 0)
libbpf: CO-RE relocating [0] struct filename: found target candidate [1115] struct filename in [vmlinux]
libbpf: prog 'do_unlinkat': relo #0: matching candidate #0 [1115] struct filename.name (0:0 @ offset 0)
libbpf: prog 'do_unlinkat': relo #0: patched insn #16 (LDX/ST/STX) off 0 -> 0
libbpf: sec 'fexit/do_unlinkat': found 1 CO-RE relocations
libbpf: prog 'do_unlinkat_exit': relo #0: kind <byte_off> (0), spec is [6] struct filename.name (0:0 @ offset 0)
libbpf: prog 'do_unlinkat_exit': relo #0: matching candidate #0 [1115] struct filename.name (0:0 @ offset 0)
libbpf: prog 'do_unlinkat_exit': relo #0: patched insn #22 (LDX/ST/STX) off 0 -> 0
libbpf: prog 'do_unlinkat': failed to attach: ERROR: strerror_r(-524)=22
libbpf: failed to auto-attach program 'do_unlinkat': -524
Failed to attach BPF skeleton

Unable to run kprobe example

I commented on #68 about how I could not get this to run on ubuntu 18.04. I was able to get minimal working, by removing the global variable. I am using kernel version 4.15.0-180-generic. The kprobe example, however, is not running for some other reason:

sudo ./kprobe
libbpf: loading object 'kprobe_bpf' from buffer
libbpf: elf: section(2) kprobe/do_unlinkat, size 152, link 0, flags 6, type=1
libbpf: sec 'kprobe/do_unlinkat': found program 'do_unlinkat' at insn offset 0 (0 bytes), code size 19 insns (152 bytes)
libbpf: elf: section(3) kretprobe/do_unlinkat, size 88, link 0, flags 6, type=1
libbpf: sec 'kretprobe/do_unlinkat': found program 'do_unlinkat_exit' at insn offset 0 (0 bytes), code size 11 insns (88 bytes)
libbpf: elf: section(4) license, size 13, link 0, flags 3, type=1
libbpf: license of kprobe_bpf is Dual BSD/GPL
libbpf: elf: section(5) .rodata, size 72, link 0, flags 2, type=1
libbpf: elf: section(6) .BTF, size 1482, link 0, flags 0, type=1
libbpf: elf: section(7) .BTF.ext, size 364, link 0, flags 0, type=1
libbpf: elf: section(8) .symtab, size 240, link 14, flags 0, type=2
libbpf: elf: section(9) .relkprobe/do_unlinkat, size 16, link 8, flags 0, type=9
libbpf: elf: section(10) .relkretprobe/do_unlinkat, size 16, link 8, flags 0, type=9
libbpf: looking for externs among 10 symbols...
libbpf: collected 0 externs total
libbpf: map 'kprobe_b.rodata' (global data): at sec_idx 5, offset 0, flags 480.
libbpf: map 0 is "kprobe_b.rodata"
libbpf: sec '.relkprobe/do_unlinkat': collecting relocation for section(2) 'kprobe/do_unlinkat'
libbpf: sec '.relkprobe/do_unlinkat': relo #0: insn #12 against '.rodata'
libbpf: prog 'do_unlinkat': found data map 0 (kprobe_b.rodata, sec 5, off 0) for insn 12
libbpf: sec '.relkretprobe/do_unlinkat': collecting relocation for section(3) 'kretprobe/do_unlinkat'
libbpf: sec '.relkretprobe/do_unlinkat': relo #0: insn #3 against '.rodata'
libbpf: prog 'do_unlinkat_exit': found data map 0 (kprobe_b.rodata, sec 5, off 0) for insn 3
libbpf: Kernel doesn't support BTF, skipping uploading it.
libbpf: map 'kprobe_b.rodata': skipped auto-creating...
libbpf: sec 'kprobe/do_unlinkat': found 2 CO-RE relocations
libbpf: CO-RE relocating [2] struct pt_regs: found target candidate [14] struct pt_regs in [vmlinux]
libbpf: prog 'do_unlinkat': relo #0: <byte_off> [2] struct pt_regs.si (0:13 @ offset 104)
libbpf: prog 'do_unlinkat': relo #0: non-matching candidate #0 <byte_off> [14] struct pt_regs (0 @ offset 0)
libbpf: prog 'do_unlinkat': relo #0: no matching targets found
libbpf: prog 'do_unlinkat': relo #0: substituting insn #0 w/ invalid insn
libbpf: prog 'do_unlinkat': relo #1: <byte_off> [7] struct filename.name (0:0 @ offset 0)
libbpf: prog 'do_unlinkat': relo #1: no matching targets found
libbpf: prog 'do_unlinkat': relo #1: substituting insn #3 w/ invalid insn
libbpf: sec 'kretprobe/do_unlinkat': found 1 CO-RE relocations
libbpf: prog 'do_unlinkat_exit': relo #0: <byte_off> [2] struct pt_regs.ax (0:10 @ offset 80)
libbpf: prog 'do_unlinkat_exit': relo #0: matching candidate #0 <byte_off> [14] struct pt_regs.ax (0:0 @ offset 80)
libbpf: prog 'do_unlinkat_exit': relo #0: patched insn #0 (LDX/ST/STX) off 80 -> 80
libbpf: prog 'do_unlinkat': relo #2: poisoning insn #12 that loads map #0 'kprobe_b.rodata'
libbpf: prog 'do_unlinkat_exit': relo #1: poisoning insn #3 that loads map #0 'kprobe_b.rodata'
libbpf: prog 'do_unlinkat': BPF program load failed: Invalid argument
libbpf: prog 'do_unlinkat': -- BEGIN PROG LOAD LOG --
0: <invalid CO-RE relocation>
failed to resolve CO-RE relocation <byte_off> [2] struct pt_regs.si (0:13 @ offset 104)
-- END PROG LOAD LOG --
libbpf: failed to load program 'do_unlinkat'
libbpf: failed to load object 'kprobe_bpf'
libbpf: failed to load BPF skeleton 'kprobe_bpf': -22
Failed to load and verify BPF skeleton

add new example

I wrote an Example for helloworld,But pr cannot submit

Process the bpftool file of x86-64 on the ARM architecture when compiling the BPF program

Hello:
My system uses the ARM architecture, but the bpftool file is x86-64. As a result, the following error message is displayed when the BPF program is compiled:
/bin/sh: /root/tools/libbpf-bootstrap/tools/bpftool: cannot execute binary file: Exec format error
What am I supposed to do with it?

Support multiple languages / platforms

Creating this issue to decide directory structure and code layout.

Current structure:

.
├── libbpf
├── LICENSE
├── README.md
├── src
│   ├── bootstrap.bpf.c
│   ├── bootstrap.c
│   ├── bootstrap.h
│   ├── fentry.bpf.c
│   ├── fentry.c
│   ├── kprobe.bpf.c
│   ├── kprobe.c
│   ├── Makefile
│   ├── minimal.bpf.c
│   ├── minimal.c
│   ├── uprobe.bpf.c
│   ├── uprobe.c
│   ├── vmlinux_508.h
│   └── vmlinux.h -> vmlinux_508.h
└── tools
    ├── bpftool
    └── gen_vmlinux_h.sh

Proposed structure:

.
├── examples
│   ├── c
│   │   ├── bootstrap
│   │   │   ├── bootstrap.bpf.c
│   │   │   ├── bootstrap.c
│   │   │   ├── bootstrap.h
│   │   │   └── Makefile
│   │   ├── fentry
│   │   │   ├── fentry.bpf.c
│   │   │   ├── fentry.c
│   │   │   └── Makefile
│   │   ├── kprobe
│   │   │   ├── kprobe.bpf.c
│   │   │   ├── kprobe.c
│   │   │   └── Makefile
│   │   ├── Makefile
│   │   ├── minimal
│   │   │   ├── Makefile
│   │   │   ├── minimal.bpf.c
│   │   │   └── minimal.c
│   │   └── uprobe
│   │       ├── Makefile
│   │       ├── uprobe.bpf.c
│   │       └── uprobe.c
│   └── rust
│       ├── Cargo.toml
│       ├── Makefile
│       └── xdp
│           ├── Cargo.toml
│           ├── Makefile
│           └── src
│               ├── bpf
│               │   └── xdp.bpf.c
│               └── main.rs
├── libbpf
├── LICENSE
├── README.md
├── tools
│   ├── bpftool
│   └── gen_vmlinux_h.sh
└── vmlinux
    ├── Makefile
    ├── vmlinux_508.h
    └── vmlinux.h -> vmlinux_508.h

Basic idea is top level entries (tools/, vmlinux/, etc.) are shared between all examples. Anything language / platform specific must be completely contained to the examples/* directory. Each language/platform has a makefile that does the right thing.

Not sure if each individual example should have its own makefile. Maybe it'd be nice to be able to cd into any example and just type make.

LSM example

hi,
Recently, I was trying to implement LSM using BPF. but my program has no output.

my ebpf.c

#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include  <errno.h>

char _license[] SEC("license") = "GPL";

SEC("lsm/file_open")
int BPF_PROG(file_open_handler, struct file *file, int ret)
{
	bpf_printk("Hello, world\n");
	return ret;
}

loader.c

#include <stdio.h>
#include <linux/bpf.h>
#include <bpf/libbpf.h>
#include <bpf/bpf.h>
#include <assert.h>
#include <unistd.h>
#include "lsm_kern.h"

static int libbpf_print_fn(enum libbpf_print_level level, const char *format, va_list args)
{
        return vfprintf(stderr, format, args);
}

int main(int argc,char *argv)
{
	libbpf_set_print(libbpf_print_fn);
	struct lsm_kern *skel = NULL;
	int err = 0;

	skel = lsm_kern__open_and_load();
	if(!skel){
		printf("load error!\n");
	}
	err = lsm_kern__attach(skel);
	if(err)
		printf("attach error!\n");
	while(1){
		sleep(100);
	}
	return 0;
}

Makefile(make output)

clang -g -O2 -target bpf -D__TARGET_ARCH_x86 -I../libbpf/include/uapi -idirafter /usr/lib/llvm-14/lib/clang/14.0.0/include -idirafter /usr/local/include -idirafter /usr/include/x86_64-linux-gnu -idirafter /usr/include  -c lsm_kern.c -o lsm_kern.o
bpftool gen skeleton lsm_kern.o > lsm_kern.h
gcc -o loader loader.c /usr/lib64/libbpf.a /usr/lib/x86_64-linux-gnu/libz.a /usr/lib/x86_64-linux-gnu/libelf.a

load and attach out

libbpf: loading object 'lsm_kern' from buffer
libbpf: elf: section(3) lsm/file_open, size 136, link 0, flags 6, type=1
libbpf: sec 'lsm/file_open': found program 'file_open_handler' at insn offset 0 (0 bytes), code size 17 insns (136 bytes)
libbpf: elf: section(4) .rellsm/file_open, size 32, link 26, flags 40, type=9
libbpf: elf: section(5) license, size 4, link 0, flags 3, type=1
libbpf: license of lsm_kern is GPL
libbpf: elf: section(6) .rodata, size 48, link 0, flags 2, type=1
libbpf: elf: section(16) .BTF, size 727, link 0, flags 0, type=1
libbpf: elf: section(18) .BTF.ext, size 144, link 0, flags 0, type=1
libbpf: elf: section(26) .symtab, size 432, link 1, flags 0, type=2
libbpf: looking for externs among 18 symbols...
libbpf: collected 0 externs total
libbpf: map 'lsm_kern.rodata' (global data): at sec_idx 6, offset 0, flags 480.
libbpf: map 0 is "lsm_kern.rodata"
libbpf: sec '.rellsm/file_open': collecting relocation for section(3) 'lsm/file_open'
libbpf: sec '.rellsm/file_open': relo #0: insn #5 against '.rodata'
libbpf: prog 'file_open_handler': found data map 0 (lsm_kern.rodata, sec 6, off 0) for insn 5
libbpf: sec '.rellsm/file_open': relo #1: insn #11 against '.rodata'
libbpf: prog 'file_open_handler': found data map 0 (lsm_kern.rodata, sec 6, off 0) for insn 11
libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': 0
libbpf: map 'lsm_kern.rodata': created successfully, fd=4

but cat /sys/kernel/debug/tracing/trace_pipe not output

strace -o log ./loader

bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\08\0\0\08\0\0\0\t\0\0\0\0\0\0\0\0\0\0\1"..., btf_log_buf=NULL, btf_size=89, btf_log_size=0, btf_log_level=0}, 28) = 3
...
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_LSM, insn_cnt=17, insns=0x56299e5aa620, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 15, 30), prog_flags=0, prog_name="file_open_handl", prog_ifindex=0, expected_attach_type=BPF_LSM_MAC, prog_btf_fd=3, func_info_rec_size=8, func_info=0x56299e5a9000, func_info_cnt=1, line_info_rec_size=16, line_info=0x56299e5a8b70, line_info_cnt=5, attach_btf_id=32663, attach_prog_fd=0, fd_array=NULL}, 144) = 5
...
bpf(BPF_RAW_TRACEPOINT_OPEN, {raw_tracepoint={name=NULL, prog_fd=5}}, 144) = 6

what is the function int BPF_PROG(do_unlinkat, int dfd, struct filename *name) arguments means ?

in function: int BPF_PROG(do_unlinkat, int dfd, struct filename *name)
I don't know why the function's first argument is "do_unlinkat", I find nothing from all included header files. And i change the "do_unlinkat" to random string, such as "xxxyyyy". And everything is ok. so what is the first argument means?

Add install dependencies section to README

For Ubuntu 20.04 I had to install libelf-dev and clang to build examples.
For Ubuntu 16.04 I had to install libz in addition to above.