libbpf-rs XDP example attach error about libbpf-bootstrap

[anakryiko]

@chenhengqi I was able to attach the app without errors by calling:

libbpf_sys::bpf_set_link_xdp_fd(ifindex, prog_fd, libbpf_sys::XDP_FLAGS_SKB_MODE)

This also worked, so I'm assuming it does generic mode by default:

If you don't specify any flag, it will pick driver mode, if driver supports it. Otherwise it falls back to generic mode.

A couple of thoughts given this experiment:

* `bpf_set_link_xdp_fd` uses `netlink.c`, but your implementation uses `bpf_program__attach_xdp` which ends up calling `bpf_link_create` in `bpf.c`. I don't believe the netlink.c API I used here ends up calling `bpf_link_create`. Does this mean that libbpf gives you two ways to load an XDP app?

Yes. On newer kernels bpf_link-based XDP attachment is supported. bpf_link-based attachment will auto-detach BPF program automatically when user-space program crashes or exits, unless you pin it in BPF FS. You can still use netlink-based API, if you prefer, but you won't get benefits of bpf_link infrastructure.

[chenhengqi]

@chenhengqi would you be able to give me some guidance on how to do that? I’m new to libbpf.

I looked into doing this, but it seems like the libbpf-rs Program API doesn’t expose a libbpf function that takes the SKB_MODE flag as an argument.

Or did you mean compiling and loading the BPF example without the libbpf-rs loader?

You need bpf_set_link_xdp_fd from libbpf-sys right after let mut skel = open_skel.load()?; .

[chenhengqi]

You should provide an ifindex as the first argument.

[jfernandez]

@chenhengqi I'm getting the same error when I provide the ifindex (1 and 2):

vagrant@vagrant:/vagrant$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:48:7f:14 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
vagrant@vagrant:/vagrant$ sudo ./target/debug/xdppass 2
libbpf: loading object 'xdppass_bpf' from buffer
libbpf: elf: section(3) xdp, size 120, link 0, flags 6, type=1
libbpf: sec 'xdp': found program 'xdp_pass' at insn offset 0 (0 bytes), code size 15 insns (120 bytes)
libbpf: elf: section(4) .rodata.str1.1, size 16, link 0, flags 32, type=1
libbpf: elf: skipping unrecognized data section(4) .rodata.str1.1
libbpf: elf: section(5) license, size 4, link 0, flags 3, type=1
libbpf: license of xdppass_bpf is GPL
libbpf: elf: section(12) .BTF, size 669, link 0, flags 0, type=1
libbpf: elf: section(14) .BTF.ext, size 220, link 0, flags 0, type=1
libbpf: elf: section(21) .symtab, size 912, link 1, flags 0, type=2
libbpf: looking for externs among 38 symbols...
libbpf: collected 0 externs total
libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': 0
libbpf: sec 'xdp': found 2 CO-RE relocations
libbpf: prog 'xdp_pass': relo #0: kind <byte_off> (0), spec is [2] struct xdp_md.data_end (0:1 @ offset 4)
libbpf: CO-RE relocating [0] struct xdp_md: found target candidate [20548] struct xdp_md in [vmlinux]
libbpf: prog 'xdp_pass': relo #0: matching candidate #0 [20548] struct xdp_md.data_end (0:1 @ offset 4)
libbpf: prog 'xdp_pass': relo #0: patched insn #0 (LDX/ST/STX) off 4 -> 4
libbpf: prog 'xdp_pass': relo #1: kind <byte_off> (0), spec is [2] struct xdp_md.data (0:0 @ offset 0)
libbpf: prog 'xdp_pass': relo #1: matching candidate #0 [20548] struct xdp_md.data (0:0 @ offset 0)
libbpf: prog 'xdp_pass': relo #1: patched insn #1 (LDX/ST/STX) off 0 -> 0
libbpf: prog 'xdp_pass': failed to attach to xdp: Invalid argument
thread 'main' panicked at 'Problem attaching the app: System(-22)', src/main.rs:44:23
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[chenhengqi]

@jfernandez

Could you please update this program to use SKB_MODE locally and see if it works with your VM setup ?

[jfernandez]

@chenhengqi would you be able to give me some guidance on how to do that? I’m new to libbpf.

I looked into doing this, but it seems like the libbpf-rs Program API doesn’t expose a libbpf function that takes the SKB_MODE flag as an argument.

Or did you mean compiling and loading the BPF example without the libbpf-rs loader?

[jfernandez]

I used the libbpf-cargo command to build an ELF file, and tried loading it with iproute2. I also get this invalid argument (-22) error, but app is loaded.

vagrant@vagrant:/vagrant/target/bpf$ cargo libbpf build
vagrant@vagrant:/vagrant/target/bpf$ sudo ip link set dev eth0 xdp obj xdppass.bpf.o sec xdp

BTF debug data section '.BTF' rejected: Invalid argument (22)!
 - Length:       667
Verifier analysis:

magic: 0xeb9f
version: 1
flags: 0x0
hdr_len: 24
type_off: 0
type_len: 268
str_off: 268
str_len: 375
btf_total_size: 667
[1] PTR (anon) type_id=2
[2] STRUCT xdp_md size=24 vlen=6
        data type_id=3 bits_offset=0
        data_end type_id=3 bits_offset=32
        data_meta type_id=3 bits_offset=64
        ingress_ifindex type_id=3 bits_offset=96
        rx_queue_index type_id=3 bits_offset=128
        egress_ifindex type_id=3 bits_offset=160
[3] TYPEDEF __u32 type_id=4
[4] INT unsigned int size=4 bits_offset=0 nr_bits=32 encoding=(none)
[5] FUNC_PROTO (anon) return=6 args=(1 ctx)
[6] INT int size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
[7] FUNC xdp_pass type_id=5
[8] INT char size=1 bits_offset=0 nr_bits=8 encoding=SIGNED
[9] ARRAY (anon) type_id=8 index_type_id=10 nr_elems=4
[10] INT __ARRAY_SIZE_TYPE__ size=4 bits_offset=0 nr_bits=32 encoding=(none)
[11] VAR __license type_id=9 linkage=1
[12] DATASEC license size=0 vlen=1 size == 0

vagrant@vagrant:/vagrant/target/bpf$ ip link list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdpgeneric qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:48:7f:14 brd ff:ff:ff:ff:ff:ff
    prog/xdp id 220 
    altname enp0s3

Google tells me that LLVM 10 emits incompatible BTF: cilium/ebpf#43
When I ran the suggested solution llvm-strip ./xxx.bpf.o --no-strip-all -R .BTF, I was able to load it using iproute2 without any warnings/errors.

libbpr-rs still has issues loading that "stripped" ELF file.

[chenhengqi]

I used the libbpf-cargo command to build an ELF file, and tried loading it with iproute2. I also get this invalid argument (-22) error, but app is loaded.

vagrant@vagrant:/vagrant/target/bpf$ cargo libbpf build
vagrant@vagrant:/vagrant/target/bpf$ sudo ip link set dev eth0 xdp obj xdppass.bpf.o sec xdp

BTF debug data section '.BTF' rejected: Invalid argument (22)!
 - Length:       667
Verifier analysis:

magic: 0xeb9f
version: 1
flags: 0x0
hdr_len: 24
type_off: 0
type_len: 268
str_off: 268
str_len: 375
btf_total_size: 667
[1] PTR (anon) type_id=2
[2] STRUCT xdp_md size=24 vlen=6
        data type_id=3 bits_offset=0
        data_end type_id=3 bits_offset=32
        data_meta type_id=3 bits_offset=64
        ingress_ifindex type_id=3 bits_offset=96
        rx_queue_index type_id=3 bits_offset=128
        egress_ifindex type_id=3 bits_offset=160
[3] TYPEDEF __u32 type_id=4
[4] INT unsigned int size=4 bits_offset=0 nr_bits=32 encoding=(none)
[5] FUNC_PROTO (anon) return=6 args=(1 ctx)
[6] INT int size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
[7] FUNC xdp_pass type_id=5
[8] INT char size=1 bits_offset=0 nr_bits=8 encoding=SIGNED
[9] ARRAY (anon) type_id=8 index_type_id=10 nr_elems=4
[10] INT __ARRAY_SIZE_TYPE__ size=4 bits_offset=0 nr_bits=32 encoding=(none)
[11] VAR __license type_id=9 linkage=1
[12] DATASEC license size=0 vlen=1 size == 0

vagrant@vagrant:/vagrant/target/bpf$ ip link list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdpgeneric qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:48:7f:14 brd ff:ff:ff:ff:ff:ff
    prog/xdp id 220 
    altname enp0s3

Google tells me that LLVM 10 emits incompatible BTF: cilium/ebpf#43
When I ran the suggested solution llvm-strip ./xxx.bpf.o --no-strip-all -R .BTF, I was able to load it using iproute2 without any warnings/errors.

libbpr-rs still has issues loading that "stripped" ELF file.

iproute2 uses its own BPF loader, I think it's not compatible with libbpf loader.

[jfernandez]

@chenhengqi I was able to attach the app without errors by calling:

libbpf_sys::bpf_set_link_xdp_fd(ifindex, prog_fd, libbpf_sys::XDP_FLAGS_SKB_MODE)

This also worked, so I'm assuming it does generic mode by default:

libbpf_sys::bpf_set_link_xdp_fd(ifindex, prog_fd, 0)

Explicitly telling it to use the driver mode failed with this message:

libbpf_sys::bpf_set_link_xdp_fd(ifindex, prog_fd, libbpf_sys::XDP_FLAGS_DRV_MODE)

vagrant@vagrant:/vagrant$ sudo ./target/debug/shieldwall 2
libbpf: elf: skipping unrecognized data section(4) .rodata.str1.1
libbpf: Kernel error message: underlying driver does not support XDP in native mode
Error: System error, errno: 22

FYI - for the above, I didn't do the BTF "stripping" to the ELF file. I just recompiled it with cargo libbpf build and then regenerated the skeleton with cargo libbpf gen

A couple of thoughts given this experiment:

• bpf_set_link_xdp_fd uses netlink.c, but your implementation uses bpf_program__attach_xdp which ends up calling bpf_link_create in bpf.c. I don't believe the netlink.c API I used here ends up calling bpf_link_create. Does this mean that libbpf gives you two ways to load an XDP app?
• I'm not sure the issue is my VM not having native driver support. libbpf clearly complained about this with my experiment by logging this line libbpf: Kernel error message: underlying driver does not support XDP in native mode. But in the error log I pasted initially, this is not found. Both cases return a 22 errno though.

[jfernandez]

I installed Ubuntu 21.04 on a spare PC I have, and the example compiles fine without any issues.

The problem must be due to the VM (Vagrant bento/ubuntu-20.10) I'm using on my Macbook. I plan to continue development on the PC without a VM, so I'm effectively unblocked at this point. Happy to help debug this more if you are interested in tracking down the actual problem.

[chenhengqi]

Fine. Will take a look later.

[jfernandez]

@anakryiko thanks for the additional info. Once I got the example working, I was wondering how the user-space app was auto-detaching when I exited out. I thought libbpf-rs was doing something behind the scenes. This explains it.

[anakryiko]

@jfernandez are there any remaining issues or we can close this?

[jfernandez]

@anakryiko closing the issue. I forgot to follow up, but I believe this was due to the kernel version on the VM I used initially. I think folks on 5.11 + should be fine with these examples.