Kernel version 5.18.2-1 not supported about tetragon

[willfindlay]

If you end up going the first route, here are the helm values you're looking for:

tetragon/install/kubernetes/values.yaml

Line 14 in 402d16f

extraHostPathMounts: []

tetragon/install/kubernetes/values.yaml

Line 118 in 402d16f

btf: ""

[willfindlay]

Is this a KinD cluster by any chance? If so you would need to make sure the file is present in the KinD container in order for extraHostPathMounts to work properly.

[Forsworns]

Is this a KinD cluster by any chance? If so you would need to make sure the file is present in the KinD container in order for extraHostPathMounts to work properly.

Yes, it'a local cluster created via KinD. Thanks, I understand it now. The host path is in fact the KinD container in this situation

[willfindlay]

You need to compile your kernel with BTF info. Check for the existence of /sys/kernel/btf.

[willfindlay]

Alternatively, you can generate the BTF yourself and load it into the Tetragon pod but that's a bit more of an advanced use case.

[nisainan]

Is there any demo or guide of fixing this problem?

[jjsluck0907]

No sys/kernel/btf。Is there any relevant documentation? Thank you

[willfindlay]

https://www.kernel.org/doc/html/latest/bpf/btf.html#btf-generation This explains how to generate the BTF manually. You could then set the helm values such that this is mounted into the Tetragon container and used by Tetragon.

Alternatively, you'll need to recompile your kernel with BTF support. That involves setting some kconfig flags and compiling it with a recent version of pahole installed on your system.

[jjsluck0907]

BTF is enabled by default using cenots 8 。
Do not upgrade the latest kernel. The default kernel version will not be reported as unsupported kernel when installing tetragon。
Upgrade the latest kernel version that is not supported by the same message。
https://github.com/aquasecurity/btfhub/blob/main/docs/supported-distros.md

[Forsworns]

If you end up going the first route, here are the helm values you're looking for:

tetragon/install/kubernetes/values.yaml

Line 14 in 402d16f

extraHostPathMounts: []

tetragon/install/kubernetes/values.yaml

Line 118 in 402d16f

btf: ""

I download a pre-built btf file from BTF-Hub and place it to /var/lib/tetragon/.
Then I read the files in install/kubernetes/templates and change the install/kubernetes/values.yaml to

extraHostPathMounts: 
- name: "btf-dir"
  mountPath: /var/lib/tetragon/
  mountPropagation: HostToContainer
  readOnly: false
  ...
  btf: "/var/lib/tetragon/4.18.0-147.5.1.el8_1.x86_64.btf"

However, the file is not detected and tetragon reports the btf file does not exists.

Here are related logs:

> ls /var/lib/tetragon/

4.18.0-147.5.1.el8_1.x86_64.btf 

> kubectl logs -f "$(kubectl get pod -n kube-system | grep tetragon | awk '{print $1}')" -n kube-system -c tetragon

time="2022-06-20T02:07:35Z" level=warning msg="BPF filesystem is going to be mounted automatically in /run/cilium/bpffs. However, it probably means that Cilium is running inside container and BPFFS is not mounted on the host. for more information, see: https://cilium.link/err-bpf-mount"
time="2022-06-20T02:07:35Z" level=info msg="Available sensors" sensors=
time="2022-06-20T02:07:35Z" level=info msg="Registered tracing sensors" sensors="kprobe sensor, tracepoint sensor"
time="2022-06-20T02:07:35Z" level=info msg="Registered probe types" types="tracepoint sensor, kprobe sensor"
**time="2022-06-20T02:07:35Z" level=fatal msg="Failed to start tetragon" error="tetragon, aborting kernel autodiscovery failed: User specified BTF does not exist: stat /var/lib/tetragon/4.18.0-147.5.1.el8_1.x86_64.btf: no such file or directory"**

> kubectl describe pod "$(kubectl get pod -n kube-system | grep tetragon | awk '{print $1}')" -n kube-system
tetragon:
   Mounts:
      /etc/tetragon from tetragon-config (ro)
      /procRoot from host-proc (rw)
      /sys/fs/bpf from bpf-maps (rw)
      /var/lib/tetragon/ from btf-dir (rw)
      /var/run/cilium from cilium-run (rw)
      /var/run/cilium/tetragon from export-logs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c462t (ro)

[kkourt]

However, the file is not detected and tetragon reports the btf file does not exists.

Does the file var/lib/tetragon/4.18.0-147.5.1.el8_1.x86_64.btf exist on the host or in the tetragon pod? If it exists on the host, but not in the pod then the tetragon agent will not be able to find it.

[Forsworns]

If it exists on the host, but not in the pod then the tetragon agent will not be able to find it.

Sure, the file is not in the pod. But is it expected?

The extraHostPathMounts has mounted the host path /var/lib/tetragon/ to the tetragon pod. I thought the btf file should be shared in this way.

Did I misunderstand the effects of extraHostPathMounts?

[kkourt]

If it exists on the host, but not in the pod then the tetragon agent will not be able to find it.

Sure, the file is not in the pod. But is it expected?

The extraHostPathMounts has mounted the host path /var/lib/tetragon/ to the tetragon pod. I thought the btf file should be shared in this way.

Did I misunderstand the effects of extraHostPathMounts?

Ah right! I think you are correct!

In that case, I'm not sure why we are getting that error if the file exists:

tetragon/pkg/btf/btf.go

Lines 86 to 89 in 1b7d231

	btfFile, err = observerFindBTF(ctx, lib, btf)
	if err != nil {
	return fmt.Errorf("tetragon, aborting kernel autodiscovery failed: %w", err)
	}

tetragon/pkg/btf/btf.go

Lines 72 to 74 in 1b7d231

	if err := btfFileExists(btf); err != nil {
	return btf, fmt.Errorf("User specified BTF does not exist: %w", err)
	}

tetragon/pkg/btf/btf.go

Lines 25 to 27 in 1b7d231

	func btfFileExists(file string) error {
	_, err := os.Stat(file)
	return err

Would it be possible to kubectl exec into the tetragon pod and check that the file is indeed there?

[kkourt]

Could it be that there the btf value is not properly passed?

At the beginning of the tetragon logs, there should be a line that prints the full configuration:

tetragon/cmd/tetragon/main.go

Lines 107 to 108 in 1b7d231

	log.WithField("version", version.Version).Info("Starting tetragon")
	log.WithField("config", viper.AllSettings()).Info("config settings")

Could you please check it out?

[Forsworns]

Could it be that there the btf value is not properly passed?

Yes, they are correctly set. Tetragon works well :) I omitted it in the above comments, but I remembered it.