Comments (9)
yinjiping
commented on June 3, 2024
@Spartan-65 你好
perf_event ： 周期性采样的事件触发时会执行这个programe (perf_event), 主要用于获取函数栈信息然后推送给用户层处理。
相关load, attach, buffer reader可看下面关联代码。
- load
https://github.com/deepflowio/deepflow/blob/main/agent/src/ebpf/user/load.c#L443 - attach：
https://github.com/deepflowio/deepflow/blob/main/agent/src/ebpf/user/tracer.c#L961
deepflow/agent/src/ebpf/user/probe.c
Line 289 in 520fc9d
- perf reader构建：
deepflow/agent/src/ebpf/user/tracer.c
Line 439 in 4dbb235
deepflow/agent/src/ebpf/user/tracer.c
Line 1137 in 4dbb235
from deepflow.
yinjiping
commented on June 3, 2024
@Spartan-65 您好，这个问题大概是这个内存泄露造成的，可查看这个修复 #6218
from deepflow.
Spartan-65
commented on June 3, 2024
@yinjiping 谢谢，我编译测试一下看有没有问题了
from deepflow.
Spartan-65
commented on June 3, 2024
我执行
bpftool prog命令看到了这些bpf程序，想要请教 """61: perf_event name bpf_perf_event tag 42b497adb6d498f9 gpl"""这个程序是怎么挂载的呢？作用是什么？因为最近碰到一个perf event ringbuf 内存访问错误的问题# bpftool prog 4: kprobe name runtime_execute tag 90d70e4151b22e21 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 432B jited 282B memlock 4096B map_ids 13,30,27 5: kprobe name enter_runtime_n tag 508507df8f0510b7 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 504B jited 336B memlock 4096B map_ids 13,30,29 6: kprobe name exit_runtime_ne tag 29e441df4186a27d gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 984B jited 573B memlock 4096B map_ids 13,30,29,25 7: tracepoint name bpf_func_sched_ tag 3b5385d5ca3c4923 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 392B jited 256B memlock 4096B map_ids 13,30,18,27 8: tracepoint name bpf_func_sched_ tag bc05acdc0a1ffcaa gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 248B jited 177B memlock 4096B map_ids 13,18 9: tracepoint name bpf_func_sys_en tag 5568b62fd1905a97 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 976B jited 569B memlock 4096B map_ids 13,6 10: tracepoint name bpf_func_sys_ex tag 271c699cd6661848 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 29280B jited 18555B memlock 32768B map_ids 6,30,13,15,19,8,12,17,16,28 11: tracepoint name bpf_func_sys_en tag 6997a6a438af7c55 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 976B jited 569B memlock 4096B map_ids 13,5 12: tracepoint name bpf_func_sys_ex tag 6a45212bc75bcd88 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 29288B jited 18557B memlock 32768B map_ids 5,30,13,15,19,8,12,17,16,28 13: tracepoint name bpf_func_sys_en tag b059b443717b4b91 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 22680B jited 13398B memlock 24576B map_ids 13,7,6 14: tracepoint name bpf_func_sys_ex tag 93bd5c3f51a5c4ea gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 29176B jited 18486B memlock 32768B map_ids 6,30,13,15,19,8,12,17,16,28 15: tracepoint name bpf_func_sys_en tag 81e748a081f1ff7a gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 1000B jited 586B memlock 4096B map_ids 13,5 16: tracepoint name bpf_func_sys_ex tag 2a71ee76920e9a90 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 29184B jited 18488B memlock 32768B map_ids 5,30,13,15,19,8,12,17,16,28 17: kprobe name __sys_sendmsg tag 955c941b768968b7 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 1032B jited 628B memlock 4096B map_ids 13,6 18: tracepoint name bpf_func_sys_ex tag f5dc7dff3e9975c2 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 33376B jited 21167B memlock 36864B map_ids 6,30,13,15,19,8,12,17,16,28 19: kprobe name __sys_sendmmsg tag d5c800fea58771c0 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 1080B jited 660B memlock 4096B map_ids 13,6 20: tracepoint name bpf_func_sys_ex tag 0c0a86c366044739 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 33368B jited 21188B memlock 36864B map_ids 6,30,13,15,19,8,12,17,16,28 21: kprobe name __sys_recvmsg tag a3dbfd161a97cf25 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 1056B jited 646B memlock 4096B map_ids 13,5 22: tracepoint name bpf_func_sys_ex tag b7e2f0d26ae95d85 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 33384B jited 21166B memlock 36864B map_ids 5,30,13,15,19,8,12,17,16,28 23: kprobe name __sys_recvmmsg tag 2a63994d875b8a84 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 1152B jited 678B memlock 4096B map_ids 13,5 24: tracepoint name bpf_func_sys_ex tag eec4ed4edb0c6786 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 33376B jited 21187B memlock 36864B map_ids 5,30,13,15,19,8,12,17,16,28 25: kprobe name do_writev tag e4515a37a3b2d195 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 1000B jited 581B memlock 4096B map_ids 13,6 26: tracepoint name bpf_func_sys_ex tag 0694bd8dfa866169 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 33376B jited 21167B memlock 36864B map_ids 6,30,13,15,19,8,12,17,16,28 27: kprobe name do_readv tag 8a62658eec7ab95f gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 1000B jited 581B memlock 4096B map_ids 13,5 28: tracepoint name bpf_func_sys_ex tag 5923ee41b6f8c668 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 33384B jited 21166B memlock 36864B map_ids 5,30,13,15,19,8,12,17,16,28 29: tracepoint name bpf_func_sys_en tag 3951d66b5c406e63 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 14112B jited 7966B memlock 16384B map_ids 13,7,19,5,24,20 30: tracepoint name bpf_func_sys_ex tag 33609d4bdd97b5a4 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 2344B jited 1462B memlock 4096B map_ids 5,13,22,9,15 31: tracepoint name bpf_func_sys_en tag 9b9052a01064d564 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 456B jited 314B memlock 4096B map_ids 9,18 32: tracepoint name bpf_func_sys_ex tag a5a8e89f4e9161b6 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 992B jited 620B memlock 4096B map_ids 23,19,24 33: tracepoint name bpf_func_sys_ex tag 099c054eb523d1c0 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 168B jited 137B memlock 4096B map_ids 20 34: tracepoint name bpf_func_sys_ex tag 099c054eb523d1c0 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 168B jited 137B memlock 4096B map_ids 20 35: tracepoint name bpf_func_sys_en tag 0c77e12b0c5b5e23 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 168B jited 137B memlock 4096B map_ids 20 36: tracepoint name bpf_prog_tp__ou tag 6ea3a0a11a294595 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 11120B jited 6125B memlock 12288B map_ids 9,5,6,11,18 37: kprobe name bpf_prog_kp__ou tag 6ea3a0a11a294595 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 11120B jited 6125B memlock 12288B map_ids 9,5,6,11,18 38: tracepoint name bpf_prog_tp__pr tag 6819113e0d1444d3 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 26392B jited 17177B memlock 28672B map_ids 8,19,17,5,6,16,15 39: kprobe name bpf_prog_kp__pr tag 6819113e0d1444d3 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 26392B jited 17177B memlock 28672B map_ids 8,19,17,5,6,16,14 40: tracepoint name bpf_prog_tp__da tag 785f606d1f9530bf gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 15040B jited 8905B memlock 16384B map_ids 8,19,6,5,24,15,22,27,26,25,23,9,13,20 41: kprobe name bpf_prog_kp__da tag d79783c6940899a5 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 15112B jited 8995B memlock 16384B map_ids 8,19,6,5,24,22,27,26,25,23,9,13,14,20 42: tracepoint name bpf_prog_tp__io tag 3c37195612772952 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 13944B jited 7801B memlock 16384B map_ids 5,22,27,6,26,25,23,13,11,9,15 43: kprobe name uprobe_go_tls_w tag 6f6fa8145ac5bbaa gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 5016B jited 2787B memlock 8192B map_ids 30,10,13,27,31 44: kprobe name uprobe_go_tls_w tag d637b901d45011ec gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 27568B jited 17652B memlock 28672B map_ids 30,27,31,6,13,19,8,12,17,16,14,28 45: kprobe name uprobe_go_tls_r tag 8580da5597b46608 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 5016B jited 2795B memlock 8192B map_ids 30,10,13,27,31 46: kprobe name uprobe_go_tls_r tag 41daa736adf10af5 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 28536B jited 18255B memlock 28672B map_ids 30,27,31,13,28,5,19,8,12,17,16,14 47: kprobe name uprobe_go_http2 tag 5a7b5c44527e36e6 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 14280B jited 8025B memlock 16384B map_ids 13,30,10,28,22,24,19,27,26,25,23,18 48: kprobe name uprobe_go_http2 tag 56fda62222a21045 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 13848B jited 7695B memlock 16384B map_ids 13,30,10,28,22,24,19,27,26,25,23,18 49: kprobe name uprobe_go_http2 tag ef874244e219635a gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 22008B jited 13201B memlock 24576B map_ids 13,30,10,28,22,24,19,27,26,25,23,18 50: kprobe name uprobe_go_http2 tag 11ece519ef186152 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 18224B jited 10718B memlock 20480B map_ids 13,30,10,28,22,24,19,27,26,25,23,18 51: kprobe name uprobe_go_http2 tag 825fb7b70e966a7f gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 22040B jited 13222B memlock 24576B map_ids 13,30,10,28,22,24,19,27,26,25,23,18 52: kprobe name uprobe_go_loopy tag 2ec1f68f1c2df330 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 24088B jited 14134B memlock 24576B map_ids 13,30,10,28,22,24,19,27,26,25,23,18 53: kprobe name uprobe_go_http2 tag bc3926591c97b90c gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 22232B jited 13350B memlock 24576B map_ids 13,30,10,28,22,24,19,27,26,25,23,18 54: kprobe name uprobe_go_http2 tag 950f179507243a4f gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 22232B jited 13350B memlock 24576B map_ids 13,30,10,28,22,24,19,27,26,25,23,18 55: kprobe name uprobe_golang_o tag 130d4399f8d639b5 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 6488B jited 3618B memlock 8192B map_ids 13,30,10,22,24,19,18 56: kprobe name uprobe_golang_o tag f5c26e66db3f2fac gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 6312B jited 3516B memlock 8192B map_ids 13,30,10,22,24,19,18 57: kprobe name uprobe_openssl_ tag 89c1f779d8e5568e gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 1160B jited 653B memlock 4096B map_ids 13,21 58: kprobe name uprobe_openssl_ tag 0f5542801a1e60ac gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 27648B jited 17408B memlock 28672B map_ids 21,6,13,19,8,12,17,16,14,30,28 59: kprobe name uprobe_openssl_ tag 0ba94b3749b7d3f2 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 1160B jited 653B memlock 4096B map_ids 13,21 60: kprobe name uprobe_openssl_ tag b2436988f47b4557 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 27656B jited 17410B memlock 28672B map_ids 21,5,13,19,8,12,17,16,14,30,28 61: perf_event name bpf_perf_event tag 42b497adb6d498f9 gpl loaded_at 2024-04-22T17:06:22+0800 uid 0 xlated 2072B jited 1133B memlock 4096B map_ids 34,35,32,36,33
抱歉，我之前没有没表达清楚。
情况是这样: 我在部署了deepflow的集群node上运行falco(也是使用perf event机制收集内核事件)，会使得falco 通过perf_event_open打开的fd再映射通过mmap映射到内存上的ringbuf 异常
falco会在这里出现退出，这块是读取的data_len 太大了
scap_next buffer corruption
所以我想问的是这个perf event 的bpf函数是否会对其他程序使用perf event机制产生影响?
from deepflow.
yinjiping
commented on June 3, 2024
我执行
bpftool prog命令看到了这些bpf程序，想要请教 """61: perf_event name bpf_perf_event tag 42b497adb6d498f9 gpl"""这个程序是怎么挂载的呢？作用是什么？因为最近碰到一个perf event ringbuf 内存访问错误的问题# bpftool prog 4: kprobe name runtime_execute tag 90d70e4151b22e21 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 432B jited 282B memlock 4096B map_ids 13,30,27 5: kprobe name enter_runtime_n tag 508507df8f0510b7 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 504B jited 336B memlock 4096B map_ids 13,30,29 6: kprobe name exit_runtime_ne tag 29e441df4186a27d gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 984B jited 573B memlock 4096B map_ids 13,30,29,25 7: tracepoint name bpf_func_sched_ tag 3b5385d5ca3c4923 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 392B jited 256B memlock 4096B map_ids 13,30,18,27 8: tracepoint name bpf_func_sched_ tag bc05acdc0a1ffcaa gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 248B jited 177B memlock 4096B map_ids 13,18 9: tracepoint name bpf_func_sys_en tag 5568b62fd1905a97 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 976B jited 569B memlock 4096B map_ids 13,6 10: tracepoint name bpf_func_sys_ex tag 271c699cd6661848 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 29280B jited 18555B memlock 32768B map_ids 6,30,13,15,19,8,12,17,16,28 11: tracepoint name bpf_func_sys_en tag 6997a6a438af7c55 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 976B jited 569B memlock 4096B map_ids 13,5 12: tracepoint name bpf_func_sys_ex tag 6a45212bc75bcd88 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 29288B jited 18557B memlock 32768B map_ids 5,30,13,15,19,8,12,17,16,28 13: tracepoint name bpf_func_sys_en tag b059b443717b4b91 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 22680B jited 13398B memlock 24576B map_ids 13,7,6 14: tracepoint name bpf_func_sys_ex tag 93bd5c3f51a5c4ea gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 29176B jited 18486B memlock 32768B map_ids 6,30,13,15,19,8,12,17,16,28 15: tracepoint name bpf_func_sys_en tag 81e748a081f1ff7a gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 1000B jited 586B memlock 4096B map_ids 13,5 16: tracepoint name bpf_func_sys_ex tag 2a71ee76920e9a90 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 29184B jited 18488B memlock 32768B map_ids 5,30,13,15,19,8,12,17,16,28 17: kprobe name __sys_sendmsg tag 955c941b768968b7 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 1032B jited 628B memlock 4096B map_ids 13,6 18: tracepoint name bpf_func_sys_ex tag f5dc7dff3e9975c2 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 33376B jited 21167B memlock 36864B map_ids 6,30,13,15,19,8,12,17,16,28 19: kprobe name __sys_sendmmsg tag d5c800fea58771c0 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 1080B jited 660B memlock 4096B map_ids 13,6 20: tracepoint name bpf_func_sys_ex tag 0c0a86c366044739 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 33368B jited 21188B memlock 36864B map_ids 6,30,13,15,19,8,12,17,16,28 21: kprobe name __sys_recvmsg tag a3dbfd161a97cf25 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 1056B jited 646B memlock 4096B map_ids 13,5 22: tracepoint name bpf_func_sys_ex tag b7e2f0d26ae95d85 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 33384B jited 21166B memlock 36864B map_ids 5,30,13,15,19,8,12,17,16,28 23: kprobe name __sys_recvmmsg tag 2a63994d875b8a84 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 1152B jited 678B memlock 4096B map_ids 13,5 24: tracepoint name bpf_func_sys_ex tag eec4ed4edb0c6786 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 33376B jited 21187B memlock 36864B map_ids 5,30,13,15,19,8,12,17,16,28 25: kprobe name do_writev tag e4515a37a3b2d195 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 1000B jited 581B memlock 4096B map_ids 13,6 26: tracepoint name bpf_func_sys_ex tag 0694bd8dfa866169 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 33376B jited 21167B memlock 36864B map_ids 6,30,13,15,19,8,12,17,16,28 27: kprobe name do_readv tag 8a62658eec7ab95f gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 1000B jited 581B memlock 4096B map_ids 13,5 28: tracepoint name bpf_func_sys_ex tag 5923ee41b6f8c668 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 33384B jited 21166B memlock 36864B map_ids 5,30,13,15,19,8,12,17,16,28 29: tracepoint name bpf_func_sys_en tag 3951d66b5c406e63 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 14112B jited 7966B memlock 16384B map_ids 13,7,19,5,24,20 30: tracepoint name bpf_func_sys_ex tag 33609d4bdd97b5a4 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 2344B jited 1462B memlock 4096B map_ids 5,13,22,9,15 31: tracepoint name bpf_func_sys_en tag 9b9052a01064d564 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 456B jited 314B memlock 4096B map_ids 9,18 32: tracepoint name bpf_func_sys_ex tag a5a8e89f4e9161b6 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 992B jited 620B memlock 4096B map_ids 23,19,24 33: tracepoint name bpf_func_sys_ex tag 099c054eb523d1c0 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 168B jited 137B memlock 4096B map_ids 20 34: tracepoint name bpf_func_sys_ex tag 099c054eb523d1c0 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 168B jited 137B memlock 4096B map_ids 20 35: tracepoint name bpf_func_sys_en tag 0c77e12b0c5b5e23 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 168B jited 137B memlock 4096B map_ids 20 36: tracepoint name bpf_prog_tp__ou tag 6ea3a0a11a294595 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 11120B jited 6125B memlock 12288B map_ids 9,5,6,11,18 37: kprobe name bpf_prog_kp__ou tag 6ea3a0a11a294595 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 11120B jited 6125B memlock 12288B map_ids 9,5,6,11,18 38: tracepoint name bpf_prog_tp__pr tag 6819113e0d1444d3 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 26392B jited 17177B memlock 28672B map_ids 8,19,17,5,6,16,15 39: kprobe name bpf_prog_kp__pr tag 6819113e0d1444d3 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 26392B jited 17177B memlock 28672B map_ids 8,19,17,5,6,16,14 40: tracepoint name bpf_prog_tp__da tag 785f606d1f9530bf gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 15040B jited 8905B memlock 16384B map_ids 8,19,6,5,24,15,22,27,26,25,23,9,13,20 41: kprobe name bpf_prog_kp__da tag d79783c6940899a5 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 15112B jited 8995B memlock 16384B map_ids 8,19,6,5,24,22,27,26,25,23,9,13,14,20 42: tracepoint name bpf_prog_tp__io tag 3c37195612772952 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 13944B jited 7801B memlock 16384B map_ids 5,22,27,6,26,25,23,13,11,9,15 43: kprobe name uprobe_go_tls_w tag 6f6fa8145ac5bbaa gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 5016B jited 2787B memlock 8192B map_ids 30,10,13,27,31 44: kprobe name uprobe_go_tls_w tag d637b901d45011ec gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 27568B jited 17652B memlock 28672B map_ids 30,27,31,6,13,19,8,12,17,16,14,28 45: kprobe name uprobe_go_tls_r tag 8580da5597b46608 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 5016B jited 2795B memlock 8192B map_ids 30,10,13,27,31 46: kprobe name uprobe_go_tls_r tag 41daa736adf10af5 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 28536B jited 18255B memlock 28672B map_ids 30,27,31,13,28,5,19,8,12,17,16,14 47: kprobe name uprobe_go_http2 tag 5a7b5c44527e36e6 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 14280B jited 8025B memlock 16384B map_ids 13,30,10,28,22,24,19,27,26,25,23,18 48: kprobe name uprobe_go_http2 tag 56fda62222a21045 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 13848B jited 7695B memlock 16384B map_ids 13,30,10,28,22,24,19,27,26,25,23,18 49: kprobe name uprobe_go_http2 tag ef874244e219635a gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 22008B jited 13201B memlock 24576B map_ids 13,30,10,28,22,24,19,27,26,25,23,18 50: kprobe name uprobe_go_http2 tag 11ece519ef186152 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 18224B jited 10718B memlock 20480B map_ids 13,30,10,28,22,24,19,27,26,25,23,18 51: kprobe name uprobe_go_http2 tag 825fb7b70e966a7f gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 22040B jited 13222B memlock 24576B map_ids 13,30,10,28,22,24,19,27,26,25,23,18 52: kprobe name uprobe_go_loopy tag 2ec1f68f1c2df330 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 24088B jited 14134B memlock 24576B map_ids 13,30,10,28,22,24,19,27,26,25,23,18 53: kprobe name uprobe_go_http2 tag bc3926591c97b90c gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 22232B jited 13350B memlock 24576B map_ids 13,30,10,28,22,24,19,27,26,25,23,18 54: kprobe name uprobe_go_http2 tag 950f179507243a4f gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 22232B jited 13350B memlock 24576B map_ids 13,30,10,28,22,24,19,27,26,25,23,18 55: kprobe name uprobe_golang_o tag 130d4399f8d639b5 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 6488B jited 3618B memlock 8192B map_ids 13,30,10,22,24,19,18 56: kprobe name uprobe_golang_o tag f5c26e66db3f2fac gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 6312B jited 3516B memlock 8192B map_ids 13,30,10,22,24,19,18 57: kprobe name uprobe_openssl_ tag 89c1f779d8e5568e gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 1160B jited 653B memlock 4096B map_ids 13,21 58: kprobe name uprobe_openssl_ tag 0f5542801a1e60ac gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 27648B jited 17408B memlock 28672B map_ids 21,6,13,19,8,12,17,16,14,30,28 59: kprobe name uprobe_openssl_ tag 0ba94b3749b7d3f2 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 1160B jited 653B memlock 4096B map_ids 13,21 60: kprobe name uprobe_openssl_ tag b2436988f47b4557 gpl loaded_at 2024-04-22T17:06:21+0800 uid 0 xlated 27656B jited 17410B memlock 28672B map_ids 21,5,13,19,8,12,17,16,14,30,28 61: perf_event name bpf_perf_event tag 42b497adb6d498f9 gpl loaded_at 2024-04-22T17:06:22+0800 uid 0 xlated 2072B jited 1133B memlock 4096B map_ids 34,35,32,36,33抱歉，我之前没有没表达清楚。 情况是这样: 我在部署了deepflow的集群node上运行falco(也是使用perf event机制收集内核事件)，会使得falco 通过perf_event_open打开的fd再映射通过mmap映射到内存上的ringbuf 异常
falco会在这里出现退出，这块是读取的data_len 太大了 scap_next buffer corruption
所以我想问的是这个perf event 的bpf函数是否会对其他程序使用perf event机制产生影响?
@Spartan-65
deepflow agent实际调用的attach接口在这里 https://github.com/iovisor/bcc/blob/679166bdee74302b46b14c3a8fe5c3db7198d3f4/src/cc/libbpf.c#L1729
实际上我们知道：syscall(__NR_perf_event_open） 会生成一个事件的fd, 这个事件会被kernel perf子系统捕获，并交给事先绑定的eBPF prog 处理。(事件和eBPF prog 会通过ioctl(event_fd, PERF_EVENT_IOC_SET_BPF, progfd)绑定)。
可以有多个采样事件并互不影响（不同的事件由各自指定eBPF prog处理）。
perf ringbuffer (BPF_MAP_TYPE_PERF_EVENT_ARRAY)， 这种buffer是deepflow自身创建的内存和perf buffer关联
这一部分的底层接口 https://github.com/iovisor/bcc/blob/679166bdee74302b46b14c3a8fe5c3db7198d3f4/src/cc/libbpf.c#L1601
mmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_SHARED, reader->fd, 0) // reader->fd 是perf buffer的perCPU 事件fd
不会对其他程序产生影响，这是Linux kernel perf子系统机制保障的。
另外，您这边提到的ringbuffer 应该是 BPF_MAP_TYPE_RINGBUF 这种类型的map， 它和我们使用的eBPF perfbuf是完全不一样的东西，deepflow agent没有使用BPF_MAP_TYPE_RINGBUF 这种类型的map。
from deepflow.
Spartan-65
commented on June 3, 2024
@yinjiping falco 使用的也是 BPF_MAP_TYPE_PERF_EVENT_ARRAY，使用方式基本上是一样的，但是falco不会有这个bpf prog，perf event机制应该是内核实现的，不用加载perf_event的bpf prog。意思是perf event 的bpf 程序是bcc加载的？
perf_event_fd:
https://github.com/falcosecurity/libs/blob/master/userspace/libscap/engine/bpf/scap_bpf.c#L1607
mmap映射:
https://github.com/falcosecurity/libs/blob/master/userspace/libscap/engine/bpf/scap_bpf.c#L1629
环形队列 error的位置:
https://github.com/falcosecurity/libs/blob/master/userspace/libscap/ringbuffer/ringbuffer.h#L272
在error的位置部分打印信息
pe->len: 2896438784, pe->type: 35682, pe->nparams:12528, dev->m_sn_len: 20696 dev->m_lastreadsize: 57118428
Syscall event drop monitoring:
- event drop detected: 0 occurrences
- num times actions taken: 0
2024-04-26T09:38:57+0000: Closing event source 'syscall'
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:
Error: scap_next buffer corruption这个pe->type范围在几百以内，说明这个队列指针偏移可能错了，不知道为什么在装了deepflow的节点上这块会出错，挺奇怪的
from deepflow.
sharang
commented on June 3, 2024
方便找到它，我先打开。
from deepflow.
yinjiping
commented on June 3, 2024
@yinjiping falco 使用的也是 BPF_MAP_TYPE_PERF_EVENT_ARRAY，使用方式基本上是一样的，但是falco不会有这个bpf prog，perf event机制应该是内核实现的，不用加载perf_event的bpf prog。意思是perf event 的bpf 程序是bcc加载的？ perf_event_fd: https://github.com/falcosecurity/libs/blob/master/userspace/libscap/engine/bpf/scap_bpf.c#L1607 mmap映射: https://github.com/falcosecurity/libs/blob/master/userspace/libscap/engine/bpf/scap_bpf.c#L1629 环形队列 error的位置: https://github.com/falcosecurity/libs/blob/master/userspace/libscap/ringbuffer/ringbuffer.h#L272
在error的位置部分打印信息
pe->len: 2896438784, pe->type: 35682, pe->nparams:12528, dev->m_sn_len: 20696 dev->m_lastreadsize: 57118428 Syscall event drop monitoring: - event drop detected: 0 occurrences - num times actions taken: 0 2024-04-26T09:38:57+0000: Closing event source 'syscall' Events detected: 0 Rule counts by severity: Triggered rules by rule name: Error: scap_next buffer corruption这个pe->type范围在几百以内，说明这个队列指针偏移可能错了，不知道为什么在装了deepflow的节点上这块会出错，挺奇怪的
@Spartan-65 想和您确认一下，这个和deepflow agent正相关吗？
即： 在这个节点上，启用deepflow agent会有存在问题，而把deepflow agent停用后不会出现您描述的问题，是这样吗？
另外，Linux系统是什么，具体的内核版本是多少？
from deepflow.
Spartan-65
commented on June 3, 2024
@yinjiping 目前看是的，有deepflow的节点falco就会出错
uname -a
Linux bclinux-test 4.19.25-204.el7.bclinux.x86_64 #1 SMP Wed Dec 23 15:41:17 CST 2020 x86_64 x86_64 x86_64 GNU/Linux可以单点部署falco ebpf模式测试
docker run --pull=always -i -t --rm --privileged -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro -v /etc:/host/etc -e FALCO_DRIVER_LOADER_OPTIONS=ebpf falcosecurity/falcofrom deepflow.
Related Issues (20)
- [BUG] 自定义开发wasm 遇见的几个问题 HOT 6
- fix performance issue of lookup_all_by_epc()
- 关于暂存数据上报的问题 HOT 3
- [BUG] flame graph HOT 2
- [BUG] 火焰图无法关联redis span HOT 1
- [BUG] panic when parse redis HOT 1
- k8s pod 数据大量缺失 HOT 10
- grafana 查询traceid attribute.requestid 不生效[BUG] HOT 1
- trace 看板展示异常 HOT 2
- [FR] 关于clickhouse的deepflow_system库中deepflow_system_agent和deepflow_system_server两张表的维度声明
- [FR] Questions about the deepflow_system_agent and deepflow_system_server tables in ClickHouse HOT 1
- On the issue of insufficient stack space HOT 2
- QA:能否帮忙介绍一下fastpath中索引ACL流程？ HOT 1
- [BUG] wasm 插件 memory overflow 导致agent pod 重启 HOT 1
- [BUG] v6.4 Trace Completion API 调用返回500
- [FR] Agent对Traefik性能影响基准测试 HOT 5
- [BUG] MySQL注释添加全局id后grafana界面无trace id
- grafana distributed-tracing 查询错误[BUG] HOT 2
- [BUG] clickhouse数据库乱码 HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deepflow.