wkz / ply

Light-weight Dynamic Tracer for Linux

ply

Documentation and language reference is available at wkz.github.io/ply.

A light-weight dynamic tracer for Linux that leverages the kernel's BPF VM in concert with kprobes and tracepoints to attach probes to arbitrary points in the kernel. Most tracers that generate BPF bytecode are based on the LLVM based BCC toolchain. ply on the other hand has no required external dependencies except for libc. In addition to x86_64, ply also runs on aarch64, arm, riscv64, riscv32, and powerpc. Adding support for more ISAs is easy.

ply follows the Little Language approach of yore, compiling ply scripts into Linux BPF programs that are attached to kprobes and tracepoints in the kernel. The scripts have a C-like syntax, heavily inspired by dtrace(1) and, by extension, awk(1).

The primary goals of ply are:

• Expose most of the BPF tracing feature-set in such a way that new scripts can be whipped up very quickly to test different hypotheses.

• Keep dependencies to a minimum. Right now Flex and Bison are required at build-time, leaving libc as the only runtime dependency. Thus, ply is well suited for embedded targets.

If you need more fine-grained control over the kernel/userspace interaction in your tracing, checkout the bcc project which compiles C programs to BPF using LLVM in combination with a python userspace recipient to give you the full six degrees of freedom.

Examples

Here are some one-liner examples to show the kinds of questions that ply can help answer.

What is the distribution of the returned sizes from read(2)s to the VFS?

ply 'kretprobe:vfs_read { @["size"] = quantize(retval); }'

Which processes are receiving errors when reading from the VFS?

ply 'kretprobe:vfs_read if (retval < 0) { @[pid, comm, retval] = count(); }'

Which files are being opened, by who?

ply 'kprobe:do_sys_open { printf("%v(%v): %s\n", comm, uid, str(arg1)); }'

When sending packets, where are we coming from?

ply 'kprobe:dev_queue_xmit { @[stack] = count(); }'

From which hosts and ports are we receiving TCP resets?

ply 'tracepoint:tcp/tcp_receive_reset {
	printf("saddr:%v port:%v->%v\n",
		data->saddr, data->sport, data->dport);
}'

Build and Installation

ply uses GNU's autotools as its build system. When building from a Git clone, use the following steps:

./autogen.sh   # to generate the configure script
./configure
make
make install   # you probably need to be root for this

Contributing

Contributions are welcome! To help you on your way, the test/ directory contains ready-made infrastructure to:

• Test cross-compilation on all supported architectures.
• Run a simple test suite on a range of machines using QEMU system emulation.
• Run interactive sessions on QEMU machines.

A GitHub Action is setup to run these jobs. Please make sure to test your changes locally before opening a PR to avoid unnecessary review cycles.

Maintainers

ply is developed and maintained by Tobias Waldekranz. Please direct all bug reports and pull requests towards the official Github repo.

ply's Issues

Embedded linux system：Verifying kprobe... ERR:-22

hi,
I got an error when I tried to run ply ，My system is an embedded development board, the model is TI am5748 IDK ，It is an arm A15 (armv7l) architecture cpu。

error message:

root@am57xx-evm:~# ply -T
Verifying kernel config (/proc/config.gz)... OK
Ensuring that debugfs is mounted... OK
Verifying kprobe... ERR:-22
ERROR
Verifying tracepoint... /bin/sh: line 79:  1203 Aborted                 (core dumped) $PLYBIN 'tracepoint:sched/sched_switch { exit(0); }' 2> /dev/null
ERROR


root@am57xx-evm:~# ply -dS "tracepoint:sched/sched_process_exit { ${1} }"
ply: provider/tracepoint.c:197: tracepoint_parse: Assertion `offs == type_offsetof(t, t->sou.fields[n - 1].name)' failed.
Aborted (core dumped)


root@am57xx-evm:~# cat test.ply
#!/usr/bin/env ply
kprobe:do_sys_open
{
            printf("%v(%v): %s\n",
               comm, pid, str(arg1));
}
root@am57xx-evm:~# ./test.ply
error: unable to load kprobe:do_sys_open, errno:22
error: no output from kernel bpf verifier, retry with debugging enabled
ERR:-22



root@am57xx-evm:~# zgrep "BPF" /proc/config.gz
# CONFIG_CGROUP_BPF is not set
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
# CONFIG_BPF_JIT_ALWAYS_ON is not set
# CONFIG_NETFILTER_XT_MATCH_BPF is not set
# CONFIG_BPFILTER is not set
CONFIG_NET_CLS_BPF=y
# CONFIG_NET_ACT_BPF is not set
CONFIG_BPF_JIT=y
# CONFIG_BPF_STREAM_PARSER is not set
CONFIG_HAVE_EBPF_JIT=y
# CONFIG_NBPFAXI_DMA is not set
CONFIG_BPF_EVENTS=y
# CONFIG_TEST_BPF is not set

root@am57xx-evm:~# zgrep "KPROBE" /proc/config.gz
CONFIG_KPROBES=y
CONFIG_HAVE_KPROBES=y
CONFIG_KPROBE_EVENTS=y
# CONFIG_KPROBES_SANITY_TEST is not set
# CONFIG_ARM_KPROBES_TEST is not set

root@am57xx-evm:~# zgrep "FTRACE" /proc/config.gz
# CONFIG_PSTORE_FTRACE is not set
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_FTRACE=y
# CONFIG_FTRACE_SYSCALLS is not set
CONFIG_DYNAMIC_FTRACE=y
CONFIG_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_FTRACE_MCOUNT_RECORD=y
# CONFIG_FTRACE_STARTUP_TEST is not set


root@am57xx-evm:~# cat /sys/kernel/debug/tracing/events/sched/sched_process_exit/format
name: sched_process_exit
ID: 56
format:
        field:unsigned short common_type;       offset:0;       size:2; signed:0;
        field:unsigned char common_flags;       offset:2;       size:1; signed:0;
        field:unsigned char common_preempt_count;       offset:3;       size:1; signed:0;
        field:int common_pid;   offset:4;       size:4; signed:1;
        field:unsigned short common_migrate_disable;    offset:8;       size:2; signed:0;
        field:unsigned short common_padding;    offset:10;      size:2; signed:0;

        field:char comm[16];    offset:16;      size:16;        signed:0;
        field:pid_t pid;        offset:32;      size:4; signed:1;
        field:int prio; offset:36;      size:4; signed:1;

print fmt: "comm=%s pid=%d prio=%d", REC->comm, REC->pid, REC->prio

My linux is ：PREEMTP_RT

root@am57xx-evm:~# uname -a
Linux am57xx-evm 4.19.94-rt39-ga242ccf3f1 #2 SMP PREEMPT RT Sun May 30 17:58:43 PDT 2021 armv7l GNU/Linux

root@am57xx-evm:~# ply -v
ply  (linux-version:267067~4.19.59)

What should I do？

Thanks You,
unintialized

ply: compile.c:215: ply_compile: Assertion `!err' failed

When I trace more than 9 system functions in a ply file, an error will always be reported when running ply. This is why, this is the error code: ply: compile.c:215: ply_compile: Assertion `!err' failed.
Aborted (core dumped)

How can aarch64_be be supported?

i want to use ply in aarch64 big endian， so i clone aarch64.c to aarch64_be.c， but it doesn't work normally, it means it can't print kernel function name, but kernel function addr。 so would you tell me how to support aarch64_be? thanks very much！

build docs

I got this far:

./autogen.sh
./configure
make

But then it can't find bpf.h:

root@bgregg-build-i-xxxxxxxx:/mnt/src/ply# make
Making all in src
make[1]: Entering directory `/mnt/src/ply/src'
make  all-am
make[2]: Entering directory `/mnt/src/ply/src'
  CC       lang/ply-parse.o
In file included from lang/parse.y:25:0:
lang/ast.h:29:23: fatal error: linux/bpf.h: No such file or directory
compilation terminated.
make[2]: *** [lang/ply-parse.o] Error 1
make[2]: Leaving directory `/mnt/src/ply/src'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/mnt/src/ply/src'
make: *** [all-recursive] Error 1

I have linux/bpf.h in my kernel build directory, and it's also under /lib/modules/uname -r/source/include

But it isn't under /usr/include. And "make headers_install" doesn't put it there...

printing issue of aggregated data on arm 32

On arm 32 bit system the following print occured when printing comms in an aggregated map.

# ply 't:sched/sched_switch { @[data->next_comm] = count(); }'
ply: active
^Cply: deactivating

@:
{ kworker/u4:3    }: 1
{ 
000:  70 6c 79 00 68 00 67 65  74 74 79 00 00 00 00 00   ply.h.ge tty.....
 }: 1
{ 
000:  73 73 64 70 64 00 72 2f  30 00 00 00 00 00 00 00   ssdpd.r/ 0.......
 }: 1
{ kworker/0:1     }: 2
{ kworker/1:3     }: 3
{ rcu_preempt     }: 6
{ swapper/1       }: 12
{ watchdogd       }: 24
{ swapper/0       }: 42

Unable to build correctly on Ubuntu

Heya,
While trying to build ply on a ubuntu 14.04.5 with HWE, I get the following error:

$~/ply# make
Making all in src
make[1]: Entering directory `/root/ply/src'
  LEX      lang/lex.c
  YACC     lang/parse.c
updating lang/parse.h
make  all-am
make[2]: Entering directory `/root/ply/src'
  CC       lang/ply-lex.o
  CC       lang/ply-parse.o
In file included from lang/parse.y:25:0:
../src/include/ply/ast.h:32:23: fatal error: linux/bpf.h: No such file or directory
 #include <linux/bpf.h>
                       ^
compilation terminated.
make[2]: *** [lang/ply-parse.o] Error 1
make[2]: Leaving directory `/root/ply/src'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/root/ply/src'
make: *** [all-recursive] Error 1

I know there is the option to pass the kernel dir:

$ ~/ply# find / -name bpf.h
/usr/src/linux-headers-4.4.0-66/include/linux/bpf.h
/usr/src/linux-headers-4.4.0-66/include/uapi/linux/bpf.h
/usr/src/linux-headers-3.13.0-108-generic/include/config/netfilter/xt/match/bpf.h
/usr/src/linux-headers-3.13.0-108-generic/include/config/net/cls/bpf.h
/usr/src/linux-headers-4.4.0-66-generic/include/config/bpf.h
/usr/src/linux-headers-4.4.0-66-generic/include/config/netfilter/xt/match/bpf.h
/usr/src/linux-headers-4.4.0-66-generic/include/config/test/bpf.h
/usr/src/linux-headers-4.4.0-66-generic/include/config/net/act/bpf.h
/usr/src/linux-headers-4.4.0-66-generic/include/config/net/cls/bpf.h
/usr/include/bcc/compat/linux/bpf.h

trying to use kerneldir flag:

$~/ply# ./configure --with-kerneldir=/usr/src/linux-headers-4.4.0-66/
checking for a BSD-compatible install... /usr/bin/install -c
[...]
config.status: creating src/config.h
config.status: src/config.h is unchanged
config.status: executing depfiles commands

$ ~/ply# make
Making all in src
make[1]: Entering directory `/root/ply/src'
  HEADERS  /usr/src/linux-headers-4.4.0-66/(x86_64)
make -C /usr/src/linux-headers-4.4.0-66/ ARCH=x86_64 INSTALL_HDR_PATH=/root/ply/src/.kernel headers_install
make[2]: Entering directory `/usr/src/linux-headers-4.4.0-66'
  CHK     include/generated/uapi/linux/version.h
make[3]: *** No rule to make target `arch/x86/entry/syscalls/syscall_32.tbl', needed by `arch/x86/entry/syscalls/../../include/generated/asm/syscalls_32.h'.  Stop.
make[2]: *** [archheaders] Error 2
make[2]: Leaving directory `/usr/src/linux-headers-4.4.0-66'
make[1]: *** [.kernel/include/linux/version.h] Error 2
make[1]: Leaving directory `/root/ply/src'
make: *** [all-recursive] Error 1

Is there anything I'm doing incorrectly?

Thanks,
Ricardo

tracepoint self-test failed

Hello,

I came across a self-test failure while I was setting up ply inside a Buildroot-generated x86-64 Linux image:

# ply --version
ply  (linux-version:393479~6.1.7)

# ply -T
WARN: Unable to verify kernel config
Ensuring that debugfs is mounted... OK
Verifying kprobe... OK
Verifying tracepoint... [ 2254.003304] traps: ply[170] trap divide error ip:7f4a30f0594f sp:7fff766703a0 error:0 in libply.so.0.0.0[7f4a30efe000+f000]
ERROR
Verifying special... OK
Verifying interval... OK

# uname -a
Linux test 6.1.7 #1 SMP PREEMPT_DYNAMIC Thu Jan 19 22:46:28 EST 2023 x86_64 GNU/Linux

Do you think it might be an issue related to the kernel version? At this point, it's the latest.

kprobe seems to be okay.

# cat opensnoop.py 
#!/sbin/ply

kprobe:do_sys_open*
{
    path[kpid] = str(arg1);
}

kretprobe:do_sys_open*
{
    printf("PID: %v PROCESS: %v PATH: %v RETVAL: %d\n", pid, comm, path[kpid], retval);
    delete path[kpid];
}

# ply opensnoop.py 
^CPID:   175 PROCESS: ply             PATH: /sys/kernel/debug/tracing/events/ply175/p5608861e6ad0_do_sys_openat2/id                                                         RETVAL: 12

I added following options to x86-64 config file before building the Linux image via Buildroot:

CONFIG_BPF_SYSCALL=y
CONFIG_UPROBES=y
CONFIG_TRACEPOINTS=y
CONFIG_FTRACE=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_PERF_EVENTS=y

CONFIG_NOP_TRACER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_TRACER_MAX_TRACE=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_GENERIC_TRACER=y
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
CONFIG_SCHED_TRACER=y
CONFIG_TRACER_SNAPSHOT=y
CONFIG_STACK_TRACER=y

I'm doing this kinda thing for the first time, so please let me know if you need more information. Thanks!

ply v0.2 isn't runnable on anything I tried yet

13:21:25 wkz/ply git:(master) ✗ » ./src/ply scripts/ptysnoop.ply
info: creating kallsyms cache
warning: unable to create kallsyms cache: Operation not permitted
<input>:3:19-40: error: unknown symbol 'strcmp'.
13:21:27 wkz/ply git:(master) ✗ » sudo ./src/ply scripts/ptysnoop.ply
[sudo] password for aktau: 
<input>:3:19-40: error: unknown symbol 'strcmp'.
13:21:36 wkz/ply git:(master) ✗ » sudo ./src/ply scripts/opensnoop.ply


14: error: syntax error, unexpected AGG
13:21:41 wkz/ply git:(master) ✗ » sudo ./src/ply scripts/execsnoop.ply
6: error: syntax error, unexpected '=', expecting ';'
13:21:47 wkz/ply git:(master) ✗ » sudo ./src/ply scripts/net-rx.ply
6: error: syntax error, unexpected AGG

You're probably aware of this, I just wanted to try it out. Do you have some updated scripts which work? I assume you try running it after making a change :D.

It might be nice to have a "compile-but-don't-execute" mode so that you can run some CI (travis) on every commit.

Thanks for ply(1), it's the neatest little bytecode compiler I know.

tracepoint_parse bug

trace event:
syscalls/sys_enter_close

result in: offs = 16, 8+4=12, 16 != 12, it maybe need the packed dealing .

name: sys_enter_close
ID: 623
format:
field:unsigned short common_type; offset:0; size:2; signed:0;
field:unsigned char common_flags; offset:2; size:1; signed:0;
field:unsigned char common_preempt_count; offset:3; size:1; signed:0;
field:int common_pid; offset:4; size:4; signed:1;

field:int __syscall_nr;	offset:8;	size:4;	signed:1;
field:unsigned int fd;	offset:16;	size:8;	signed:0;

print fmt: "fd: 0x%08lx", ((unsigned long)(REC->fd))

ply not compiling on linux <v5.5 because of BPF_FUNC_probe_read_kernel

When compiling with a kernel version < v5.5 it fails with:

ir.c: In function 'ir_emit_read_to_sym':
ir.c:419:31: error: 'BPF_FUNC_probe_read_kernel' undeclared (first use in this function); did you mean 'BPF_FUNC_probe_read_str'?
419 | ir_emit_insn(ir, CALL(BPF_FUNC_probe_read_kernel), 0, 0);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~

Looking at the linux repo it turned out that the function 'bpf_probe_read_kernel' where introduced with commit 6ae08ae3dea2cfa03dd3665a3c8475c2d429ef47 ("bpf: Add probe_read_{user, kernel} and probe_read_{user, kernel}_str helpers") into the kernel.

What's the reason about this error: syntax error, unexpected IDENT, expecting ')' ?

I want to trace kernel struct and derenference some elements and I've tried the following：
cat test.ply
#!/sbin/ply

#include <net/sock.h>

kprobe:tcp_set_state
{
printf("%v %v %v %v\n", pid, comm, str(((struct sock *)arg0)->sk_state), arg1);
}

but there's an error:

cross compiling issue, run time error

I am trying to compile ply for arm64:
I have my the cross compiled libraries like libc.so.c at
/home/demera/proj/thirdparty/openwrt-1505/staging_dir/toolchain-axel-qca/lib/
and my cross compiled tool chain like aarch64-openwrt-linux-gnu-gcc at
/home/demera/proj/thirdparty/openwrt-1505/staging_dir/toolchain-axel-qca/bin
so I exported the PATH variable to the "bin" directory and configured it like this:
./configure --host=aarch64-openwrt-linux-gnu --with-kerneldir=/home/demera/proj/linux/
make gives a warning like this at the end, but builds the binary:

aarch64-openwrt-linux-gnu-gcc: warning: environment variable 'STAGING_DIR' not defined
  CCLD     ply
aarch64-openwrt-linux-gnu-gcc: warning: environment variable 'STAGING_DIR' not defined
aarch64-openwrt-linux-gnu-gcc: warning: environment variable 'STAGING_DIR' not defined
/home/demera/proj/thirdparty/openwrt-1505/staging_dir/toolchain-axel-qca/lib/gcc/aarch64-openwrt-linux-gnu/5.3.0/../../../../aarch64-openwrt-linux-gnu/bin/ld: skipping incompatible /usr/lib/libc.so when searching for -lc
/home/demera/proj/thirdparty/openwrt-1505/staging_dir/toolchain-axel-qca/lib/gcc/aarch64-openwrt-linux-gnu/5.3.0/../../../../aarch64-openwrt-linux-gnu/bin/ld: skipping incompatible /usr/lib/libc.so.6 when searching for libc.so.6
/home/demera/proj/thirdparty/openwrt-1505/staging_dir/toolchain-axel-qca/lib/gcc/aarch64-openwrt-linux-gnu/5.3.0/../../../../aarch64-openwrt-linux-gnu/bin/ld: skipping incompatible /usr/lib/libc_nonshared.a when searching for libc_nonshared.a

But when I run the program on the target system, I get:
EB:/home/demera# ./ply -c 'kprobe:netif_rx { printf("%v\n", stack()) }'
ERR pvdr_resolve_call : in 'kprobe:netif_rx', unknown function 'stack'

I suspect the linker issue above, so my question is:
How can I ask the build system to look for libraries under the cross compiled libraries ?

Thanks.

Deactivating on lost events

I find that ply will simply deactivate and quit with a message saying it lost events in some cases. In my particular example I was trying to find all the files emacs opens on launch, but a trivial example is to open two shells, launch ply on one of them with this:

sudo ply 'kprobe:do_sys_open / !strcmp(comm, "find") / { printf("%v(%v): %s\n", comm, pid, str(arg1)); }'

In another shell run a find command like find / -type f -print

After a little bit ply quits with this message:

error: lost 52 eventsply: deactivating
ERR:75

Happy to attach the output of ply -S if you think it's useful.

compile error

make all-recursive
make[1]: 正在进入目录 /home/frank/ply' Making all in include make[2]: 正在进入目录 /home/frank/ply/include'
make[2]: 没有什么可以做的为 all'。 make[2]:正在离开目录 /home/frank/ply/include'
Making all in src
make[2]: 正在进入目录 /home/frank/ply/src' Making all in libply make[3]: 正在进入目录 /home/frank/ply/src/libply'
make all-am
make[4]: 正在进入目录 /home/frank/ply/src/libply' CC arch/libply_la-aarch64.lo CC aux/libply_la-kallsyms.lo CC aux/libply_la-perf_event.lo CC aux/libply_la-printxf.lo CC aux/libply_la-syscall.lo CC aux/libply_la-utils.lo CC built-in/libply_la-aggregation.lo CC built-in/libply_la-built-in.lo CC built-in/libply_la-buffer.lo CC built-in/libply_la-flow.lo CC built-in/libply_la-math.lo In file included from /home/frank/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/aarch64-linux-gnu/libc/usr/include/linux/bpf.h:11:0, from ../../include/ply/ir.h:14, from ../../include/ply/sym.h:13, from ../../include/ply/ply.h:12, from built-in/math.c:11: built-in/math.c: In function 'relop_ir_post': built-in/math.c:253:32: error: 'BPF_JLT' undeclared (first use in this function); did you mean 'BPF_JGT'? jmp.code |= BPF_OP(unsignd ? BPF_JLT : BPF_JSLT); ^ built-in/math.c:253:32: note: each undeclared identifier is reported only once for each function it appears in built-in/math.c:253:42: error: 'BPF_JSLT' undeclared (first use in this function); did you mean 'BPF_JLT'? jmp.code |= BPF_OP(unsignd ? BPF_JLT : BPF_JSLT); ^ built-in/math.c:257:42: error: 'BPF_JSLE' undeclared (first use in this function); did you mean 'BPF_JSLT'? jmp.code |= BPF_OP(unsignd ? BPF_JLE : BPF_JSLE); ^ make[4]: *** [built-in/libply_la-math.lo] 错误 1 make[4]:正在离开目录 /home/frank/ply/src/libply'
make[3]: *** [all] 错误 2
make[3]:正在离开目录 /home/frank/ply/src/libply' make[2]: *** [all-recursive] 错误 1 make[2]:正在离开目录 /home/frank/ply/src'
make[1]: *** [all-recursive] 错误 1
make[1]:正在离开目录 `/home/frank/ply'
make: *** [all] 错误 2

feature request: add module name to backtrace

Hi,
Is it possible to print the module name next to the backtrace ?
If it is built-in to the kernel you can leave it blank, but if it is
part of a module, can you please include the module name in the backtrace ?
eg. input_function+234 [mymodule]

Thanks.

Problem with strcmp in some cases

Was trying to do some syscall counter today and noticed that in one of my cases the strcmp function in the filter section did not work. Consider the following failing example

kprobe:sys_* / strcmp(comm, "ply") /
{
    @[caller] = count();
}

However, the following example works perfectly well where I changed the map to comm instead of caller.

kprobe:sys_* / strcmp(comm, "ply") /
{
    @[comm] = count();
}

Also, removing the strcmp filter works without any problems even when using caller in the map, thus turning my suspission towards the strcmp function.

question about cross compile

I try to do cross compiling for ply like below, but it seems that it doesn't take effect.

export PATH=$PATH:/home/jgsun/repo/buildroot-arm64/output/host/opt/ext-toolchain/bin
export ARCH=arm64
export CROSS_COMPILE=aarch64-none-linux-gnu-
make
file src/ply/ply-ply.o
src/ply/ply-ply.o: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), with debug_info, not stripped

help writing a program using ply

Hi,
I am able to trigger a print on a kprobe entering into a function.
At an offset into one of the arguments to a function there is a Mac-address.
How do I trigger a print only if the Mac-address matches a certain value ?
eg. if ((arg2+33)[0] == 0xab && (arg2+33)[1] == 0xcd )
printf("%v\n", stack());

Thanks.

ply one-liners

Doesn't seem that ply can be used at the command line yet. Eg, something like:

ply -e 'kprobe:SyS_execve { printf("running: %s", arg(1)); }'

Segmentation fault in loc_assign_pre

Running ply compiled from f66d77d, the following script produces a segv:

kprobe:SyS_read
{
        $sizes.quantize(arg(2))
}

When loc_assign is called on n->dyn.call.func being NULL.

Kernel: Linux 4.9.0-1-amd64 #1 SMP Debian 4.9.2-2 (2017-01-12) x86_64

Backtrace:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  loc_assign_pre (n=0x55d017614370, _probe=<optimized out>) at annotate.c:103
103			return n->dyn.call.func->loc_assign(n);
(gdb) bt
#0  loc_assign_pre (n=0x55d017614370, _probe=<optimized out>) at annotate.c:103
#1  0x000055d0156c7900 in node_walk (n=0x55d017614370, pre=0x55d0156cad60 <loc_assign_pre>, post=0x0, ctx=0x55d01761a690) at lang/ast.c:583
#2  0x000055d0156c7b26 in _node_walk_list (head=<optimized out>, pre=pre@entry=0x55d0156cad60 <loc_assign_pre>, post=post@entry=0x0, ctx=ctx@entry=0x55d01761a690) at lang/ast.c:567
#3  0x000055d0156c7955 in node_walk (n=0x55d017619670, pre=0x55d0156cad60 <loc_assign_pre>, post=0x0, ctx=0x55d01761a690) at lang/ast.c:631
#4  0x000055d0156c7995 in node_walk (n=0x55d017619910, pre=0x55d0156cad60 <loc_assign_pre>, post=0x0, ctx=0x55d01761a690) at lang/ast.c:627
#5  0x000055d0156c79b5 in node_walk (n=0x55d01761a3d0, pre=0x55d0156cad60 <loc_assign_pre>, post=0x0, ctx=0x55d01761a690) at lang/ast.c:603
#6  0x000055d0156c7b26 in _node_walk_list (head=<optimized out>, pre=pre@entry=0x55d0156cad60 <loc_assign_pre>, post=post@entry=0x0, ctx=ctx@entry=0x55d01761a690) at lang/ast.c:567
#7  0x000055d0156c7955 in node_walk (n=0x55d01761a690, pre=0x55d0156cad60 <loc_assign_pre>, post=0x0, ctx=0x55d01761a690) at lang/ast.c:631
#8  0x000055d0156cb478 in loc_assign (script=<optimized out>) at annotate.c:180
#9  annotate_script (script=<optimized out>) at annotate.c:433
#10 0x000055d0156c40bc in main (argc=<optimized out>, argv=<optimized out>) at ply.c:212

Ply-2.0.0 missing support for targets other than x86-64

Missing files aarch64.c, arm.c and powerpc.c under lib/arch. These exists on the master branch but are missing in the release 2.0.0

How

ply in QEMU: error: unable to create map 'stdbuf', errno:38

Hi Sir

I tried to run ply in QEMU aarch64, but I met failure error: unable to create map 'stdbuf', errno:38, would you please help with it?

root@/lib# uname -a
Linux qemu-aarch64 5.6.11 #34 SMP Thu Oct 8 20:01:26 CST 2020 aarch64 GNU/Linux

root@/lib# cat opensoop.ply
#!/usr/bin/env ply

kprobe:do_sys_open
{
printf("%v(%v): %s\n",
comm, pid, str(arg1));
}

root@/lib# ./opensoop.ply
info: creating kallsyms cache
error: unable to create map 'stdbuf', errno:38
ERR:-38
root@/lib# ply -h
ply - Dynamic tracing utility

Usage:
ply [options]
ply [options]

Options:
-c COMMAND Run COMMAND in a shell, exit upon completion.
-d Enable debug output.
-e Exit after compiling.
-h Print usage message and exit.
-k Keep going in face of trace buffer overruns.
-S Show generated BPF.
-v Print version information.

root@/lib# ply -v
ply 7ce38a7e73-dirty (linux-version:267277~4.20.13)

debugfs needs to be mounted before starting ply

When starting ply without a debugfs mounted at /sys/kernel/debug there's an error:

$ ply <file>
info: creating kallsyms cache
ERR:-2

My proposal is to check for a mounted debugfs and automatically mount it if not present. I can also implement this.

print with 12 characters on arm32 not working

When using print with 12 charachters, such as "hello world!", produces bad output on an arm 32 bit processor.

# ply 'k:sys_epoll_wait { print("hello world!"); exit(0); 
}'
ply: active

000:  68 65 6c 6c 6f 20 77 6f  72 6c 64 21 69 ac 01 00   hello wo rld!i...

ply: deactivating

Also adding another 8, 16, 24 ... characters and so on creates the same problem.

errno 22 on kprobe loading but the ply script runs!

This is on a Redhat Enterprise 64 bit system. It is strange that I can run my ply scripts but when specifying the -d option it fails to run the script and the syscall to bpf returns -1 with errno set to 22.

# uname -a
3.10.0-1160.6.1.el7.x86_64 #1 SMP Tue Nov 10 17:23:25 PST 2020 x86_64 x86_64 x86_64 GNU/Linux

# /usr/local/sbin/ply -T
Verifying kernel config (/boot/config-3.10.0-1160.6.1.el7.x86_64)... OK
Ensuring that debugfs is mounted... OK
Verifying kprobe... OK
Verifying tracepoint... OK

# /usr/local/sbin/ply -v
ply 2.1.1-14-ge25c913 (linux-version:199168~3.10.0)

The error message and bpf instructions:

# /usr/local/sbin/ply 'kprobe:SyS_read { @["cnt"] = count();}' -dS


-- globals
unsigned long @{struct :anon_0x17455f0}
kprobe:SyS_read
{}void
    @=void
        []unsigned long
            @unsigned long{struct :anon_0x17455f0}
            :structstruct :anon_0x17455f0
                "cnt"char[8]
        countunsigned long


-- locals
-- ir
;; >pre  {}()
;; >pre  @=()
;; >pre  []()
;; >pre  @()
;; >post @()
;; >pre  :struct()
;; >pre  "cnt"
;; >post "cnt"
  0	stw	[bp - 0x8], #0x746e63
  1	stw	[bp - 0x4], #0x0
;; >post :struct()
;; >post []()
  2	ldmap	r1, @
  3	ldw	r0, #0x0
  4	movq	r2, bp
  5	addq	r2, #-0x8
  6	call	map_lookup_elem
  7	jeq	r0, #0x0, +6
  8	movq	r1, bp
  9	addq	r1, #-0x10
 10	movq	r2, #0x8
 11	movq	r3, r0
 12	call	probe_read
 13	ja	+1
L1:
 14	stq	[bp - 0x10], #0x0
L2:
;; >pre  count()
;; >post count()
 15	ldq	r0, [bp - 0x10]
 16	addq	r0, #0x1
 17	stq	[bp - 0x10], r0
;; >post @=()
 18	ldmap	r1, @
 19	ldw	r0, #0x0
 20	movq	r2, bp
 21	addq	r2, #-0x8
 22	movq	r3, bp
 23	addq	r3, #-0x10
 24	movq	r4, #0x0
 25	call	map_update_elem
;; >post {}()
 26	exit
error: unable to load kprobe:SyS_read, errno:22
warning: was ply built against the running kernel?
ERR:-22

Despite the error, the ply script runs when -d is not provided!!

# /usr/local/sbin/ply 'kprobe:SyS_read { @["cnt"] = count();}'
ply: active
^Cply: deactivating

@:
{ cnt     }: 460

I have attached a debugger and see that in syscall.c:int bpf_prog_load() the syscall to
return syscall(__NR_bpf, BPF_PROG_LOAD, &attr, sizeof(attr));
returns -1 with errno = 22. According to man of bpf this is EINVAL which is due to bad bpf program:

"EINVAL For BPF_PROG_LOAD, indicates an attempt to load an invalid
program. eBPF programs can be deemed invalid due to
unrecognized instructions, the use of reserved fields,
jumps out of range, infinite loops or calls of unknown
functions."
I could not find such a problem in the dumped bpf program.

Any ideas?
Is there a way to get more details form the kernel? The contents of vlog where an empty string despite log_level = 1.

Also for reference here is the output of the following basic kprobe:

# /usr/local/sbin/ply 'kprobe:SyS_read { exit(0);}' -dS


-- globals
int stdbuf{u32}
kprobe:SyS_read
{}void
    bwritevoid
        ctxvoid __bpf *
        stdbufint{u32}
        :structstruct :anon_0x1fa85d0
            0u64
            :structstruct :anon_0x1fa8510
                0int


-- locals
void __bpf *ctx
-- ir
  0	movq	r2, r1
;; >pre  {}()
;; >pre  bwrite()
;; >pre  ctx()
;; >post ctx()
;; >pre  stdbuf()
;; >post stdbuf()
;; >pre  :struct()
  1	stw	[bp - 0x4], #0x0
;; >pre  <0>
;; >post <0>
;; >pre  :struct()
;; >pre  <0>
;; >post <0>
;; >post :struct()
  2	stw	[bp - 0x8], #0x0
;; >post :struct()
  3	stq	[bp - 0x10], #0x0
;; >post bwrite()
  4	movq	r1, r2
  5	ldmap	r2, stdbuf
  6	ldw	r0, #0x0
  7	movw	r3, #-0x1
  8	movq	r4, bp
  9	addq	r4, #-0x10
 10	movq	r5, #0x10
 11	call	perf_event_output
;; >post {}()
 12	exit
error: unable to load kprobe:SyS_read, errno:22
warning: was ply built against the running kernel?
ERR:-22

build ply for arm64 target

Hi,
I'm trying to build ply for arm64 target but somehow I'm unable to do it:
I used configure like this:
./configure --target=arm64 --with-kerneldir=/home/kedaru/proj/base/linux/

But I still get x86_64 ply file after a make. How do I compile ply executable for arm64 target ?

Thanks.

YACC build error in ubuntu 18.04

hi, I cloned the code and met below error when building ply in ubuntu 18.04:

2173 ./autogen.sh
2174 ./configure
2175 make

make  all-recursive
make[1]: Entering directory '/home/jinboli/ply/ply2/ply'
Making all in include
make[2]: Entering directory '/home/jinboli/ply/ply2/ply/include'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/jinboli/ply/ply2/ply/include'
Making all in src
make[2]: Entering directory '/home/jinboli/ply/ply2/ply/src'
Making all in libply
make[3]: Entering directory '/home/jinboli/ply/ply2/ply/src/libply'
  YACC     grammar.c
Usage: byacc [options] filename

Options:
  -b file_prefix        set filename prefix (default "y.")
  -B                    create a backtracking parser
  -d                    write definitions (.tab.h)
  -i                    write interface (y.tab.i)
  -g                    write a graphical description
  -l                    suppress #line directives
  -L                    enable position processing, e.g., "%locations"
  -o output_file        (default ".tab.c")
  -p symbol_prefix      set symbol prefix (default "yy")
  -P                    create a reentrant parser, e.g., "%pure-parser"
  -r                    produce separate code and table files (y.code.c)
  -s                    suppress #define's for quoted names in %token lines
  -t                    add debugging support
  -v                    write description (y.output)
  -V                    show version information and exit
Makefile:817: recipe for target 'grammar.c' failed
make[3]: *** [grammar.c] Error 1
make[3]: Leaving directory '/home/jinboli/ply/ply2/ply/src/libply'
Makefile:355: recipe for target 'all-recursive' failed
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory '/home/jinboli/ply/ply2/ply/src'
Makefile:456: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/home/jinboli/ply/ply2/ply'
Makefile:367: recipe for target 'all' failed
make: *** [all] Error 2

The yacc version is:
$yacc -V
yacc - 1.9 20140715

Could anyone give me some help? I'm new to ply. thanks.

ERR:75 & error: lost 36 events

test case:

cat syscalls_sys_enter_close.ply

#!/usr/bin/env ply

tracepoint:syscalls/sys_enter_close
{
printf(" fd:%v\n", data->fd);
}

the last output is like this:

error: lost 36 events
error: lost 36 events
ERR:75

i read the buffer_loop in buffer.c, i guess "poll "has Inferior performance than "epoll". or the buffer overflows. or because of perf 's cost(why not ftrace?)

buffer_q_drain:

(gdb) p *(q->mem)
$29 = {version = 0, compat_version = 0, lock = 4, index = 0, offset = 0, time_enabled = 2078963, time_running = 2078963, {capabilities = 2, {
cap_bit0 = 0, cap_bit0_is_deprecated = 1, cap_user_rdpmc = 0, cap_user_time = 0, cap_user_time_zero = 0, cap_____res = 0}}, pmc_width = 48,
time_shift = 0, time_mult = 0, time_offset = 0, time_zero = 0, size = 96, __reserved = '\000' <repeats 947 times>, data_head = 8152,
data_tail = 4064, data_offset = 4096, data_size = 4096, aux_head = 0, aux_tail = 0, aux_offset = 0, aux_size = 0}

Does the "field:__data_loc " supported?

test case:
tracepoint：irq/irq_handler_entry

Symptom：it Cann't print data->name because data->name's type is "field:__data_loc char[] name; offset:12; size:4; signed:0".

name: irq_handler_entry
ID: 52
format:
field:unsigned short common_type; offset:0; size:2; signed:0;
field:unsigned char common_flags; offset:2; size:1; signed:0;
field:unsigned char common_preempt_count; offset:3; size:1; signed:0;
field:int common_pid; offset:4; size:4; signed:1;

field:int irq;	offset:8;	size:4;	signed:1;
field:__data_loc char[] name;	offset:12;	size:4;	signed:0;

print fmt: "irq=%d name=%s", REC->irq, __get_str(name).

segmentation fault in if statement

As one example, I want to print a header the first time I enter a probe. In order to do that I check if a map h[0] is set and if it is not i should print the header information. This works if I use "if (h[0])" but not the other way around, i.e. using "if (!h[0])" instead. This gives segmentation fault. My example is shown below

kretprobe:vfs_read
{
        if (!h[0])
                printf("%-18s %6s %6s %16s %10s\n", "Time",  "PID", "UID", "COMM", "BYTES");

        printf("%v %6d %6d %16s %10d\n", walltime, pid, uid, comm, retval);
        h[0] = 1;
}