doytsujin / bpftrace-1

High-level tracing language for Linux eBPF - development moved to https://github.com/iovisor/bpftrace

New repository https://github.com/iovisor/bpftrace

Development of BPFtrace has moved to https://github.com/iovisor/bpftrace

This repository will no longer be updated.

BPFtrace

BPFtrace is a high-level tracing language for Linux enhanced Berkeley Packet Filter (eBPF) available in recent Linux kernels (4.x). BPFtrace uses LLVM as a backend to compile scripts to BPF-bytecode and makes use of BCC for interacting with the Linux BPF system, as well as existing Linux tracing capabilities: kernel dynamic tracing (kprobes), user-level dynamic tracing (uprobes), and tracepoints. The BPFtrace language is inspired by awk and C, and predecessor tracers such as DTrace and SystemTap.

For instructions on building BPFtrace, see INSTALL.md

Examples

Count system calls:

kprobe:[Ss]y[Ss]_*
{
  @[func] = count()
}

Attaching 376 probes...
^C

...
@[sys_open]: 579
@[SyS_ioctl]: 686
@[sys_bpf]: 730
@[sys_close]: 779
@[SyS_read]: 825
@[sys_write]: 1031
@[sys_poll]: 1796
@[sys_futex]: 2237
@[sys_recvmsg]: 2634

Produce a histogram of amount of time (in nanoseconds) spent in the read() system call:

kprobe:sys_read
{
  @start[tid] = nsecs;
}

kretprobe:sys_read / @start[tid] /
{
  @times = quantize(nsecs - @start[tid]);
  delete(@start[tid]);
}

Attaching 2 probes...
^C

@start[9134]: 6465933686812

@times:
[0, 1]                 0 |                                                    |
[2, 4)                 0 |                                                    |
[4, 8)                 0 |                                                    |
[8, 16)                0 |                                                    |
[16, 32)               0 |                                                    |
[32, 64)               0 |                                                    |
[64, 128)              0 |                                                    |
[128, 256)             0 |                                                    |
[256, 512)           326 |@                                                   |
[512, 1k)           7715 |@@@@@@@@@@@@@@@@@@@@@@@@@@                          |
[1k, 2k)           15306 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[2k, 4k)             609 |@@                                                  |
[4k, 8k)             611 |@@                                                  |
[8k, 16k)            438 |@                                                   |
[16k, 32k)            59 |                                                    |
[32k, 64k)            36 |                                                    |
[64k, 128k)            5 |                                                    |

Print paths of any files opened along with the name of process which opened them:

kprobe:sys_open
{
  printf("%s: %s\n", comm, str(arg0))
}

Attaching 1 probe...
git: .git/objects/70
git: .git/objects/pack
git: .git/objects/da
git: .git/objects/pack
git: /etc/localtime
systemd-journal: /var/log/journal/72d0774c88dc4943ae3d34ac356125dd
DNS Res~ver #15: /etc/hosts
DNS Res~ver #16: /etc/hosts
DNS Res~ver #15: /etc/hosts
^C

Whole system profiling (TODO make example check if kernel is on-cpu before recording):

profile:hz:99
{
  @[stack] = count()
}

Attaching 1 probe...
^C

...
@[
_raw_spin_unlock_irq+23
finish_task_switch+117
__schedule+574
schedule_idle+44
do_idle+333
cpu_startup_entry+113
start_secondary+344
verify_cpu+0
]: 83
@[
queue_work_on+41
tty_flip_buffer_push+43
pty_write+83
n_tty_write+434
tty_write+444
__vfs_write+55
vfs_write+177
sys_write+85
entry_SYSCALL_64_fastpath+26
]: 97
@[
cpuidle_enter_state+299
cpuidle_enter+23
call_cpuidle+35
do_idle+394
cpu_startup_entry+113
rest_init+132
start_kernel+1083
x86_64_start_reservations+41
x86_64_start_kernel+323
verify_cpu+0
]: 150

Probe types

kprobes

Attach a BPFtrace script to a kernel function, to be executed when that function is called:

kprobe:sys_read { ... }

uprobes

Attach script to a userland function:

uprobe:/bin/bash:readline { ... }

tracepoints

Attach script to a statically defined tracepoint in the kernel:

tracepoint:sched:sched_switch { ... }

Tracepoints are guaranteed to be stable between kernel versions, unlike kprobes.

timers

Run the script at specified time intervals:

profile:hz:99 { ... }

profile:s:1 { ... }

profile:ms:20 { ... }

profile:us:1500 { ... }

Multiple attachment points

A single probe can be attached to multiple events:

kprobe:sys_read,kprobe:sys_write { ... }

Wildcards

Some probe types allow wildcards to be used when attaching a probe:

kprobe:SyS_* { ... }

Predicates

Define conditions for which a probe should be executed:

kprobe:sys_open / uid == 0 / { ... }

Builtins

The following variables and functions are available for use in bpftrace scripts:

Variables:

• pid - Process ID (kernel tgid)
• tid - Thread ID (kernel pid)
• uid - User ID
• gid - Group ID
• nsecs - Nanosecond timestamp
• cpu - Processor ID
• comm - Process name
• stack - Kernel stack trace
• ustack - User stack trace
• arg0, arg1, ... etc. - Arguments to the function being traced
• retval - Return value from function being traced
• func - Name of the function currently being traced

Functions:

• quantize(int n) - Produce a log2 histogram of values of n
• count() - Count the number of times this function is called
• delete(@x) - Delete the map element passed in as an argument
• str(char *s) - Returns the string pointed to by s
• printf(char *fmt, ...) - Write to stdout
• sym(void *p) - Resolve kernel address
• usym(void *p) - Resolve user space address (incomplete)
• reg(char *name) - Returns the value stored in the named register