oracle / dtrace-utils

DTrace-utils contains the DTrace port to Linux

Linux DTrace

This is the official Linux port of the DTrace tracing tool.

The source is posted here on github.com in the hope that it increases the visibility for our work and to make it even easier for people to access the source. We also use this repository to work with developers in the Linux community.

The main development branch is devel.

We provide prebuilt x86_64 and aarch64 DTrace userspace packages for Oracle Linux 9 (UEK7 kernel), Oracle Linux 8 (UEK7 or UEK6 kernel), and Oracle Linux 7 (UEK6 kernel). These packages are based on the Oracle Linux specific release branch 2.0-branch.

The latest packages can be obtained from the following Oracle Linux yum server repositories:

• Oracle Linux 9 (x86_64) UEK Release 7: https://yum.oracle.com/repo/OracleLinux/OL9/UEKR7/x86_64/
• Oracle Linux 9 (aarch64) BaseOS Latest: https://yum.oracle.com/repo/OracleLinux/OL9/baseos/latest/aarch64/
• Oracle Linux 8 (x86_64) UEK Release 7: https://yum.oracle.com/repo/OracleLinux/OL8/UEKR7/x86_64/
• Oracle Linux 8 (aarch64) UEK Release 7: https://yum.oracle.com/repo/OracleLinux/OL8/UEKR7/aarch64/
• Oracle Linux 8 (x86_64) UEK Release 6: https://yum.oracle.com/repo/OracleLinux/OL8/UEKR6/x86_64/
• Oracle Linux 8 (aarch64) BaseOS Latest: https://yum.oracle.com/repo/OracleLinux/OL8/baseos/latest/aarch64/
• Oracle Linux 7 (x86_64) UEK Release 6: https://yum.oracle.com/repo/OracleLinux/OL7/UEKR6/x86_64/
• Oracle Linux 7 (aarch64) UEK Release 6: https://yum.oracle.com/repo/OracleLinux/OL7/UEKR6/aarch64/

Source code for the UEK kernel is available on github in the linux-uek repo.

Table of Contents

• 1. License
• 2. How to Build DTrace
  • 2.1. Prerequisites
    • 2.1.1. Verify that binutils is recent enough...
    • 2.1.2. ... or install a recent-enough binutils
    • 2.1.3. Install other necessary packages
  • 2.2. Build DTrace
• 3. Testing
• 4. How to Run DTrace
• 5. Questions
• 6. Pull Requests and Support
• 7. Building Previous Releases of DTrace
  • 7.1. Prerequisites
  • 7.2. Build a kernel with add-on features that DTrace uses
  • 7.3. Build DTrace

1. License

DTrace is licensed under the UPL 1.0 (Universal Permissive License). A copy is included in this repository as the LICENSE file.

2. How to Build DTrace

The build instructions focus on building DTrace for the upstream Linux kernel. We will use the 5.14.9 release, but the instructions apply to most recent Linux kernel versions.

2.1. Prerequisites

Please read this section carefully before moving over to the build documentation to ensure your environment is properly configured.

2.1.1. Verify that binutils is recent enough...

DTrace uses a type introspection system called CTF. This is supported by upstream GCC and GNU Binutils. Make sure you have binutils 2.36 or later installed.

If your distro provides libctf.so in a binutils development package, you need to install that too.

2.1.2. or install a recent-enough binutils

If your distro provides binutils 2.36 or later, you should install it. If not, you can build your own local copy (which can be configured with a --prefix specific to itself to avoid disturbing the distro version).

2.1.3. Install other necessary packages

A few other packages are required, either for building or at runtime. They should be part of most Linux distributions today and can be installed via the package manager of your distro. The table below gives package names for Debian and Oracle Linux.

For building:

Prerequisite	Debian	Oracle Linux
glibc headers	libc6-dev	glibc-headers
glibc (static)	(in libc6-dev)	glibc-static
glibc (32-bit dev)	libc6-dev-i386	glibc-devel.i686
bison	bison	bison
flex	flex	flex
BPF gcc	gcc-bpf	gcc-bpf-unknown-none
BPF binutils	binutils-bpf	binutils-bpf-unknown-none
libpcap dev	libpcap-dev	libpcap-devel
wireshark	wireshark	wireshark
valgrind	valgrind	valgrind-devel
fuse3 or fuse	libfuse3-dev or libfuse-dev	fuse3-devel or fuse-devel
kernel headers	linux-headers--	kernel-uek-devel-
	linux-headers--common

At runtime:

Prerequisite	Debian	Oracle Linux
wireshark	wireshark	wireshark
fuse3 or fuse	libfuse3-3 or libfuse2	fuse3-devel or fuse-devel

2.2. Build DTrace

The simplest way of building DTrace is done by issuing the following commands from the DTrace source tree:

make
sudo make install

Some distributions install the BPF gcc and binutils under different names. You can specify the executables to use using the BPFC and BPFLD variables. E.g. on Debian you would use:

make BPFC=bpf-gcc BPFLD=bpf-ld
sudo make install

In this scenario the build system expects that kernel sources can be found at /lib/modules/$(uname -r)/build. It is also expected that the kernel used for userspace build is compatible with the utils (the exported userspace headers in the include/uapi/linux/dtrace are compatible with the utils version being built).

You can point at a different kernel version with the KERNELS variable, e.g.

make KERNELS="5.16.8"

as long as the source tree that kernel was built with remains where it was when that kernel was installed.

See the GNUmakefile for more options (building translators against multiple different kernels at once, building against kernel sources found in different places, building against a kernel built with O=, etc.)

'make help' might also be of interest.

3. Testing

A testsuite is provided in the source tree. It is run using make in conjunction with one of the following targets:

target	description
check-verbose	Tests binaries from build directory. Prints result of every tests.
check-installed-verbose	Tests binaries installed in the system. Prints result of every tests.

Example:

sudo make check-verbose

Logs from test runs are stored in test/log/<run_number>/. The most recent logs are also available by via the test/log/current/ symbolic link.

The testsuite itself has more dependencies that need to be installed. The full list is available in the dtrace.spec file.

4. How to run DTrace

The dtrace binary is installed in /usr/sbin/dtrace. Currently, you can only run dtrace with root privileges. Be careful not to confuse the utility with SystemTap's dtrace script, which is installed in /usr/bin/dtrace.

5. Questions

For questions, please ask on the dtrace mailing list. Older discussions are archived here.

We have a #linux-dtrace IRC channel on irc.libera.chat, you can ask questions there as well.

6. Pull Requests and Support

We currently do not accept pull requests via GitHub, please contact us via the mailing list or IRC channel listed above.

The source code for DTrace is published here without support. Compiled binaries are provided as part of Oracle Linux, which is free to download, distribute and use. Support for DTrace is included in Oracle Linux support subscriptions. Individual packages and updates are available on the Oracle Linux yum server.

7. Building Previous Releases of DTrace

For versions of DTrace prior to 2.0.0-1.13.2 a small number of kernel patches patches are needed, which are here. Branches named starting v2/* are suitable.

To build such versions of dtrace follow the steps below.

7.1. Prerequisites

See above.

At this point, you will need an additional step:

7.2. Build a kernel with add-on features that DTrace uses

DTrace 2.0.0-1.13.1 and earlier depend on a few extra kernel features that are not available in the upstream kernel:

• CTF type information extraction
• /proc/kallmodsyms
• New system call: waitfd()

As noted above, patches that implement these features are available from the v2/* branches in our Linux kernel repository for features that DTrace uses: https://github.com/oracle/dtrace-linux-kernel

The customary way to obtain these patches is to clone an appropriate upstream kernel version (5.8.1 or later) and to merge our updated tree into it. The patch set is quite small, but may need some tweaking if it is being merged into a newer tree. We occasionally push a new patched tree to our github repo, so do look around the repo to see if there is a branch with better match.

Oracle Linux's UEK7 provides a simple alternative if you don't want to patch your own kernel.

Building of a patched kernel from sources is also straightforward. Please consult Documentation/process/changes.rst in the kernel source tree to ensure all required dependencies are installed first. The following steps should build and install the kernel.

# Start with your preferred .config options.  Features DTrace needs should
# enable themselves automatically.
make olddefconfig
make

# This step will produce vmlinux.ctfa, which holds all CTF data for the kernel
# and its modules.  If it doesn't work, you don't have a toolchain that is
# recent enough.
make ctf

# Install with root privileges.
sudo make INSTALL_MOD_STRIP=1 modules_install
sudo make install

It is preferred (but not required) to reboot into your new kernel before trying to build DTrace. (If you don't reboot into it, you need to specify some extra options when running make.)

7.3. Build DTrace

See above.

dtrace-utils's Issues

strjoin() can store lenghts beyond strsize

The strjoin() implementation is currently simply adding the length of the two component strings and storing that as length for the result. This can cause the stored length to be larger than strsize.

tst.multiagg3.d fails: formatted printa()

invalid user access in action

Using dtrace on a mac with M2 chip, even though I run the command “csrutil enable --without dtrace”, it still reports an error when running the script.

dtrace: error on enabled probe ID 659 (ID 4252: pid39182:libdyld.dylib:dyld_program_sdk_at_least:entry): invalid user access in action #1
dtrace: error on enabled probe ID 674 (ID 4237: pid39182:libdyld.dylib:_dyld_for_each_objc_class:entry): invalid user access in action #1
dtrace: error on enabled probe ID 661 (ID 4250: pid39182:libdyld.dylib:tlv_get_addr:entry): invalid user access in action #1
dtrace: error on enabled probe ID 661 (ID 4250: pid39182:libdyld.dylib:tlv_get_addr:entry): invalid user access in action #1
dtrace: error on enabled probe ID 661 (ID 4250: pid39182:libdyld.dylib:tlv_get_addr:entry): invalid user access in action #1
dtrace: error on enabled probe ID 661 (ID 4250: pid39182:libdyld.dylib:tlv_get_addr:entry): invalid user access in action #1
dtrace: error on enabled probe ID 692 (ID 4219: pid39182:libdyld.dylib:_dyld_get_objc_selector:entry): invalid user access in action #1

Do not report unused aggregations

Aggregations may be described in a D script but never used (e.g., probes using them never fire). In this case, the aggregations should not be reported.

Consider

        BEGIN {
            x = 1234;
            @a = sum(x);
            exit(0)
        }
        tick-7s {
            @b = min(x);
            @c = max(x)
        }

Aggregation @A will have data, but @b and @c will not. Here is the behavior with DTv1:

                    1234

That is, the sum() aggregation is reported while the min() and max() aggregations, having no data, are not. In contrast, here is DTv2:

        DTrace 2.0.0 [Pre-Release with limited functionality]
                        1234
         9223372036854775807
         -9223372036854775808

Here, the min() and max() aggregations are reported as well.

TLS variables (self->x) can be declared with an array datatype but cannot be used as such

dtrace -s /dev/stdin << EOT

self int n[1];
BEGIN { self->n[0] = 1; }
EOT

dtrace: script '/dev/stdin' matched 1 probe
dtrace: error on enabled probe ID 1 (ID 1: dtrace:::BEGIN): invalid address (0x0) in action #1 at DIF offset 24

Aggregations are not appropriately cleared

Consider:

    BEGIN {
        @s = sum(1);
        printa(@s);
        printa(@s);
        printa(@s);
        exit(0)
    }

One sees the anticipated behavior -- "1" printed three times -- with DTv1. In contrast, with DTv2, the output is

                1
                2
                3

Or, consider

    'tick-5 { @s = sum(1); printa(@s); } tick-1100ms { exit(0) }'
    'tick-2s { @s = sum(1); printa(@s); } tick-11s { exit(0) }'

We would expect the aggregation to increase from 1 to 5, with it being printed five times along the way. The difference is the timing, and output can depend on buffering, aggregation rate, and so on. With DTv1, we observe

                5
                5
                5
                5
                5

("fast" firing rate) and

                1
                2
                3
                4
                5

("slow" firing rate) respectively, both reasonable results. In contrast, with DTv2, we get

                1
                3
                6
               10
               15

substr() stores the wrong string length

The result string of a substr() subroutine call can have the wrong length stored in its prefix. This can cause issues with other string operations that the substr() provides arguments for.

Inconsistent behaviour between trace(strtok(...)) and s = strtok(...); trace(s)

Consider the following:

dtrace -n BEGIN'{ x = "a|b|c|"; trace(x); trace(strtok(x, "|")); trace(strtok(NULL, "|")); trace(strtok(NULL, "|")); trace(strtok(NULL, "|")); exit(0); }'
dtrace: description 'BEGIN' matched 1 probe
dtrace: error on enabled probe ID 2 (ID 1: dtrace:::BEGIN): invalid address (0x0) in action #1

vs

dtrace -n BEGIN'{ x = "a|b|c|"; trace(x); trace(strtok(x, "|")); trace(strtok(NULL, "|")); trace(strtok(NULL, "|")); s = strtok(NULL, "|"); trace(s); exit(0); }'
dtrace: description 'BEGIN' matched 1 probe
CPU ID FUNCTION:NAME
9 1 :BEGIN a|b|c| a b c

In the second case, s is output as the empty string while the first example would indicate that the result of the strtok is NULL.

dt_cg_store_var() needs to copy terminating NUL byte for strings

Consider
/*
* Determine the amount of data to be copied. It is
* the lesser of the size of the identifier and the
* size of the data being copied in.
*/
srcsz = dt_node_type_size(dnp->dn_right);
if (dt_node_is_string(dnp))
srcsz += DT_STRLEN_BYTES;
size = MIN(srcsz, size);

"srcsz = dt_node_type_size(dnp->dn_right)" includes the terminating NUL byte for string constants but not for regular string types.

"size = idp->di_size" generally will not include the NUL terminating byte either.

Check and fix this code on both the gvar/lvar and the tvar code paths.

How to convert other types to char* or string with dtrace

In using dtrace, if other types can be converted to character types, then you can easily perform comparison operations, so the logic is much easier to write. But why I didn't find a good way to convert type, usually report error l type conversion failed, such as convert ustack(2) or usym(ucaller) to string type, please ask if there is a good way or suggestion, I will be very grateful!

bpf-helpers.h

I was curious to check this out (and was compiling the devel branch), but when compiling I get

bpf/agg_lqbin.c:7:10: fatal error: bpf-helpers.h: No such file or directory

I had some trouble tracking down where this file is supposed to be coming from,
e.g. libbpf has bpf_helpers.h with a dash, the kernel has a bpf-helper.h non-plural,

I assume perhaps it is supposed to be coming from binutils/sim/bpf, in which case my binutils is too new,
given this file was removed in the commit sim/bpf: desCGENization of the BPF simulator

It would be helpful to know if I am on the right track regarding which dependency should be providing this header.

dtrace: could not enable tracing: BPF program load for '...' failed: No space left on device

A script like
x = 1;
@ = quantize(x);
@ = quantize(x);
[....97 similar lines...]
@ = quantize(x);
fails, printing about 16 Mbyte of BPF verifier log and ending in:
dtrace: could not enable tracing: BPF program load for '...' failed:
No space left on device

RFE: please provide packaging friendly versioning

Current git tag 2.0.0-1.14 does not fulfill proper versioning criteria because it uses in the version tag dash which is forbidden charakter in version string. 🤔

Repo name clearly shows that project/package under name dtrace-utils or just dtrace (like it is in included in this repo dtrace.spec) so it is hard to say what is that 2.0.0. Is trat part the actual version string which suppose to be used?
If yes it should be more like 2.0.0.1.14.

test test

test issue

Need better diagnostics for when dynvarsize < dt_maxtlslen

We allocate space for dynamics variables, such as TLS variables, statically. Users have control over the space with the DTrace option dynvarsize. The size should be picked based on the largest variable size. Statically, we track the largest TLS variable size with dt_maxtlslen.

The size also depends on how many variables must be stored concurrently, however, and this cannot be known statically. Therefore, it is possible that there is insufficient storage and variables must be "dropped" at run time.

Nevertheless, we can provide more helpful diagnostics in at least one case: when dynvarsize < dt_maxtlslen. That is, static analysis tells us that there are TLS variables and yet we will allocate space for none of them.

Consider, for example:

# cat z.d
struct foo {
    long long a1, a2, a3, a4, a5, a6, a7, a8, a9;
};
self struct foo x;
BEGIN {
    self->a = 1;
    exit(0);
}
# dtrace -xdynvarsize=32 -s z.d
dtrace: script 'z.d' matched 1 probe
EUGENE 32 / 72 = 1
dtrace: error on enabled probe ID 2 (ID 1: dtrace:::BEGIN): invalid address (0x0) in action #1

Though the large struct is never used, there is insufficient storage even for a 4-byte int.

One can argue about how likely such scenarios are. The main point, however, is that we can easily generate helpful diagnostics for the case that dynvarsize < dt_maxtlslen.

Unrecognized PERF_RECORD_* causes "dtrace: processing aborted: Success"

When dt_consume_one() encounters an unrecognized hdr->type -- for example, PERF_RECORD_LOST -- it returns DTRACE_WORKSTATUS_ERROR without running dt_set_errno(dtp, errno). Then, dt_consume_cpu() passes DTRACE_WORKSTATUS_ERROR up the call stack through dtrace_consume() and dtrace_work(). In cmd/dtrace.c in main(), the dtrace_work() error return value leads to the dfatal("processing aborted") call. The unset dtp->dt_errno adds the "Success" string. The result is that processing is aborted with only the puzzling explanation that the status is "Success."

Runs of the test suite can produce occasional such failures, which are typically not reproducible upon retrial.

Here is one way to demonstrate the problem:
# dtrace -n 'ksys_write:entry { trace(1); }'
with output directed to the terminal (albeit possibly within a 'script' session to capture output). On my VM, even as many as ten million lines of output might be shown over the course of even a minute, but then the run aborts with the telltale message.

TLS struct variables should be allocated only for nonzero values

For thread-local variables and associative arrays, the documentation says that unassigned variables are considered to be initialized to zero. Storage is not allocated until nonzero values are assigned. Storage is to be freed when zero values are assigned.

These semantics are not well handled for struct variables. Consider

    struct foo {
        int bar, baz;
    } x;

We cannot assign zero to x to free its storage since x=0 is not defined. And we cannot initialize a member x.bar=1 unless x already exists.

These issues are not new to the port of DTrace onto BPF. They existed already in the legacy DTrace implementation on Linux.

2.0.0-1.1: build fails with `__KERNEL__` redefined

[tkloczko@pers-jacek dtrace-utils-2.0.0-1.14]$ make
M4: libdtrace/procfs.d.in
Macro "TASK_RUNNING" has no definition on this system.
libdtrace/procfs.m4:14:9: warning: "__KERNEL__" redefined
   14 | #define __KERNEL__
      |         ^~~~~~~~~~
<command-line>: note: this is the location of the previous definition
Error: Not all tokens in libdtrace/procfs.d.in defined in headers.