Docs: clarify the kind of error about libipt

[markus-metzger]

This would be an error in the trace. TIPs are not deferred for uncompressed returns so we must either get a taken TNT or a TIP.

Can you decode that trace with libipt's ptxed?

[vext01]

TIPs are not deferred for uncompressed returns so we must either get a taken TNT or a TIP.

That's what I thought, yeah.

This would be an error in the trace

Is that known to happen on some CPUs? I did read the erratas in your header to see if I might be being bitten, but nothing seemed relevant.

I've seen this kind of thing on:

• Intel(R) Xeon(R) CPU E3-1240 v6 @ 3.70GHz
• Intel(R) Xeon(R) Gold 6254 CPU @ 3.10GHz

Can you decode that trace with libipt's ptxed?

That's a good idea. If the trace can be decoded with that, then it's an issue with my decoder (which I'm not ruling out at this point!)

Will report back. Thanks.

[vext01]

Hi Markus,

Took a little while to instrument our system and get all the info out, but I did it!

I have a ptxed invocation that looks like this:

~/source/libipt/build/bin/ptxed \
    --pt trace_payload \
    --raw executable_binary:0x00200000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMDebugInfoPDB.so.16git:0x7f2637d76000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMTextAPI.so.16git:0x7f2637e3a000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMCoroutines.so.16git:0x7f2637e6c000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMDebugInfoDWARF.so.16git:0x7f2637eec000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMDebugInfoCodeView.so.16git:0x7f2637fc6000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMInstCombine.so.16git:0x7f2638070000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMVectorize.so.16git:0x7f263822f000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMSymbolize.so.16git:0x7f263847e000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMObjCARCOpts.so.16git:0x7f26384b5000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMDemangle.so.16git:0x7f26384f8000 \
    --raw /usr/lib/x86_64-linux-gnu/libtinfo.so.6.4:0x7f263854b000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMAsmParser.so.16git:0x7f2638589000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMScalarOpts.so.16git:0x7f263865a000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMFrontendOpenMP.so.16git:0x7f2638af0000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMBitReader.so.16git:0x7f2638b73000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMAnalysis.so.16git:0x7f2638cb3000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMObject.so.16git:0x7f26392b3000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMCodeGen.so.16git:0x7f26394df000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMBitstreamReader.so.16git:0x7f2639d64000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMYkPasses.so.16git:0x7f2639d77000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMBitWriter.so.16git:0x7f2639d93000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMSelectionDAG.so.16git:0x7f2639e3e000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMOption.so.16git:0x7f263a1ed000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMDebugInfoMSF.so.16git:0x7f263a204000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMPasses.so.16git:0x7f263a267000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMGlobalISel.so.16git:0x7f263a3eb000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMInstrumentation.so.16git:0x7f263a54b000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMAsmPrinter.so.16git:0x7f263a6d6000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMSupport.so.16git:0x7f263a85b000 \
    --raw /usr/lib/x86_64-linux-gnu/libz.so.1.2.13:0x7f263aa76000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMRemarks.so.16git:0x7f263aa9c000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMMC.so.16git:0x7f263aafb000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMCore.so.16git:0x7f263acc2000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMIRPrinter.so.16git:0x7f263b11b000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMAggressiveInstCombine.so.16git:0x7f263b124000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMJITLink.so.16git:0x7f263b156000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMTransformUtils.so.16git:0x7f263b254000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMProfileData.so.16git:0x7f263b55f000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMipo.so.16git:0x7f263b666000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMYkIR.so.16git:0x7f263b9f9000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMOrcJIT.so.16git:0x7f263ba41000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMLinker.so.16git:0x7f263bc10000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMOrcShared.so.16git:0x7f263bc40000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMOrcTargetProcess.so.16git:0x7f263bc50000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMBinaryFormat.so.16git:0x7f263bc76000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMMCParser.so.16git:0x7f263bca8000 \
    --raw /usr/lib/x86_64-linux-gnu/libm.so.6:0x7f263bd31000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMX86Desc.so.16git:0x7f263bf58000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMWindowsDriver.so.16git:0x7f263c1f1000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMMCDisassembler.so.16git:0x7f263c1fa000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMX86CodeGen.so.16git:0x7f263c267000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMTargetParser.so.16git:0x7f263c7ea000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMTarget.so.16git:0x7f263c81d000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMCFGuard.so.16git:0x7f263c82d000 \
    --raw /usr/lib/x86_64-linux-gnu/libgcc_s.so.1:0x7f263c837000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMIRReader.so.16git:0x7f263c856000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMRuntimeDyld.so.16git:0x7f263c87f000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMExecutionEngine.so.16git:0x7f263c913000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMMCJIT.so.16git:0x7f263c936000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMX86AsmParser.so.16git:0x7f263c947000 \
    --raw /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30:0x7f263ca99000 \
    --raw /usr/lib/x86_64-linux-gnu/libc.so.6:0x7f263cc45000 \
    --raw /home/vext01/research/yk/target/debug/deps/libykcapi.so:0x7f263cf5d000 \
    --raw /home/vext01/research/ykllvm/build/lib/libLLVMX86Info.so.16git:0x7f263d276000 \
    --raw /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2:0x7f263d289000

# didn't pass the vdso for now, hoping I won't need it.
#--raw [vdso]:0x7ffce3761000

I generated the --raw arguments by parsing the /proc/<PID>/maps file for the process in question.

When I run this, I get:

[enabled]
[exec mode: 64-bit]
00007f263cd1cafb 16 05 00 00 48 83 bd 10 fb ff ff 00 0f 85 33 [xed decode error: (2) GENERAL_ERROR]
[29, 7f263cd1cafb: reconstruct error: decoder out of sync]
00007f263cd1cb1a  mov rbx, qword ptr [rbp-0x4f8]
00007f263cd1cb21  mov rax, qword ptr [r14]
00007f263cd1cb24  add rax, qword ptr [r14+0x10]
00007f263cd1cb28  cmp rbx, rax
00007f263cd1cb2b  jnb 0x7f263cd1d020
00007f263cd1cb31  mov r12, rbx
00007f263cd1cb34  mov ebx, dword ptr [rbp-0x4b8]
00007f263cd1cb3a  jmp 0x7f263cd1cb54
00007f263cd1cb54  mov rax, qword ptr [r14+0x8]
00007f263cd1cb58  mov rdx, r14
00007f263cd1cb5b  mov esi, ebx
00007f263cd1cb5d  lea r13, ptr [r12*8]
00007f263cd1cb65  mov rdi, qword ptr [rax+r12*8]
00007f263cd1cb69  call 0x7f263cd1bac0
00007f263cd1bac0  sub rsp, 0xa8
00007f263cd1bac7  mov rax, qword ptr fs:[0x28]
00007f263cd1bad0  mov qword ptr [rsp+0x98], rax
00007f263cd1bad8  xor eax, eax
00007f263cd1bada  and esi, 0x200
00007f263cd1bae0  jnz 0x7f263cd1bb20
[29, 7f263cd1bae0: error: trace stream does not match query]

Which doesn't look good. The 0x16 (10110) looks like it could be a short TNT packet? Does it fail to decode this and then the decoder goes out of sync?

ptdump has no problems decoding the raw packets:

0000000000000000  02820282028202820282028202820282  psb
0000000000000010  02032500                          cbr        25
0000000000000014  0223                              psbend
0000000000000016  00                                pad
0000000000000017  00                                pad
0000000000000018  00                                pad
0000000000000019  00                                pad
000000000000001a  00                                pad
000000000000001b  00                                pad
000000000000001c  00                                pad
000000000000001d  9901                              mode.exec  cs.l
000000000000001f  71fbcad13c267f                    tip.pge    3: 00007f263cd1cafb
0000000000000026  00                                pad
0000000000000027  00                                pad
0000000000000028  08                                tnt.8      ..
0000000000000029  4d34c9023d                        tip        2: ????????3d02c934
000000000000002e  0c                                tnt.8      !.
000000000000002f  00                                pad
0000000000000030  4dc0efca3c                        tip        2: ????????3ccaefc0
0000000000000035  1e                                tnt.8      !!!
...

But I don't see any 0x16 0x05 sequence in the raw payload...

Do you see anything obviously wrong?

Thanks

[markus-metzger]

Those 16 05 00 00 48 83 bd 10 fb ff ff 00 0f 85 33 are instruction bytes. Since the error is at the very beginning, I suspect an issue with the memory mapping. Looking at the ptxed arguments, this would be /usr/lib/x86_64-linux-gnu/libc.so.6:0x7f263cc45000.

Unless you only want to decode right after recording, it is advisable to make copies of all files that were used in the trace and decode from those copies. The original files may change when updating the system between recording and decoding.

[vext01]

Hi Markus,

I got it working. I wasn't passing the file offsets of the executable segments to ptxed, so the code it saw was total nonsense.

The trace now decodes just fine with ptxed, so the bugs are on us :)

Thanks again for your help. I'm going to close this.

(I wrote a little tool to help automate the job of copying all of the various artifacts into a directory for future reference, or for sending to other people for debugging purposes: https://github.com/vext01/pt-standalone)

P.S. do you think it would be worthwhile me starting somewhere for people to discuss PT (and maybe other hardware tracers) like a discord/mattermost/irc channel or something? Would you see yourself using such a resource?