`unexpected packet context` in initial sync point about libipt

[tklengyel]

@deyixtan Could you record the processor trace buffer using kfx --ptcov --debug mode instead of using xen-vmtrace? I tried the following on several boxes and the resulting buffer was correct:

echo -n -e '\xCC' > cc
sudo ./kernel-fuzzer-for-xen-project/rwmem --domid <guest domain id> --read <base of __x64_sys_setuid> --file backup --limit 1
xxd backup
sudo ./kernel-fuzzer-for-xen-project/rwmem --domid <guest domain id> --write <base of __x64_sys_setuid> --file cc --limit 1
sudo ./kernel-fuzzer-for-xen-project/kfx --domid <guest domain id> --setup --harness breakpoint --start-byte <byte from backup>
sudo ./forkvm <guest domain id>
sudo ./stepper --domid <fork id> --limit 10
sudo ./forkvm <guest domain id>
sudo ./kernel-fuzzer-for-xen-project/rwmem --domid <fork 2 id> --write <ip shown for step 10 in stepper output> --file cc --limit 1
sudo ./rwmem --domid 1 --list-pages | tail -n1
sudo ./kernel-fuzzer-for-xen-project/kfx --domid <fork 2 id> --json ubuntu.json --address <address from rwmem output> --input /dev/zero --input-limit 1 --harness breakpoint --start-byte <byte from backup> --ptcov --debug

[markus-metzger]

Hello Deyi,

A missing PSBEND is indeed a trace collection issue. What causes this I cannot say. I am not aware of a scenario where PSBEND would not be generated. But it may get overwritten later due to other issues. Does the rest of the trace decode (using ptxed or a similar tool) without error?

The trace seems to consist of multiple chunks and each chunk starts with that same issue. Is this file the concatenation of multiple trace buffers?

There is an erratum [ICL082, TGL052] affecting the processors you listed that may cause very weird-looking traces. This should already be fixed in the OS by forcing ToPA on those processors but it's still worth checking. Would you be able to check whether PT output is configured to ToPA or to single-range when you're seeing those issues? This is configured in IA32_RTIT_CTL[8].

[deyixtan]

Hello Markus, apologies for the late reply.

What do you mean by concatenation of multiple trace buffers?

Anyways, IA32_RTIT_CTL[8] is set to 0, so I believe the PT output was configured to single-range.
Is that the reason why I am seeing such a weird looking trace? Are there other ways I might be able resolve this issue with single-range scheme?

Also may I know where can I find more info about this erratum [ICL082, TGL052]? Thanks!

[markus-metzger]

The trace file contained trace at regular offsets with nothing (i.e. all zeroes) in between. Each section starts with the same issue. This looked to me like multiple traces were concatenated into a single file.

Errata are documented in specification updates; you can find them here: https://www.intel.com/content/www/us/en/products/docs/processors/core/core-technical-resources.html.

Could you try with ToPA instead of single-range?

[deyixtan]

Unfortunately I am not able to try ToPA instead of single-range.
I am currently generating traces for Xen domUs and it currently does not support ToPA configuration.

Based on the erratum workaround, I tried to configure single-range output to be < 4KB, and it is still generating corrupted trace.

For a simpler example, I am trying to trace a __x64_sys_setuid linux syscall, from 0xffffffff810be736 to 0xffffffff810be743.
I've attached a raw trace + dump file that was generated with the corresponding disassembly.

Dump of assembler code for function __x64_sys_setuid:
   0xffffffff810be730 <+0>:	nopl   0x0(%rax,%rax,1)
   0xffffffff810be735 <+5>:	push   %rbp
   0xffffffff810be736 <+6>:	mov    0x70(%rdi),%rdi
   0xffffffff810be73a <+10>:	mov    %rsp,%rbp
   0xffffffff810be73d <+13>:	callq  0xffffffff810be620 <__sys_setuid>
   0xffffffff810be742 <+18>:	pop    %rbp
   0xffffffff810be743 <+19>:	retq   
End of assembler dump.

By any chance this issue stemming from something else?

[markus-metzger]

OK, that rules out this erratum.

The raw trace file is only 1.1KB; I expected 4KB. Do you know if recording wrapped around in the small buffer? If possible, please send the full buffer that was configured to the hardware as-is. Let's see if the PSBEND is there or not.

[deyixtan]

I had configured vmtrace_buf_kb=2048 in my domU config. I am using a modified version of kfx to start/stop VM execution at specific address by configuring a software breakpoint harness. I set up my start and end harness at 0xffffffff810be736 and 0xffffffff810be743 respectively, before I proceed with my VM trace generation. As such, I don't think the recording was big enough such that it was wrapped around in a small buffer.

I have attached a archive file containing the egenerated trace with the ptdump and some commands I used for your reference.

[markus-metzger]

Could you describe how I would reproduce this step by step? The more detailed, the better.

[deyixtan]

The fork of intel/kernel-fuzzer-for-xen-project was set up in a fresh installation of Ubuntu 20.04 by following their respective documentations.

I wanted to generate a simple trace of the setuid syscall. As such, I used KF/x to pause the VM at entry of the syscall with its harness capability. Then I can generate a trace from that entry point onward.

toy.c

#include <ctype.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

unsigned int convert(char *st) {
  char *x;
  for (x = st ; *x ; x++) {
    if (!isdigit(*x))
      return 0L;
  }
  return (strtoul(st, 0L, 10));
}

int main(int argc, char** argv) {
    if (argc != 2) {
        printf("usage: ./setuid <new uid>\n");   
        return 1; 
    }

    int result = setuid(convert(argv[1]));
    printf("Result: %d\nNew UID: %d\n", result, getuid());

    return 0;
}

Compile toy.c in guest VM.

From host VM, run the following commands to generate the trace supposedly between __x64_sys_setuid + 0xA and __x64_sys_setuid + 0x13:

echo -n -e '\xCC' > cc
echo -n -e '\xC3' > c3
sudo ./kernel-fuzzer-for-xen-project/rwmem --domid <guest domain id> --write <base of __x64_sys_setuid + 0xA> --file cc --limit 1
sudo ./kernel-fuzzer-for-xen-project/rwmem --domid <guest domain id> --write <__x64_sys_setuid + 0x0C> --file cc --limit 1
sudo ./kernel-fuzzer-for-xen-project/kfx --domid <guest domain id> --json ubuntu.json --setup -c --harness breakpoint --start-byte 0x48
sudo ./kernel-fuzzer-for-xen-project/rwmem --domid <guest domain id> --write <__x64_sys_setuid + 0x0C>  --file c3 --limit 1

The address of __x64_sys_setuid can be found with sudo cat /proc/kallsyms | grep "__x64_sys_setuid". A sample of the disassembly can be found from previous comments in this thread.

After executing the set of commands should generate a buf file which can be fed into ptdump.
./vmtaint/libipt/build/bin/ptdump buf > dump.log

[markus-metzger]

@deyixtan were you able to try out @tklengyel's suggestion?

[deyixtan]

@markus-metzger @tklengyel Thank you, it works! My apologies, missed out on your earlier messages.