error when running bench: [file.c] assertion '(f = fopen(path, "w"))' failed: No such file or directory about sgx-step

[jovanbulck]

This is most likely a failure when trying to read from /dev/cpu/x/msr/ when setting up the APIC timer. This is only needed to dynamically retrieve the APIC base address, which is most likely fixed at 0xfee00000 anyway. So you could try to set APIC_CONFIG_MSR to zero to fall back to the static APIC base remapping instead:

https://github.com/jovanbulck/sgx-step/blob/master/libsgxstep/apic.c#L48

But be aware: it seems like you're trying to run SGX-Step in a cloud environment, which is not something I have tried before or SGX-Step was designed for. When running a VM you won't have the same control over hardware as when running SGX-Step natively, so things like APIC virtualization etc could easily complicate things and affect the framework's functionality. Would be cool if it works for you in a VM, but no guarantees at all -- I'd be interested to hear though if you get positive results :)

[Darktulip]

Hi Jo,

Thank you for your kind reply. You reminded me to look at the source code there.

Actually, I got f = fopen(path, "w"). So it seems to be a problem of writing instead of reading.

I checked the project and found that the statement ASSERT( (f = fopen(path, "w")) ); only exists in function file_write_int in file.c here https://github.com/jovanbulck/sgx-step/blob/master/libsgxstep/file.c#L77.

Then I checked the functions that call it. They are in sched.c here https://github.com/jovanbulck/sgx-step/blob/master/libsgxstep/sched.c#L99.

So the project will first check HAS_PSTATE and HAS_TURBO and then decide the return value. And these two are related to WITH_PSTATE and WITH_TURBO in makefile here https://github.com/jovanbulck/sgx-step/blob/master/libsgxstep/Makefile#L8.

I wrote a makefile to check the values. But I found there are neither 0 nor 1. They might be something like NULL.

Here is the makefile I wrote and the output.

.PHONY:all
CFLAGS = -g
all:
	@echo "WITH_PSTATE = $(WITH_PSTATE)"
	@echo "WITH_TURBO = $(WITH_TURBO)"

ifneq "$(WITH_PSTATE)" "0"
	@echo "WITH_PSTATE not 0"
endif

ifneq "$(WITH_TURBO)" "0"
	@echo "WITH_TURBO not 0"
endif

ifneq "$(WITH_PSTATE)" "1"
	@echo "WITH_PSTATE not 1"
endif

ifneq "$(WITH_TURBO)" "1"
	@echo "WITH_TURBO not 1"
endif

ifneq "$(WITH_PSTATE)" ""
	@echo "WITH_PSTATE not NULL"
endif

ifneq "$(WITH_TURBO)" ""
	@echo "WITH_TURBO not NULL"
endif

WITH_PSTATE = 
WITH_TURBO = 
WITH_PSTATE not 0
WITH_TURBO not 0
WITH_PSTATE not 1
WITH_TURBO not 1

So I guess these two might be the problem and change the original "0" to "".

I ran the bench again. This time it stuck at [LD] main.o -o app. The whole makefile output is here.

NUM=100 STRLEN=1 make parse
[===] Enclave [===]
[RM] encl.o asm.o asm_nop.o encl.unsigned.so encl.so libencl_proxy.a
[RM] encl_t.o encl_u.o encl_t.h encl_t.c encl_u.h encl_u.c
[===] ../../libsgxstep [===]
[RM] apic.o cpu.o debug.o enclave.o file.o foreshadow.o gdt.o idt.o pt.o sched.o spy.o aep_trampoline.o irq_entry.o rtm.o transient.o libsgx-step.a
[RM] main.o app
[===] Enclave [===]
[GEN] sgx_edger8r encl.edl
[CC]  encl_t.c (trusted edge)
[CC]  encl.c (core)
[AS]  asm.S (core)
[AS]  asm_nop.S (core)
[LD]   encl.o asm.o asm_nop.o encl_t.o -lsgx_trts encl.unsigned.so
[SGN] encl.unsigned.so
[CC]  encl_u.c (untrusted edge)
[AR]   libencl_proxy.a
[===] ../../libsgxstep [===]
[CC]  apic.c
[CC]  cpu.c
[CC]  debug.c
[CC]  enclave.c
[CC]  file.c
[CC]  foreshadow.c
[CC]  gdt.c
[CC]  idt.c
[CC]  pt.c
[CC]  sched.c
[CC]  spy.c
[AS]  aep_trampoline.S
[AS]  irq_entry.S
[AS]  rtm.S
[AS]  transient.S
[AR]  libsgx-step.a
[CC]  main.c
[LD] main.o -o app

I waited for around 10 seconds but it still did not move on. Is it normal here? Does the project rely on WITH_PSTATE and WITH_TURBO?

Thank you very much for your attention.

Best regards

[Darktulip]

Hi Jo,

I also tried what you said. I set APIC_CONFIG_MSR to 0 here https://github.com/jovanbulck/sgx-step/blob/master/libsgxstep/config.h#L29 by first commenting them and add #define APIC_CONFIG_MSR 0
like this.

/*
#if (M32 != 1)
	#define APIC_CONFIG_MSR         1
#else
	#define APIC_CONFIG_MSR         0
#endif
*/
#define APIC_CONFIG_MSR             0 // set APIC_CONFIG_MSR to zero to fall back to the static APIC base remapping instead

Unluckily, it doesn't work. And the output is the same as before. (I did not modify WITH_PSTATE and WITH_TURBO here).

Thank you for your suggestion and your kind attention.

Best regards

[jovanbulck]

Good you figured out where the file error came from -- it is indeed possible to disable pstate/turbo by running make WITH_PSTATE=0 WITH_TURBO=0. As you see in your Makefile, by default these env vars are simply non-existing/uninitialized, so you have to set them explicitly to zero to disable this functionality. SGX-Step does not critically rely on pstate/turbo, it's just one of the ways to make the CPU less noisy, so running with WITH_PSTATE=0 WITH_TURBO=0 should be okay.

The behavior where it seems to hang is likely because you run with make parse which redirects all output to a file for increased stability -- so you won't see any output. Check out.txt or try sudo ./app instead.

But then again: be aware I don't expect SGX-Step to work easily in VM environments..

Hope it helps, let me know if you could progress!

[Darktulip]

Hi Jo,

Thank you for your suggestion!

I forgot to check the output. Here is the output.

[idt.c] locking IRQ handler pages 0x55cb408e9000/0x55cb408ef000

--------------------------------------------------------------------------------
[main.c] Creating enclave...
--------------------------------------------------------------------------------

[sched.c] continuing on CPU 1
==== System Settings ====
    Pstate max perf pct: 0
    Pstate min perf pct: 0
    Turbo Boost:         0
    cpu pinning:         1
    Designated cpu:      1
    Running on cpu:      1
[pt.c] /dev/sgx-step opened!
==== Victim Enclave ====
    Base:   0x7fccf2800000
    Size:   4194304
    Limit:  0x7fccf2c00000
    TCS:    0x7fccf2a73000
    SSA:    0x7fccf2a74f48
    AEP:    0x55cb408e8d1b
    EDBGRD: debug
[main.c] enclave string adrs at 0x7fccf2812000
[pt.c] /dev/mem opened!
[main.c] enclave trigger code adrs at 0x7fccf2803000


--------------------------------------------------------------------------------
[main.c] Establishing user-space APIC/IDT mappings
--------------------------------------------------------------------------------

[idt.c] DTR.base=0xfffffe0000000000/size=4095 (256 entries)
[idt.c] established user space IDT mapping at 0x7fccf402e000
[idt.c] installed asm IRQ handler at 10:0x55cb408e9000
[idt.c] IDT[ 45] @0x7fccf402e2d0 = 0x55cb408e9000 (seg sel 0x10); p=1; dpl=3; type=14; ist=0
[apic.c] established local memory mapping for APIC_BASE=0xfee00000 at 0x7fccf402d000
[apic.c] APIC_ID=1000000; LVTT=ee; TDCR=3
[apic.c] APIC timer one-shot mode with division 2 (lvtt=2d/tdcr=0)
[main.c] calling enclave: attack=2; num_runs=100; timer=53
[main.c] Caught fault 11! Restoring enclave page permissions..
[main.c] ^^ enclave RIP=0x3000; ACCESSED=0
[main.c] strlen returned by enclave is 11
[main.c] attacker counted 0

If I switch to ./app, the program will also be stuck there.

When I ran lvi, it is also stuck at the end. The output of L1D is here.

==== Victim Enclave ====
[pt.c] /dev/sgx-step opened!
    Base:   0x7fd858400000
    Size:   4194304
    Limit:  0x7fd858800000
    TCS:    0x7fd858674000
    SSA:    0x7fd858675f48
    AEP:    0x7fd859bab9f1
    EDBGRD: debug
[main.c] oracle at 0x561acaab4000
[main.c] user_page at 0x561acabb4000
[main.c] enclave_page_a at 0x7fd858411000 w PTE
[pt.c] /dev/mem opened!
+-------------------------------------------------------------------------------------------+
| XD | PK | IGN | RSVD | PHYS ADRS      | IGN | G | PAT | D | A | PCD | PWT | U/S | R/W | P | 
| 0  | x  | x   | 0    | 0x000440130000 | x   | x | x   | 1 | 1 | x   | x   | 1   | 1   | 1 | 
+-------------------------------------------------------------------------------------------+
[main.c] enclave_page_b at 0x7fd858410000 w PTE
+-------------------------------------------------------------------------------------------+
| XD | PK | IGN | RSVD | PHYS ADRS      | IGN | G | PAT | D | A | PCD | PWT | U/S | R/W | P | 
| 0  | x  | x   | 0    | 0x00044012f000 | x   | x | x   | 1 | 1 | x   | x   | 1   | 1   | 1 | 
+-------------------------------------------------------------------------------------------+

It doesn't output the critical last line nor exit. I guess it is still in the while(1) loop. The results are similar when I try SB or ROP.

When I ran foreshadow, I tried with either dis_ucode_ldr or not. In this two cases, the outputs are the same as follow.

[main.c] verifying and destroying enclave secret..
 ** shadow[ 0]=0x00; enclave[ 0]=0x48 ** shadow[ 1]=0x00; enclave[ 1]=0xc2
 ** shadow[ 2]=0x00; enclave[ 2]=0x15 ** shadow[ 3]=0x00; enclave[ 3]=0xdd
 ** shadow[ 4]=0x00; enclave[ 4]=0x49 ** shadow[ 5]=0x00; enclave[ 5]=0xe5
 ** shadow[ 6]=0x00; enclave[ 6]=0x89 ** shadow[ 7]=0xdd; enclave[ 7]=0x97
 ** shadow[ 8]=0xdd; enclave[ 8]=0x84 ** shadow[ 9]=0x00; enclave[ 9]=0x0f
 ** shadow[10]=0x00; enclave[10]=0x40 ** shadow[11]=0x00; enclave[11]=0x9a
 ** shadow[12]=0x00; enclave[12]=0x57 ** shadow[13]=0x00; enclave[13]=0xb1
 ** shadow[14]=0xd6; enclave[14]=0x90 ** shadow[15]=0x00; enclave[15]=0x29
 ** shadow[16]=0xd6; enclave[16]=0x07 ** shadow[17]=0xc7; enclave[17]=0xb1
 ** shadow[18]=0x17; enclave[18]=0xb8 ** shadow[19]=0x0d; enclave[19]=0x59
 ** shadow[20]=0x17; enclave[20]=0x7b ** shadow[21]=0x2d; enclave[21]=0x1a
 ** shadow[22]=0x11; enclave[22]=0xd8 ** shadow[23]=0x3d; enclave[23]=0x5f
 ** shadow[24]=0x25; enclave[24]=0x52 ** shadow[25]=0x17; enclave[25]=0xd2
 ** shadow[26]=0x0d; enclave[26]=0xe4 ** shadow[27]=0x17; enclave[27]=0x29
 ** shadow[28]=0x5b; enclave[28]=0xf9 ** shadow[29]=0x0d; enclave[29]=0x78
 ** shadow[30]=0x17; enclave[30]=0x60 ** shadow[31]=0x25; enclave[31]=0xa2
 ** shadow[32]=0x17; enclave[32]=0x9c ** shadow[33]=0x11; enclave[33]=0x7b
 ** shadow[34]=0x17; enclave[34]=0xed ** shadow[35]=0x0d; enclave[35]=0x15
 ** shadow[36]=0x41; enclave[36]=0x0e ** shadow[37]=0x11; enclave[37]=0x94
 ** shadow[38]=0x57; enclave[38]=0xcb ** shadow[39]=0x17; enclave[39]=0x9e
 ** shadow[40]=0x41; enclave[40]=0xfb ** shadow[41]=0x17; enclave[41]=0xcf
 ** shadow[42]=0x11; enclave[42]=0x88 ** shadow[43]=0x60; enclave[43]=0xd4
 ** shadow[44]=0x2d; enclave[44]=0xa8 ** shadow[45]=0x17; enclave[45]=0x5f
 ** shadow[46]=0x0b; enclave[46]=0x2c ** shadow[47]=0x11; enclave[47]=0xb3
 ** shadow[48]=0x25; enclave[48]=0x85 ** shadow[49]=0x00; enclave[49]=0xe2
 ** shadow[50]=0x00; enclave[50]=0xeb ** shadow[51]=0x00; enclave[51]=0xf1
 ** shadow[52]=0x00; enclave[52]=0x34 ** shadow[53]=0x00; enclave[53]=0x22
 ** shadow[54]=0xc7; enclave[54]=0x82 ** shadow[55]=0x0d; enclave[55]=0x5a
 ** shadow[56]=0x2d; enclave[56]=0x7d ** shadow[57]=0x11; enclave[57]=0x40
 ** shadow[58]=0x0d; enclave[58]=0xea ** shadow[59]=0x17; enclave[59]=0xd8
 ** shadow[60]=0x57; enclave[60]=0xa8 ** shadow[61]=0x3d; enclave[61]=0x0e
 ** shadow[62]=0x25; enclave[62]=0x8b ** shadow[63]=0x11; enclave[63]=0x1f
[foreshadow.c] [FAIL] Foreshadow missed 64 bytes out of 64 :/

It really seems not successful in VM. :(

Sad.jpg

[jovanbulck]

Thanks for following up, this is indeed not unexpected when attempting single-stepping in an unprivileged VM I'm afraid.. so I'm closing this issue