Work-in-progress Gramine port about sgx-step

[aegis2501]

Hi, another quick update: you were right about the 4 page SSA frame size (at least the erip I see while stepping appears to be correct). I'll share my code once I've cleaned up my local git repos and add some usage instructions (It's kind of a mess at the moment).

[aegis2501]

Thank you for the fix :)
As for the typo: shouldn't the macro directive read:
.global \name (perhaps the global isn't actually needed?)
UPDATE: nvm, one never fails to learn something new... https://sourceware.org/binutils/docs/as/Global.html

[aegis2501]

Hello again, I've submitted a pull request for the changes I made to Gramine and SGX-Step. Sorry for the delay, I hope it will help others in their research :)

[jovanbulck]

thank you for the heads-up, this is great! I am busy this week, but I'll review and test the PR end of this week or next week and leave comments in the PR discussion :)

[jovanbulck]

Hi aegis,

The plan is to refactor SGX-Step to make it easier to interface with different SDKs and LibOSs, todo as documented in #28 . However, I have not found time for this yet and probably not any time soon..

Is there a specific LibOS you'd like to interface with?

Generally speaking, it shouldn't be too hard to port SGX-Step to a different SDK/libOS. Basically you'd need to hook the AEP and interface with the underlying driver (isgx or the upstream Linux driver todo as in #39). You could have a look at the existing SDK patch to see what to do minimally there.

So far that I know of people have (partially) ported SGX-Step to work with Graphene and SGX-LKL (#27). But there is no code available as far as I know..

I'd always be happy to merge code via a PR if you'd get anything working!

If you have specific questions on how to port a specific libOS, feel free to follow up. If not, close this issue?

[aegis2501]

Hi jovanbulck,
I was hoping on being able to interface with gramine (formally graphene). The issue you mentioned does not seem to reference graphene anywhere (apologies if I overlooked something). If you say that applying a patch is doable manually, I guess I'll give it a go :)
Thank you for your help!

[jovanbulck]

Interfacing with Gramine/Graphene is currently not supported in the code, but I think it should certainly be possible without too much porting effort.

I've seen some papers that interfaced SGX-Step with Graphene (but no code available, afaik), e.g., this paper.

The main required changes would be to figure out how to set a custom AEP and retrieve the TCS address, as described in #27 . Also, be aware that SGX-Step currently only supports the out-of-tree linux-sgx-driver (#39 ), so you'd have to check how to run Gramine with that (not sure, haven't tried myself on recent Gramine versions -- this might or might not be helpful).

Hope it helps! I'd be happy to discuss and merge any binding code for interfacing SGX-Step with Gramine, so feel free to open a PR for that :)

[jovanbulck]

Closing this for now, feel free to re-open and ask additional questions if you have concrete questions or problems on how to port to Gramine!

[aegis2501]

Hello jovanbulck,
after drawing some inspiration from your previous work (https://github.com/jovanbulck/sgx-pte), my idea for smuggling out the aep/tcs out (and back into) of the enclave would be something along the lines of:

1. patch relevant the EENTER assembly in Gramine (https://github.com/gramineproject/gramine/blob/master/pal/src/host/linux-sgx/host_entry.S#L41) similar to the way the sgx-step applies a patch to the sgx-sdk
2. pass the values through PAL wrapper functions
3. pass the PAL wrapper functions with further wrapper functions to libOS
4. adjust libsgxstep to use syscalls to get and set aep/tcs in enclave.c

Currently I've implemented 1-2 and tried to add libsgxstep directly as a dependency to the loader (of the untrusted runtime) in Gramine's meson build. However, this leads to some rather unwieldy dependencies between Gramine and SGX-Step (PAL from Gramine is a dependency of Step, whereas Step is a dependency of Gramine's loader).

I'd be happy for feedback!

[jovanbulck]

Hi aegis,

Not sure what you mean with "smuggling out the aep/tcs out (and back into) of the enclave". Note that all of SGX-Step code normally runs outside of the enclave, i.e., without support of the victim code(!) So there won't be any needs for system calls etc for building and linking with SGX-Step and using basic libsgxstep functionality.

(unrelated to this: perhaps adding a debug system call and corresponding ocall in the enclaved LibOS may still be useful to debug a victim enclave code snippet before developing a final attack, e.g., I think I did something similar in my sgx-pte code above)

I was curious so I started hacking away a bit at a proof-of-concept Gramine integration stub that you can find here and some compilation instructions in the README.

Note that integration with Gramine is currently untested and only provided as an example/starter for people wishing to experiment with SGX-Step on Gramine. Particularly, the patches were validated to successfully compile but were never actually ran(!) Furthermore, single-stepping itself is not currently provided for Gramine, but should be straightforwardly feasible based on the existing code for the Intel SDK.

Hope it helps in your research!

I'd certainly be interested to hear if the patches actually worked for you? (as I didn't have a running Gramine setup to test it on at the moment).

[aegis2501]

Hi jovanbulck,

you're absolutely right (I only realized that that was a somewhat stupid statement after writing it since the PAL layer is outside of the enclave). I guess I got a bit confused because their documentation only explains how to add libOS functionality because their philosophy is to keep the PAL as minimal as possible.

I'll try your patches out on my setup (many thanks!)

[aegis2501]

Hello,
just a quick follow-up:
I added in your patches and added a new header called (pal/src/host/linux-sgx/sgxstep.h) with:

void* sgx_get_aep(void);
void sgx_set_aep(void* aep);
void* sgx_get_tcs(void);

then included this in libsgxstep/enclave.c (replacing the patched sgx_urts.h)

With this typo fixed (https://github.com/jovanbulck/sgx-step/blob/master/sdk/gramine/0001-SGX-Step-Gramine-patches-to-reconfigure-AEP-TCS.patch#L66), register_aep_eb() seems to work.

Did you run the CI hello-world example with the kernel driver? I'm currently using legacy isgx, where attempting to find victim.base from proc maps in register_enclave_info() doesn't seem to work. The mapping that proc returns is:

$ gramine-sgx ./helloworld
[idt.c] locking IRQ handler pages 0x55c332f1d000/0x55c332f2d000
Gramine is starting. Parsing TOML manifest file, this may take some time...
[../pal/src/host/linux-sgx/host_ecalls.c] entering victim enclave
==== Victim Enclave ====
[enclave.c] (nil) - 0x10000 /dev/isgx
[enclave.c] Found /dev/isgx enclave at (nil) in /proc/self/maps
[enclave.c] 0x10000 - 0xfa69000 /dev/isgx
[enclave.c] 0xfa69000 - 0xfabc000 /dev/isgx
[enclave.c] 0xfabc000 - 0xfabd000 /dev/isgx
[enclave.c] 0xfabd000 - 0xfac2000 /dev/isgx
[enclave.c] 0xfac2000 - 0xfe95000 /dev/isgx
[enclave.c] 0xfe95000 - 0xfea5000 /dev/isgx
[enclave.c] 0xfea5000 - 0xfeb5000 /dev/isgx
[enclave.c] 0xfeb5000 - 0xfec5000 /dev/isgx
[enclave.c] 0xfec5000 - 0xfed5000 /dev/isgx
[enclave.c] 0xfed5000 - 0xff15000 /dev/isgx
[enclave.c] 0xff15000 - 0xff55000 /dev/isgx
[enclave.c] 0xff55000 - 0xff95000 /dev/isgx
[enclave.c] 0xff95000 - 0xffd5000 /dev/isgx
[enclave.c] 0xffd5000 - 0xffd9000 /dev/isgx
[enclave.c] 0xffd9000 - 0xffdd000 /dev/isgx
[enclave.c] 0xffdd000 - 0xfffd000 /dev/isgx
[enclave.c] 0xfffd000 - 0x10000000 /dev/isgx
[enclave.c] 0x10000000000 - 0x1000000d000 (null)
[enclave.c] 0x55c332efb000 - 0x55c332efe000 /usr/local/lib/x86_64-linux-gnu/gramine/sgx/loader
[enclave.c] 0x55c332efe000 - 0x55c332f1d000 /usr/local/lib/x86_64-linux-gnu/gramine/sgx/loader
[enclave.c] 0x55c332f1d000 - 0x55c332f1e000 /usr/local/lib/x86_64-linux-gnu/gramine/sgx/loader
[enclave.c] 0x55c332f1e000 - 0x55c332f2a000 /usr/local/lib/x86_64-linux-gnu/gramine/sgx/loader
[enclave.c] 0x55c332f2a000 - 0x55c332f2c000 /usr/local/lib/x86_64-linux-gnu/gramine/sgx/loader
[enclave.c] 0x55c332f2c000 - 0x55c332f2d000 /usr/local/lib/x86_64-linux-gnu/gramine/sgx/loader
[enclave.c] 0x55c332f2d000 - 0x55c332f2e000 /usr/local/lib/x86_64-linux-gnu/gramine/sgx/loader
[enclave.c] 0x55c332f2e000 - 0x55c332f36000 (null)
[enclave.c] 0x55c333958000 - 0x55c333979000 [heap]
[enclave.c] 0x7fdafc466000 - 0x7fdafc469000 (null)
[enclave.c] 0x7fdafc469000 - 0x7fdafc48b000 /usr/lib/x86_64-linux-gnu/[libc-2.31.so](http://libc-2.31.so/)
[enclave.c] 0x7fdafc48b000 - 0x7fdafc603000 /usr/lib/x86_64-linux-gnu/[libc-2.31.so](http://libc-2.31.so/)
[enclave.c] 0x7fdafc603000 - 0x7fdafc651000 /usr/lib/x86_64-linux-gnu/[libc-2.31.so](http://libc-2.31.so/)
[enclave.c] 0x7fdafc651000 - 0x7fdafc655000 /usr/lib/x86_64-linux-gnu/[libc-2.31.so](http://libc-2.31.so/)
[enclave.c] 0x7fdafc655000 - 0x7fdafc657000 /usr/lib/x86_64-linux-gnu/[libc-2.31.so](http://libc-2.31.so/)
[enclave.c] 0x7fdafc657000 - 0x7fdafc65b000 (null)
[enclave.c] 0x7fdafc65b000 - 0x7fdafc65c000 /usr/lib/x86_64-linux-gnu/[libprotobuf-c.so](http://libprotobuf-c.so/).1.0.0
[enclave.c] 0x7fdafc65c000 - 0x7fdafc662000 /usr/lib/x86_64-linux-gnu/[libprotobuf-c.so](http://libprotobuf-c.so/).1.0.0
[enclave.c] 0x7fdafc662000 - 0x7fdafc664000 /usr/lib/x86_64-linux-gnu/[libprotobuf-c.so](http://libprotobuf-c.so/).1.0.0
[enclave.c] 0x7fdafc664000 - 0x7fdafc665000 /usr/lib/x86_64-linux-gnu/[libprotobuf-c.so](http://libprotobuf-c.so/).1.0.0
[enclave.c] 0x7fdafc665000 - 0x7fdafc666000 /usr/lib/x86_64-linux-gnu/[libprotobuf-c.so](http://libprotobuf-c.so/).1.0.0
[enclave.c] 0x7fdafc666000 - 0x7fdafc668000 (null)
[enclave.c] 0x7fdafc669000 - 0x7fdafc67a000 (null)
[enclave.c] 0x7fdafc67a000 - 0x7fdafc67b000 /usr/lib/x86_64-linux-gnu/[ld-2.31.so](http://ld-2.31.so/)
[enclave.c] 0x7fdafc67b000 - 0x7fdafc69e000 /usr/lib/x86_64-linux-gnu/[ld-2.31.so](http://ld-2.31.so/)
[enclave.c] 0x7fdafc69e000 - 0x7fdafc6a6000 /usr/lib/x86_64-linux-gnu/[ld-2.31.so](http://ld-2.31.so/)
[enclave.c] 0x7fdafc6a7000 - 0x7fdafc6a8000 /usr/lib/x86_64-linux-gnu/[ld-2.31.so](http://ld-2.31.so/)
[enclave.c] 0x7fdafc6a8000 - 0x7fdafc6a9000 /usr/lib/x86_64-linux-gnu/[ld-2.31.so](http://ld-2.31.so/)
[enclave.c] 0x7fdafc6a9000 - 0x7fdafc6aa000 (null)
[enclave.c] 0x7ffc63db6000 - 0x7ffc63fb9000 [stack]
[enclave.c] 0x7ffc63ff3000 - 0x7ffc63ff7000 [vvar]
[enclave.c] 0x7ffc63ff7000 - 0x7ffc63ff9000 [vdso]
[enclave.c] 0xffffffffff600000 - 0xffffffffff601000 [vsyscall]
[enclave.c] assertion 'victim.base && "no enclave found in /proc/self/maps"' failed: Success
Aborted (core dumped)

Having the start pointer as (nil) seems a bit nonsensical, any idea what the issue might be?

UPDATE: rolling back to 9f53252 works using the ioctl method via /dev/sgx-step, not sure why tbh, since the base pointer is still (nil)

[jovanbulck]

Hi aegis,

Nice to see it helped! Not sure what typo you meant, as the patches apply and work on my end, so let me know if there's an error. I fixed the dependency on the Intel SDK header as per e6a8a95, so it should be easier to link to libsgxstep now.

I was indeed using the more recent in-kernel driver. SGX-Step indeed didn't support victim.base=0 and this should now be fixed as per 2eba798

Note that gramine loads the enclave at base address zero when using the old SGX driver, so this is normal behavior, eg see explanation here

[aegis2501]

Hello again,
2eba798 appears to work.
Another question: you mention in the readme that single-stepping is currently no provided by Gramine, could you perhaps elaborate on what must be done for this to be possible? I'm currently attempting to expand the example you provided with the benchmark app. So far I've added a fault_handler and changing the page permissions appears to work.

[jovanbulck]

okay good to hear it works for you!

yes, you'll have to modify the AEP handler similar to app/bench to get single stepping to work.

Also make sure to check out the latest commit referenced above as this fixes some possible issues with multithreaded enclaves (as may be used in gramine). You'll also have to double check the SSA frame size if you wish to print registers..

Hope it helps! If you get it to work, I'd be interested to hear :)

[jovanbulck]

great to hear it's working for you! Id be happy to merge your code if you open a PR later -- even if it's not entirely clean and perfect (especially if it doesn't require many changes outside of the sdk/gramine directory), I'm sure it could be of help to others :)

[aegis2501]

Hi, I've tried to update the pull request according to your comment :)