skalenetwork / sgxwallet

sgxwallet is the first-ever opensource high-performance hardware secure crypto wallet that is based on Intel SGX technology. First opensource product on Intel SGX whitelist. Scales to 100,000+ transactions per second. Currently supports ETH and SKALE, and will support BTC in the future. Sgxwallet is under heavy development and use by SKALE network.

sgxwallet: SKALE SGX-based hardware crypto wallet

Intro

sgxwallet is a next generation hardware secure crypto wallet that is based on Intel SGX technology. It currently supports Ethereum and SKALE, and will support Bitcoin in the future.

sgxwallet runs as a network server. Clients connect to the server, authenticate to it using TLS 1.0 protocol with client certificates, and then issue requests to the server to generate crypto keys and perform cryptographic operations. The keys are generated inside the secure SGX enclave and never leave the enclave unencrypted.

The server provides an initial registration service to issue client certificates to the clients. The administrator manually approves each registration.

sgxwallet has been tested on Ubuntu Linux 18.04.

An important note about production readiness

The sgxwallet server is still in active development and therefore should be regarded as alpha software. The development is still subject to security hardening, further testing, and breaking changes. This server has not yet been reviewed or audited for security. Please see SECURITY.md for reporting policies.

Running sgxwallet

Clone this repo

As you probably suspect, the first thing to do is to clone this repository and all it is sub-repositories.

git clone https://github.com/skalenetwork/sgxwallet.git --recurse-submodules

Try in simulation mode

The easiest way to try the sgxwallet server is to run a docker container in insecure simulation mode that emulates an SGX processor. Once you are familiar with the server, you can enable sgx on your machine and run it in secure production mode.

First install docker-compose if you dont have it

sudo apt-get install docker.io docker-compose

Then run sgxwallet using docker-compose

cd run_sgx_sim; sudo docker-compose up

Note: you need a machine that supports Intel AVX512 instruction set. Most modern Intel CPUs support it. To verify you machine supports AVX512, run

cat /proc/cpuinfo | grep avx512

Note: sgxwallet requires docker-compose for correct operation. You must always use docker-compose and avoid using raw docker tools.

Note: simulation mode is only to try sgxwallet. In production, you need to run sgxwallet on a server that supports SGX. Never run a production sgxserver in simulation mode.

Admin guide

If you are a SKALE validator and want to run sgxwallet for testnet or mainnet usage, you need a SGX-capable server.
Please refer to Admin guide for details on how to setup sgxwallet in a secure hardware mode docs/admin-guide.md.

Developer guide

If you are a SKALE developer and want to build sgxwallet from source, please refer to Developer guide docs/developer-guide.md.

Contributing

See contributing for information on how to contribute.

Libraries used by this project

• Intel-SGX-SSL by Intel
• LevelDB by Google
• libBLS by SKALE Labs
• libff by SCIPR-LAB
• Linux SGX Driver by Intel
• SGX-GMP by Intel
• SGX Software Enable by Intel

License

All contributions to sgxwallet are made under the GNU Affero General Public License v3. See LICENSE.

Copyright (C) 2019-Present SKALE Labs.

sgxwallet's Issues

empty error log in sgxwallet

sgxwallet doesn't print error message in some cases

┆Issue is synchronized with this Jira Bug

Complete and enable ZeroMQ integration

Complete and enable ZeroMQ integration

┆Issue is synchronized with this Jira Task

Upgrade SGX to > 2.5

validators have issues with it

┆Issue is synchronized with this Jira Task

add check public shares are in G2 group

The is_well_formed check in libff::alt_bn128_G2 is insufficient. It checks that the projective coordinates (X,Y,Z) satisfy the equation of the curve, but G2 is actually a subgroup of the elliptic curve points. An attacker that controls n-t participants, where n are all participants and t+1 is the number of participants required to sign a message with the common public key, will always be able to corrupt the public key so it is almost impossible to create a common signature, allowing the attacker to escape later detection during the signing phase. With a probability of 1/10069 the attacker can do the same with n-t participants.

┆Issue is synchronized with this Jira Bug

Add method to regenerate SEK

and to reencrypt db

fix sgx logging: sgx should always(on any log level) logs error messages if any

we can't get enough logs from sgx in case of some errors inside enclave. lets fix this this and ensure sgx always prints error messages if they appear

┆Issue is synchronized with this Jira Bug

use ipp-crypto in sgxwallet

use https://github.com/intel/ipp-crypto to do ECDSA signing, ECDH key exchange and finite field arithmetic for better stability and performance

┆Issue is synchronized with this Jira Story

Skaled can't connect to sgx

Preconditions
SGX: custom version
sgx cmd: -s y V b c
Skaled: custom build based on latest beta

Step to reproduce
Open zmq port on sgx(1031)
Create first schain
Create second schain

Actual result
Second schain couldn't connect to SGX. But no error log form skaled, just stuck.

In case if we restart sgx and disable zmq, skaled not trying to reconnect to sgx. If we restart skaled on all nodes it will successful reconnected.

In case if we enable zmq and spin up new schain it will stuck. Case with restart all skaled with enabled zmq not help.

┆Issue is synchronized with this Jira Bug
┆Attachments: log

Debian 10 Buster Install

Trying to install this on Debian 10 Buster (kernel 5.3.18-3-pve), so that for redundancy purposes I have two SGX wallets instead of one.

sgx_enable was built from source (make)

~/sgxwallet/sgx-software-enable# ll
total 76K
drwxr-xr-x  2 root root 4.0K May  6 15:26 .
drwxr-xr-x 22 root root 4.0K May  6 15:01 ..
-rw-r--r--  1 root root   44 May  6 15:01 .git
-rw-r--r--  1 root root  120 May  6 15:01 Makefile
-rw-r--r--  1 root root 5.7K May  6 15:01 README.md
-rw-r--r--  1 root root 3.5K May  6 15:01 sgx_capable.h
-rwxr-xr-x  1 root root  18K May  6 15:26 sgx_enable
-rw-r--r--  1 root root  13K May  6 15:01 sgx_enable.c
-rw-r--r--  1 root root  11K May  6 15:26 sgx_enable.o

~/sgxwallet/sgx-software-enable# ./sgx_enable
Intel SGX is already enabled on this system

~/sgxwallet/sgx-software-enable# cd ../scripts/

~/sgxwallet/scripts# ./sgx_linux_x64_driver_2.5.0_2605efa.bin
Unpacking Intel SGX Driver ... done.
Verifying the integrity of the install package ... done.
Installing Intel SGX Driver ...
/tmp/sgx-driver-znUl6F ~/sgxwallet/scripts
install -d /opt/intel/sgxdriver/package
install -d /opt/intel/sgxdriver/scripts
install package/* /opt/intel/sgxdriver/package
install scripts/* /opt/intel/sgxdriver/scripts
~/sgxwallet/scripts
/opt/intel/sgxdriver/package ~/sgxwallet/scripts
make -C /lib/modules/5.3.18-3-pve/build SUBDIRS=/opt/intel/sgxdriver/package modules
make[1]: Entering directory '/usr/src/linux-headers-5.3.18-3-pve'
Makefile:227: ================= WARNING ================
Makefile:228: 'SUBDIRS' will be removed after Linux 5.3
Makefile:229:
Makefile:230: If you are building an individual subdirectory
Makefile:231: in the kernel tree, you can do like this:
Makefile:232: $ make path/to/dir/you/want/to/build/
Makefile:233: (Do not forget the trailing slash)
Makefile:234:
Makefile:235: If you are building an external module,
Makefile:236: Please use 'M=' or 'KBUILD_EXTMOD' instead
Makefile:237: ==========================================
  CC [M]  /opt/intel/sgxdriver/package/sgx_main.o
  CC [M]  /opt/intel/sgxdriver/package/sgx_page_cache.o
/opt/intel/sgxdriver/package/sgx_page_cache.c: In function ‘sgx_test_and_clear_young’:
/opt/intel/sgxdriver/package/sgx_page_cache.c:126:8: error: passing argument 4 of ‘apply_to_page_range’ from incompatible pointer type [**-Werror=incompatible-pointer-types**]
        **sgx_test_and_clear_young_cb**, vma->vm_mm);
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /opt/intel/sgxdriver/package/sgx.h:72,
                 from /opt/intel/sgxdriver/package/sgx_page_cache.c:61:
./include/linux/mm.h:2679:40: note: expected ‘pte_fn_t’ {aka ‘int (*)(struct <anonymous> *, long unsigned int,  void *)’} but argument is of type ‘int (*)(pte_t *, struct page *, long unsigned int,  void *)’ {aka ‘int (*)(struct <anonymous> *, struct page *, long unsigned int,  void *)’}
           unsigned long size, pte_fn_t fn, void *data);
                               ~~~~~~~~~^~
cc1: some warnings being treated as errors
make[2]: *** [scripts/Makefile.build:290: /opt/intel/sgxdriver/package/sgx_page_cache.o] Error 1
make[1]: *** [Makefile:1655: _module_/opt/intel/sgxdriver/package] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-5.3.18-3-pve'
make: *** [Makefile:16: default] Error 2

Maybe your team has some suggestions to get around this? I don't want to install the driver directly from https://github.com/intel/linux-sgx-driver, unless instructed to do so.

PS: libprotobuf10 was installed by downloading the Debian Stretch package, unpacking it to a temp folder (not to overwrite any permissions), then copying ibprotobuf.so.10.0.0 (along with the symlink) to /usr/lib/x86_64-linux-gnu

Change getDecryptionShare to getDecryptionShares

Make getDecryptionShare in SGX to take an array of encrypted values and return an array of values.

This is because a typical TE encrypted block will include many transactions and all of them need to be

decrypted in a single call to be efficient.

If a particular value is corrupt and can not be decrypted, return error for this value, but decrypt other value.

Otherwise an attacker can cause a DoS attack by submitting a corrupt value in a transaction.

┆Issue is synchronized with this Jira Task

Exception in zmq server worker

When DKG time on admin is over, admin terminate process which responsible for DKG waiting from contract. And when DKG on contract successfully over, admin do not know about and did not ask contract about DKG

versions:
schain:3.7.1-develop.7
admin:2.0.0-develop.36

STR:
create 6+ MEDIUM schains
expected: all schains containers works and all schains block mining
actual: sgx stuck with Exception in zmq server worker:std:State check failed::_msg[_size - 1] == '}' ZMQMessage.cpp:65 after some period of schains working ~ 260 blocks

┆Issue is synchronized with this Jira Bug
┆Attachments: schain_anz.txt | sgx_logs_after_up.txt

SGX: Skaled stuck on processNextAvailableConnection

Preconditions
Skaled: 3.5.8-asan-plus-snapshots
SGX: 1.70.0-beta.0

Step to reproduce
Create schan medium type
Observe logs

Actual result
Skaled stuck on processNextAvailableConnection for 3h on 3 of 16 schain

┆Issue is synchronized with this Jira Bug
┆Attachments: file

use gnutls 3.6.7 or higher everywhere (at least sgx)

https://www.tenable.com/plugins/nessus/124210

┆Issue is synchronized with this Jira Bug

Clean Zeromq SGX error processing

Add errror codes for ZMQ error processing

┆Issue is synchronized with this Jira Bug

investigate duplicated certificates issue in openssl

sometimes skale-node cannot create ssl certificate on sgx due to internal openssl error. need to investigate it to prevent it in future

┆Issue is synchronized with this Jira Bug

RNG error on SGX server

Is this the expected behavior? I see some issues related to RNG
Using Ubuntu 18.04.4 LTS

sgxwallet_1  | Running sgxwallet version:1.45.1
sgxwallet_1  | [2020-03-28 16:00:06.761] [debug] SGX_DEBUG_FLAG = 1
sgxwallet_1  | [2020-03-28 16:00:06.946] [info] Enclave created and started successfully
sgxwallet_1  | [2020-03-28 16:00:06.946] [info] Enclave libtgmp library initialized successfully
sgxwallet_1  | ***ENCLAVE_LOG***: Initing tgmp library
sgxwallet_1  | [2020-03-28 16:00:06.947] [info] Initing wallet database ... 
sgxwallet_1  | [2020-03-28 16:00:06.947] [info] Opening wallet databases
sgxwallet_1  | [2020-03-28 16:00:07.025] [info] Successfully opened databases
sgxwallet_1  | [2020-03-28 16:00:07.025] [debug] key to read from db: SEK
sgxwallet_1  | [2020-03-28 16:00:07.025] [error] SEK was not created yet. Going to create SEK
sgxwallet_1  | ATTENTION! THIS IS YOUR KEY FOR BACK UP. PLEASE COPY IT TO THE SAFE PLACE
sgxwallet_1  | key is balblabla49e3b86908796857
sgxwallet_1  | reset: terminal attributes: No such device or address
sgxwallet_1  | 
sgxwallet_1  | [2020-03-28 16:00:07.028] [debug] key to read from db: SEK
sgxwallet_1  | Could not execute reset
sgxwallet_1  | [2020-03-28 16:00:07.028] [debug] written key: SEK
sgxwallet_1  | [2020-03-28 16:00:07.028] [debug] SEK
sgxwallet_1  | [2020-03-28 16:00:07.028] [debug] key to read from db: TEST_KEY
sgxwallet_1  | [2020-03-28 16:00:07.028] [debug] written key: TEST_KEY
sgxwallet_1  | [2020-03-28 16:00:07.028] [debug] TEST_KEY
sgxwallet_1  | [2020-03-28 16:00:07.028] [info] NO ROOT CA CERTIFICATE YET. CREATING ...
sgxwallet_1  | Generating RSA private key, 2048 bit long modulus (2 primes)
sgxwallet_1  | ....................+++++
sgxwallet_1  | ..........................+++++
sgxwallet_1  | e is 65537 (0x010001)
sgxwallet_1  | Can't load /root/.rnd into RNG
sgxwallet_1  | 139851374010816:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/root/.rnd

Skaled couldn't check incoming snapshot with new BLS signature after node rotation

Versions:
schain:3.7.5-beta.4
admin:2.0.1-beta.7
sgx_wallet:1.77.0

Preconditons:

• Schain 16/16 MEDIUM type
• 1 Active node without schains
• 1 snapshot on schain
• Snapshot interval 2 hours
• Node rotation delay 1 hour

Steps to reproduce:

1. Call node exit on the node A and wait till node B will complete BLS keys, skaled container
2. Skaled on the node B starts catchup from 0 block and couldn't download any blocks (old blocks signed with old signatures) but not crashed
3. Wait snapshot creation - 15 nodes should create snapshot with new BLS keys
4. Restart skaled on the node B with repair mode

Actual state: 15 nodes sign old snapshot with new BLS keys but node B couldn't check incoming snapshot with own BLS key
Expected state: Node B should be able to check incoming snapshot with valid BLS key

┆Issue is synchronized with this Jira Bug
┆Attachments: quiet-cursa.log

Fix automated sgxwallet signed build

Fix automated sgxwallet signed build and please actually test that it works.

┆Issue is synchronized with this Jira Task

SGX wallet didn't restart after oom kill for several hours

Preconditions
SGX wallet: 1.70.0-beta.1
20 nodes up and running
4 schains up and running, 3 of them empty blocks, last one under load(23 tx per)

Step to reproduce
Spin up schains, make load.(skaled tests like cat-cycle) or just empty blocks will enough.

Actual result
SGX restarted by OOM kill and up after 6h.
Full reboot log attached.

[Sat Mar 20 17:51:09 2021] [23474]     0 23474    16397      122   172032        0             0 cron
[Sat Mar 20 17:51:09 2021] Out of memory: Kill process 24368 (sgxwallet) score 838 or sacrifice child
[Sat Mar 20 17:51:09 2021] Killed process 24368 (sgxwallet) total-vm:41523820kB, anon-rss:27389556kB, file-rss:0kB, shmem-rss:0kB

NOTE: Let's gracefully turn off sgx not force. [~accountid:5b293e1e9bcd7518af2198f6][~accountid:5b2037f147afa959b14789e5][~accountid:5ec633056c50620c1ca56e20][~accountid:5b88195a9d17912bf10aef4c]

┆Issue is synchronized with this [Jira Bug](https://skalelabs.atlassian.net/browse/SKALE-4005)
┆Attachments: <a href="https://skalelabs.atlassian.net/rest/api/2/attachment/content/10553">sgxRebootStuck.log</a>

sgx server returns empty reply if cert is invalid

curl --cert /skale_node_data/sgx_certs/sgx.crt --key /skale_node_data/sgx_certs/sgx.key X POST data '{"id":1, "jsonrpc":"2.0","method":"importBLSKeyShare","params":{"keyShareName":"BLS_KEY:SCHAIN_ID:1625669718:NODE_ID:1:DKG_ID:0","keyShare":"0xBDCCBF9FA66FB00A44A00D614E68A294B1B8B41185A1460C64177B348A9722"}}' H 'content-type:application/json;' https://45.76.3.64:1026 k

curl: (52) Empty reply from server

Probably it could return descriptive error message?!

┆Issue is synchronized with this Jira Bug

limit sgxwallet resourses

make sure sgxwallet memory consumption never hits 100% - always restart sgxwallet if current usage more than 80%

┆Issue is synchronized with this Jira Bug

Fix zeromq close

Fix zeromq close

┆Issue is synchronized with this Jira Bug

refactor sgx_util tool

this tool seems not working properly. review the code and usage, add documentation

┆Issue is synchronized with this Jira Task

Add flag to sgxwallet

┆Issue is synchronized with this Jira Task
┆Attachments: logs.log | sgx_keys.log

sgxwallet cannot sign message with 0x prefix

sgxwallet cannot sign message with 0x prefix

┆Issue is synchronized with this Jira Bug

Add a check for linux kernel version and warn if it is 5

warn validators if using LInux Kernel v 5

┆Issue is synchronized with this Jira Task

Update SGX to the latest version

Update sgx to the latest stable version. Includes upgrading driver version, updating enclave api calls and documentation to support latest sgx version

┆Issue is synchronized with this Jira Task

<script>alert(1)</script>

xss test

Dont remove signing key in SGX signing procedure

Our current sgx signing script deletes the signing private key at the of the procedure

Since now we confirmed with Intel that signing keys are reusable, we need to make sure that we do not
remove the signing key at the completion of the signing script.

┆Issue is synchronized with this Jira Task

no response from sgx server

sometimes sgxwallet does not respond on zmq calls

┆Issue is synchronized with this Jira Bug
┆Attachments: skaleD-zmq.log

use zeromq everywhere in sgxwallet

use zeromq instead of json-rpc server for all methods. it will increase performance and remove possible memory leaks

┆Issue is synchronized with this Jira Story

Turn off automatic updates of Linux kernel

There was an issue on the mainnet - SGX Wallet was affected by Linux kernel update.
We should investigate the settings and advise validators how to turn off automatic updates

┆Issue is synchronized with this Jira Task

add multiplying function to support threshold encryption

• Add function
• Write tests
• Update documentation

┆Issue is synchronized with this Jira Task

SGX stuck on zmq check

Preconditions
Skaled: zmq-in-sgx-on-beta.2(custom build with latest consensus and latest beta(RC))
SGX: 1.72.1-develop.5
Release: 2.0
Network rinkeby

Step to reproduce
Run sgx with -c flag
Create schain

Actual result
Sgx stuck on
[2021-04-21 06:12:58.731] [error] Exception in zmq server worker:std:State check failed::_msg[_size - 1] == '}' ZMQMessage.cpp:65

┆Issue is synchronized with this Jira Bug

Re-enable client cert verification in JSON-RPC

Re-enable client cert verification in JSON-RPC

┆Issue is synchronized with this Jira Bug

Add a flag to sgxwallet to enable multithreading

By default sgxwallet is using one thread to process crypto operations.

Add a flag to enable multithreading, which leads to higher performance

┆Issue is synchronized with this Jira Bug

SGX server creates certificates that cant be verified in OpenSSL

SGX server creates certificates that cant be verified in OpenSSL

┆Issue is synchronized with this Jira Bug

SGX: request getBLSPublicKeyShareImpl failed

Preconditions
Versions
SGX: 1.70.0-beta.2

Step to reproduce
Create schain(type medium)
Observe admin log

Actual result
SGX log:
[2021-04-12 15:59:53.806] [info] BLS KEY SHARE CREATED
[2021-04-12 15:59:54.293] [info] Entering getBLSPublicKeyShareImpl
[2021-04-12 15:59:54.293] [error] getBLSPublicKeyShareImpl failed:
Log from admin added below

┆Issue is synchronized with this Jira Bug

Modify SGX Wallet release process

Automate “sgxwallet_release:latest“ release on each merge to stable branch. Includes signing secure enclave and building docker container from it.

┆Issue is synchronized with this Jira Task

Measure sgx performance for hardware and simulation modes

We have been having reports from QA that sometimes sgx server gets overloaded and stops handling requests.

We need to know how many BLS and ECDSA requests SGX server can do per second in hardware and simulation modes.

sgxwallet ram usage continuously grows on adding new bls/ecdsa keys

sgxwallet cache can use significant ram if there were created a lot of different bls / ecdsa keys. need to make this cache's size static

┆Issue is synchronized with this Jira Bug

Both IF statements prints the same output

sgxwallet/secure_enclave/secure_enclave.c

Line 163 in e0d95f3

LOG_INFO("SECURITY WARNING: sgxwallet is running in INSECURE DEBUG MODE! NEVER USE IN PRODUCTION!");

Can you please investigate why it's showing this output on production after successful registration?

make secure enclave multithreading

Intel introduced multithreading support in sdk 2.8. we use 2.13 now so we can use multithreading feature to increase sgxwallet performance under heavy load

┆Issue is synchronized with this Jira Story

sgxwallet exits with out-of-memory error

1 download and build sgxwallet
2 run it
3 crashes with out-of-memory error

(8G AWS instance!)

┆Issue is synchronized with this Jira Bug

move from c to c++ in secure enclave part

moving to c++ will increase code quality and readability as well as decrease risk of using invalid pointers

┆Issue is synchronized with this Jira Story

remove v1 dkg methods, replace them with v2 versions

v1 dkg methods are not used anymore, better to remove them

┆Issue is synchronized with this Jira Task

Fix Misconfigured thread count in simulation mode

32 total threads configured although only 25 can be supported

Move the rest of SGX wallet to ZeroMQ

Move the rest of SGX wallet to ZeroMQ.

Keet the JSON-RPC interface

┆Issue is synchronized with this Jira Task

create threshold encryption sample app

create threshold encryption sample app to illustrate methods usage and process

┆Issue is synchronized with this Jira Task

Skaled stuck use sgx with zmq

Preconditions
Skaled: zmq-in-sgx-on-beta.2(custom build with latest consensus and latest beta(RC))
SGX: 1.72.1-develop.5
Release: 2.0
Network rinkeby

Step to reproduce
Create schain

Actual result
Skaled stuck after start

NOTE: sgx cert issue

┆Issue is synchronized with this Jira Bug