skalenetwork / sgxwallet Goto Github PK

SGX server creates certificates that cant be verified in OpenSSL

┆Issue is synchronized with this Jira Bug

use zeromq instead of json-rpc server for all methods. it will increase performance and remove possible memory leaks

┆Issue is synchronized with this Jira Story

┆Issue is synchronized with this Jira Task
┆Attachments: logs.log | sgx_keys.log

Preconditions
Skaled: zmq-in-sgx-on-beta.2(custom build with latest consensus and latest beta(RC))
SGX: 1.72.1-develop.5
Release: 2.0
Network rinkeby

Step to reproduce
Run sgx with -c flag
Create schain

Actual result
Sgx stuck on
[2021-04-21 06:12:58.731] [error] Exception in zmq server worker:std:State check failed::_msg[_size - 1] == '}' ZMQMessage.cpp:65

┆Issue is synchronized with this Jira Bug

sometimes sgxwallet does not respond on zmq calls

┆Issue is synchronized with this Jira Bug
┆Attachments: skaleD-zmq.log

Preconditions
SGX: custom version
sgx cmd: -s y V b c
Skaled: custom build based on latest beta

Step to reproduce
Open zmq port on sgx(1031)
Create first schain
Create second schain

Actual result
Second schain couldn't connect to SGX. But no error log form skaled, just stuck.

In case if we restart sgx and disable zmq, skaled not trying to reconnect to sgx. If we restart skaled on all nodes it will successful reconnected.

In case if we enable zmq and spin up new schain it will stuck. Case with restart all skaled with enabled zmq not help.

┆Issue is synchronized with this Jira Bug
┆Attachments: log

By default sgxwallet is using one thread to process crypto operations.

Add a flag to enable multithreading, which leads to higher performance

┆Issue is synchronized with this Jira Bug

Can you please investigate why it's showing this output on production after successful registration?

Add errror codes for ZMQ error processing

┆Issue is synchronized with this Jira Bug

Trying to install this on Debian 10 Buster (kernel 5.3.18-3-pve), so that for redundancy purposes I have two SGX wallets instead of one.

sgx_enable was built from source (make)

~/sgxwallet/sgx-software-enable# ll
total 76K
drwxr-xr-x  2 root root 4.0K May  6 15:26 .
drwxr-xr-x 22 root root 4.0K May  6 15:01 ..
-rw-r--r--  1 root root   44 May  6 15:01 .git
-rw-r--r--  1 root root  120 May  6 15:01 Makefile
-rw-r--r--  1 root root 5.7K May  6 15:01 README.md
-rw-r--r--  1 root root 3.5K May  6 15:01 sgx_capable.h
-rwxr-xr-x  1 root root  18K May  6 15:26 sgx_enable
-rw-r--r--  1 root root  13K May  6 15:01 sgx_enable.c
-rw-r--r--  1 root root  11K May  6 15:26 sgx_enable.o

~/sgxwallet/sgx-software-enable# ./sgx_enable
Intel SGX is already enabled on this system

~/sgxwallet/sgx-software-enable# cd ../scripts/

~/sgxwallet/scripts# ./sgx_linux_x64_driver_2.5.0_2605efa.bin
Unpacking Intel SGX Driver ... done.
Verifying the integrity of the install package ... done.
Installing Intel SGX Driver ...
/tmp/sgx-driver-znUl6F ~/sgxwallet/scripts
install -d /opt/intel/sgxdriver/package
install -d /opt/intel/sgxdriver/scripts
install package/* /opt/intel/sgxdriver/package
install scripts/* /opt/intel/sgxdriver/scripts
~/sgxwallet/scripts
/opt/intel/sgxdriver/package ~/sgxwallet/scripts
make -C /lib/modules/5.3.18-3-pve/build SUBDIRS=/opt/intel/sgxdriver/package modules
make[1]: Entering directory '/usr/src/linux-headers-5.3.18-3-pve'
Makefile:227: ================= WARNING ================
Makefile:228: 'SUBDIRS' will be removed after Linux 5.3
Makefile:229:
Makefile:230: If you are building an individual subdirectory
Makefile:231: in the kernel tree, you can do like this:
Makefile:232: $ make path/to/dir/you/want/to/build/
Makefile:233: (Do not forget the trailing slash)
Makefile:234:
Makefile:235: If you are building an external module,
Makefile:236: Please use 'M=' or 'KBUILD_EXTMOD' instead
Makefile:237: ==========================================
  CC [M]  /opt/intel/sgxdriver/package/sgx_main.o
  CC [M]  /opt/intel/sgxdriver/package/sgx_page_cache.o
/opt/intel/sgxdriver/package/sgx_page_cache.c: In function ‘sgx_test_and_clear_young’:
/opt/intel/sgxdriver/package/sgx_page_cache.c:126:8: error: passing argument 4 of ‘apply_to_page_range’ from incompatible pointer type [**-Werror=incompatible-pointer-types**]
        **sgx_test_and_clear_young_cb**, vma->vm_mm);
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /opt/intel/sgxdriver/package/sgx.h:72,
                 from /opt/intel/sgxdriver/package/sgx_page_cache.c:61:
./include/linux/mm.h:2679:40: note: expected ‘pte_fn_t’ {aka ‘int (*)(struct <anonymous> *, long unsigned int,  void *)’} but argument is of type ‘int (*)(pte_t *, struct page *, long unsigned int,  void *)’ {aka ‘int (*)(struct <anonymous> *, struct page *, long unsigned int,  void *)’}
           unsigned long size, pte_fn_t fn, void *data);
                               ~~~~~~~~~^~
cc1: some warnings being treated as errors
make[2]: *** [scripts/Makefile.build:290: /opt/intel/sgxdriver/package/sgx_page_cache.o] Error 1
make[1]: *** [Makefile:1655: _module_/opt/intel/sgxdriver/package] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-5.3.18-3-pve'
make: *** [Makefile:16: default] Error 2

Maybe your team has some suggestions to get around this? I don't want to install the driver directly from https://github.com/intel/linux-sgx-driver, unless instructed to do so.

PS: libprotobuf10 was installed by downloading the Debian Stretch package, unpacking it to a temp folder (not to overwrite any permissions), then copying ibprotobuf.so.10.0.0 (along with the symlink) to /usr/lib/x86_64-linux-gnu

sgxwallet cannot sign message with 0x prefix

┆Issue is synchronized with this Jira Bug

create threshold encryption sample app to illustrate methods usage and process

┆Issue is synchronized with this Jira Task

Make getDecryptionShare in SGX to take an array of encrypted values and return an array of values.

This is because a typical TE encrypted block will include many transactions and all of them need to be

decrypted in a single call to be efficient.

If a particular value is corrupt and can not be decrypted, return error for this value, but decrypt other value.

Otherwise an attacker can cause a DoS attack by submitting a corrupt value in a transaction.

┆Issue is synchronized with this Jira Task

we can't get enough logs from sgx in case of some errors inside enclave. lets fix this this and ensure sgx always prints error messages if they appear

┆Issue is synchronized with this Jira Bug

make sure sgxwallet memory consumption never hits 100% - always restart sgxwallet if current usage more than 80%

┆Issue is synchronized with this Jira Bug

warn validators if using LInux Kernel v 5

┆Issue is synchronized with this Jira Task

curl --cert /skale_node_data/sgx_certs/sgx.crt --key /skale_node_data/sgx_certs/sgx.key X POST data '{"id":1, "jsonrpc":"2.0","method":"importBLSKeyShare","params":{"keyShareName":"BLS_KEY:SCHAIN_ID:1625669718:NODE_ID:1:DKG_ID:0","keyShare":"0xBDCCBF9FA66FB00A44A00D614E68A294B1B8B41185A1460C64177B348A9722"}}' H 'content-type:application/json;' https://45.76.3.64:1026 k

curl: (52) Empty reply from server

Probably it could return descriptive error message?!

┆Issue is synchronized with this Jira Bug

moving to c++ will increase code quality and readability as well as decrease risk of using invalid pointers

┆Issue is synchronized with this Jira Story

Intel introduced multithreading support in sdk 2.8. we use 2.13 now so we can use multithreading feature to increase sgxwallet performance under heavy load

┆Issue is synchronized with this Jira Story

• Add function
• Write tests
• Update documentation

┆Issue is synchronized with this Jira Task

Preconditions
Versions
SGX: 1.70.0-beta.2

Step to reproduce
Create schain(type medium)
Observe admin log

Actual result
SGX log:
[2021-04-12 15:59:53.806] [info] BLS KEY SHARE CREATED
[2021-04-12 15:59:54.293] [info] Entering getBLSPublicKeyShareImpl
[2021-04-12 15:59:54.293] [error] getBLSPublicKeyShareImpl failed:
Log from admin added below

┆Issue is synchronized with this Jira Bug

Move the rest of SGX wallet to ZeroMQ.

Keet the JSON-RPC interface

┆Issue is synchronized with this Jira Task

Preconditions
SGX wallet: 1.70.0-beta.1
20 nodes up and running
4 schains up and running, 3 of them empty blocks, last one under load(23 tx per)

Step to reproduce
Spin up schains, make load.(skaled tests like cat-cycle) or just empty blocks will enough.

Actual result
SGX restarted by OOM kill and up after 6h.
Full reboot log attached.

[Sat Mar 20 17:51:09 2021] [23474]     0 23474    16397      122   172032        0             0 cron
[Sat Mar 20 17:51:09 2021] Out of memory: Kill process 24368 (sgxwallet) score 838 or sacrifice child
[Sat Mar 20 17:51:09 2021] Killed process 24368 (sgxwallet) total-vm:41523820kB, anon-rss:27389556kB, file-rss:0kB, shmem-rss:0kB

NOTE: Let's gracefully turn off sgx not force. [~accountid:5b293e1e9bcd7518af2198f6][~accountid:5b2037f147afa959b14789e5][~accountid:5ec633056c50620c1ca56e20][~accountid:5b88195a9d17912bf10aef4c]

┆Issue is synchronized with this [Jira Bug](https://skalelabs.atlassian.net/browse/SKALE-4005)
┆Attachments: <a href="https://skalelabs.atlassian.net/rest/api/2/attachment/content/10553">sgxRebootStuck.log</a>

Is this the expected behavior? I see some issues related to RNG
Using Ubuntu 18.04.4 LTS

sgxwallet_1  | Running sgxwallet version:1.45.1
sgxwallet_1  | [2020-03-28 16:00:06.761] [debug] SGX_DEBUG_FLAG = 1
sgxwallet_1  | [2020-03-28 16:00:06.946] [info] Enclave created and started successfully
sgxwallet_1  | [2020-03-28 16:00:06.946] [info] Enclave libtgmp library initialized successfully
sgxwallet_1  | ***ENCLAVE_LOG***: Initing tgmp library
sgxwallet_1  | [2020-03-28 16:00:06.947] [info] Initing wallet database ... 
sgxwallet_1  | [2020-03-28 16:00:06.947] [info] Opening wallet databases
sgxwallet_1  | [2020-03-28 16:00:07.025] [info] Successfully opened databases
sgxwallet_1  | [2020-03-28 16:00:07.025] [debug] key to read from db: SEK
sgxwallet_1  | [2020-03-28 16:00:07.025] [error] SEK was not created yet. Going to create SEK
sgxwallet_1  | ATTENTION! THIS IS YOUR KEY FOR BACK UP. PLEASE COPY IT TO THE SAFE PLACE
sgxwallet_1  | key is balblabla49e3b86908796857
sgxwallet_1  | reset: terminal attributes: No such device or address
sgxwallet_1  | 
sgxwallet_1  | [2020-03-28 16:00:07.028] [debug] key to read from db: SEK
sgxwallet_1  | Could not execute reset
sgxwallet_1  | [2020-03-28 16:00:07.028] [debug] written key: SEK
sgxwallet_1  | [2020-03-28 16:00:07.028] [debug] SEK
sgxwallet_1  | [2020-03-28 16:00:07.028] [debug] key to read from db: TEST_KEY
sgxwallet_1  | [2020-03-28 16:00:07.028] [debug] written key: TEST_KEY
sgxwallet_1  | [2020-03-28 16:00:07.028] [debug] TEST_KEY
sgxwallet_1  | [2020-03-28 16:00:07.028] [info] NO ROOT CA CERTIFICATE YET. CREATING ...
sgxwallet_1  | Generating RSA private key, 2048 bit long modulus (2 primes)
sgxwallet_1  | ....................+++++
sgxwallet_1  | ..........................+++++
sgxwallet_1  | e is 65537 (0x010001)
sgxwallet_1  | Can't load /root/.rnd into RNG
sgxwallet_1  | 139851374010816:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/root/.rnd

sometimes skale-node cannot create ssl certificate on sgx due to internal openssl error. need to investigate it to prevent it in future

┆Issue is synchronized with this Jira Bug

Fix zeromq close

┆Issue is synchronized with this Jira Bug

sgxwallet doesn't print error message in some cases

┆Issue is synchronized with this Jira Bug

Preconditions
Skaled: 3.5.8-asan-plus-snapshots
SGX: 1.70.0-beta.0

Step to reproduce
Create schan medium type
Observe logs

Actual result
Skaled stuck on processNextAvailableConnection for 3h on 3 of 16 schain

┆Issue is synchronized with this Jira Bug
┆Attachments: file

validators have issues with it

┆Issue is synchronized with this Jira Task

Preconditions
Skaled: zmq-in-sgx-on-beta.2(custom build with latest consensus and latest beta(RC))
SGX: 1.72.1-develop.5
Release: 2.0
Network rinkeby

Step to reproduce
Create schain

Actual result
Skaled stuck after start

NOTE: sgx cert issue

┆Issue is synchronized with this Jira Bug

Complete and enable ZeroMQ integration

┆Issue is synchronized with this Jira Task

this tool seems not working properly. review the code and usage, add documentation

┆Issue is synchronized with this Jira Task

Fix automated sgxwallet signed build and please actually test that it works.

┆Issue is synchronized with this Jira Task

use https://github.com/intel/ipp-crypto to do ECDSA signing, ECDH key exchange and finite field arithmetic for better stability and performance

┆Issue is synchronized with this Jira Story

When DKG time on admin is over, admin terminate process which responsible for DKG waiting from contract. And when DKG on contract successfully over, admin do not know about and did not ask contract about DKG

versions:
schain:3.7.1-develop.7
admin:2.0.0-develop.36

STR:
create 6+ MEDIUM schains
expected: all schains containers works and all schains block mining
actual: sgx stuck with Exception in zmq server worker:std:State check failed::_msg[_size - 1] == '}' ZMQMessage.cpp:65 after some period of schains working ~ 260 blocks

┆Issue is synchronized with this Jira Bug
┆Attachments: schain_anz.txt | sgx_logs_after_up.txt

1 download and build sgxwallet
2 run it
3 crashes with out-of-memory error

(8G AWS instance!)

┆Issue is synchronized with this Jira Bug

sgxwallet cache can use significant ram if there were created a lot of different bls / ecdsa keys. need to make this cache's size static

┆Issue is synchronized with this Jira Bug

Versions:
schain:3.7.5-beta.4
admin:2.0.1-beta.7
sgx_wallet:1.77.0

Preconditons:

• Schain 16/16 MEDIUM type
• 1 Active node without schains
• 1 snapshot on schain
• Snapshot interval 2 hours
• Node rotation delay 1 hour

Steps to reproduce:

1. Call node exit on the node A and wait till node B will complete BLS keys, skaled container
2. Skaled on the node B starts catchup from 0 block and couldn't download any blocks (old blocks signed with old signatures) but not crashed
3. Wait snapshot creation - 15 nodes should create snapshot with new BLS keys
4. Restart skaled on the node B with repair mode

Actual state: 15 nodes sign old snapshot with new BLS keys but node B couldn't check incoming snapshot with own BLS key
Expected state: Node B should be able to check incoming snapshot with valid BLS key

┆Issue is synchronized with this Jira Bug
┆Attachments: quiet-cursa.log

Re-enable client cert verification in JSON-RPC

┆Issue is synchronized with this Jira Bug

Update sgx to the latest stable version. Includes upgrading driver version, updating enclave api calls and documentation to support latest sgx version

┆Issue is synchronized with this Jira Task

v1 dkg methods are not used anymore, better to remove them

┆Issue is synchronized with this Jira Task

We have been having reports from QA that sometimes sgx server gets overloaded and stops handling requests.

We need to know how many BLS and ECDSA requests SGX server can do per second in hardware and simulation modes.

https://www.tenable.com/plugins/nessus/124210

┆Issue is synchronized with this Jira Bug

Automate “sgxwallet_release:latest“ release on each merge to stable branch. Includes signing secure enclave and building docker container from it.

┆Issue is synchronized with this Jira Task

xss test

There was an issue on the mainnet - SGX Wallet was affected by Linux kernel update.
We should investigate the settings and advise validators how to turn off automatic updates

┆Issue is synchronized with this Jira Task

Our current sgx signing script deletes the signing private key at the of the procedure

Since now we confirmed with Intel that signing keys are reusable, we need to make sure that we do not
remove the signing key at the completion of the signing script.

┆Issue is synchronized with this Jira Task

and to reencrypt db

32 total threads configured although only 25 can be supported

The is_well_formed check in libff::alt_bn128_G2 is insufficient. It checks that the projective coordinates (X,Y,Z) satisfy the equation of the curve, but G2 is actually a subgroup of the elliptic curve points. An attacker that controls n-t participants, where n are all participants and t+1 is the number of participants required to sign a message with the common public key, will always be able to corrupt the public key so it is almost impossible to create a common signature, allowing the attacker to escape later detection during the signing phase. With a probability of 1/10069 the attacker can do the same with n-t participants.

┆Issue is synchronized with this Jira Bug