PKCS 11 provider stress tests sometimes fail about parsec

[ionut-arm]

I've re-enabled the tests in #221 but have left the code to disable them there, just in case

[hug-dev]

We have seen Travis CI failing again 😢 We disabled them again in #229

[hug-dev]

Appeared today with a different error:

[ERROR parsec_service::providers::pkcs11_provider] Login operation failed with PKCS#11: CKR_SESSION_HANDLE_INVALID (0xb3)

[ERROR parsec_service::providers::pkcs11_provider] Failed to close session 18092 due to error PKCS#11: CKR_SESSION_HANDLE_INVALID (0xb3). Continuing...

thread '<unnamed>' panicked at 'Failed to export key: PsaErrorInvalidHandle', /root/.cargo/git/checkouts/parsec-client-test-9c23f39b40cb6dae/fb15945/src/stress_test_client.rs:224:25

[anta5010]

It panicked once for me while running in a container on a MacOS laptop:

test per_provider::persistent_after::should_have_been_deleted ... ok
test per_provider::persistent_after::reuse_to_sign ... FAILED

failures:

---- per_provider::persistent_after::reuse_to_sign stdout ----
Error: PsaErrorDoesNotExist
thread 'per_provider::persistent_after::reuse_to_sign' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `0`: the test returned a termination value with a non-zero status code (1) which indicates a failure', src/libtest/lib.rs:197:5
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:84
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:61
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1025
   5: std::io::Write::write_fmt
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libstd/io/mod.rs:1426
   6: std::io::impls::<impl std::io::Write for alloc::boxed::Box<W>>::write_fmt
             at src/libstd/io/impls.rs:156
   7: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:65
   8: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:50
   9: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:193
  10: std::panicking::default_hook
             at src/libstd/panicking.rs:207
  11: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:471
  12: rust_begin_unwind
             at src/libstd/panicking.rs:375
  13: std::panicking::begin_panic_fmt
             at src/libstd/panicking.rs:326
  14: test::assert_test_result
             at ./<::std::macros::panic macros>:9
  15: mod::per_provider::persistent_after::reuse_to_sign::{{closure}}
             at tests/per_provider/persistent_after.rs:27
  16: core::ops::function::FnOnce::call_once
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libcore/ops/function.rs:232
  17: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/liballoc/boxed.rs:1022
  18: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:78
  19: std::panicking::try
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libstd/panicking.rs:270
  20: std::panic::catch_unwind
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libstd/panic.rs:394
  21: test::run_test_in_process
             at src/libtest/lib.rs:567
  22: test::run_test::run_test_inner::{{closure}}
             at src/libtest/lib.rs:474
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.


failures:
    per_provider::persistent_after::reuse_to_sign

test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 33 filtered out

error: test failed, to rerun pass '--test mod'

[hug-dev]

Happened with the stress tests of the Mbed Crypto provider, similar issue, still on the Arm machine. We might thing to disable stress testing on Travis CI until this is investigated.

See logs included.

log.txt

[ionut-arm]

After a few "tests" with a practice branch, I found that the failure only occurs on Arm platforms (I have only tried AMD64 as an alternative, though). A bit worrying, but difficult to tell what's going on. It also started failing sometimes on a persistent_after test, as seen here. The test to fail is always reuse_to_sign:

[ERROR parsec_service::front::front_end] Failed to send response; error: generic input/output error

test per_provider::persistent_after::reuse_to_sign ... FAILED

failures:

---- per_provider::persistent_after::reuse_to_sign stdout ----

Error: ConnectionError

thread 'per_provider::persistent_after::reuse_to_sign' panicked at 'assertion failed: `(left == right)`

  left: `1`,

 right: `0`: the test returned a termination value with a non-zero status code (1) which indicates a failure', <::std::macros::panic macros>:5:6

stack backtrace:

   0: backtrace::backtrace::libunwind::trace

             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88

   1: backtrace::backtrace::trace_unsynchronized

             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66

   2: std::sys_common::backtrace::_print_fmt

             at src/libstd/sys_common/backtrace.rs:77

   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt

             at src/libstd/sys_common/backtrace.rs:59

   4: core::fmt::write

             at src/libcore/fmt/mod.rs:1052

   5: std::io::Write::write_fmt

             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libstd/io/mod.rs:1426

   6: std::io::impls::<impl std::io::Write for alloc::boxed::Box<W>>::write_fmt

             at src/libstd/io/impls.rs:156

   7: std::sys_common::backtrace::_print

             at src/libstd/sys_common/backtrace.rs:62

   8: std::sys_common::backtrace::print

             at src/libstd/sys_common/backtrace.rs:49

   9: std::panicking::default_hook::{{closure}}

             at src/libstd/panicking.rs:204

  10: std::panicking::default_hook

             at src/libstd/panicking.rs:221

  11: std::panicking::rust_panic_with_hook

             at src/libstd/panicking.rs:472

  12: rust_begin_unwind

             at src/libstd/panicking.rs:380

  13: std::panicking::begin_panic_fmt

             at src/libstd/panicking.rs:334

  14: test::assert_test_result

             at ./<::std::macros::panic macros>:5

  15: mod::per_provider::persistent_after::reuse_to_sign::{{closure}}

             at tests/per_provider/persistent_after.rs:27

  16: core::ops::function::FnOnce::call_once

             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libcore/ops/function.rs:232

  17: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once

             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/liballoc/boxed.rs:1015

  18: __rust_maybe_catch_panic

             at src/libpanic_unwind/lib.rs:86

  19: std::panicking::try

             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libstd/panicking.rs:281

  20: std::panic::catch_unwind

             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libstd/panic.rs:394

  21: test::run_test_in_process

             at src/libtest/lib.rs:539

  22: test::run_test::run_test_inner::{{closure}}

             at src/libtest/lib.rs:452

note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[hug-dev]

Disabling stress tests does not seem to make things better...
We might want to use the allow_failures option on Travis:

# Executing our tests on Arm64 with Travis CI
# The TPM provider Dockerfile does not build on Arm so the all-providers and tpm-provider tests
# are not executed on Aarch64.
arch: arm64
services:
    - docker
jobs:
    include:
    - name: "Integration tests using Mbed Crypto provider"
      env:
      - DOCKER_IMAGE_NAME=mbed-crypto-provider
      - DOCKER_IMAGE_PATH=tests/per_provider/provider_cfg/mbed-crypto
      # Stress test fails on Travis CI, see #116
      - SCRIPT="tests/ci.sh --no-stress-test mbed-crypto"
    allow_failures:
    - name: "Integration tests using PKCS 11 provider"
      env:
      - DOCKER_IMAGE_NAME=pkcs11-provider
      - DOCKER_IMAGE_PATH=tests/per_provider/provider_cfg/pkcs11
      # Stress test fails on Travis CI, see #116
      - SCRIPT="tests/ci.sh --no-stress-test pkcs11"
script:
- docker build -t $DOCKER_IMAGE_NAME $DOCKER_IMAGE_PATH
- docker run -v $(pwd):/tmp/parsec -w /tmp/parsec $DOCKER_IMAGE_NAME /tmp/parsec/$SCRIPT

[hug-dev]

It seems like the last build which failed with that same error is from May 14th so maybe the issue is now fixed 🤓 ?

[hug-dev]

We disabled all Travis builds for now