C_OpenSession deadlocks if C_INITIALIZE(.flags=CKF_OS_LOCKING_OK) is called about tpm2-pkcs11

[PeterHuewe]

Okay the problem lies in the following order:
src/lib/session.c , session_open line 70:
session_table_lock(global.s_table);
session_table_lock now holds global.s_table mutex
line 71:
rv = check_max_sessions(is_rw);
which calls

static CK_RV check_max_sessions(bool is_rw) {

    unsigned long cnt = session_table_get_cnt(global.s_table, is_rw);
    return (cnt > MAX_NUM_OF_SESSIONS) ?
        CKR_SESSION_COUNT : CKR_OK;
}

which calls

unsigned long session_table_get_cnt(session_table *t, bool is_rw) {
    session_table_lock(t);
    unsigned long tmp = is_rw ? t->rw_cnt : t->cnt;
    session_table_unlock(t);
    return tmp;
}

--> session_table_get_cnt wants to take the mutex again on global.s_table (which is already held)
-> deadlock.

Question is whether the locking in session_open can simply be removed?

[williamcroberts]

I think you need to remove the locking from session_table_get_cnt(). The lock on the session table is required when adding an entry via session_table_new_ctx_unlocked(). Probably make a new session_table_get_count_unlocked() or something like that.

I am also OK with a general refactoring of the session code layout, not happy with how that currently turned out.

FYI: I have a --debug in configure that should use a PTHREAD_MUTEX_ERRORCHECK attributed thread.

[williamcroberts]

This seems to be fixed in #47 and #48