RSA-PSS on Windows fail to verify signature with salt len != hash len about oscrypto

[vxgmichel]

Note that the problem can be reproduced on linux by switching from libcrypto 3 to libcrypto 1 between signing and verifying.

Consider those two programs, sign.py and verify.py:

# sign.py

import sys
import oscrypto
base_path = sys.argv[1]
oscrypto.use_openssl(f"{base_path}/libcrypto.so", f"{base_path}/libssl.so")

import oscrypto.asymmetric
print("Libcrypto version:", *oscrypto._openssl.asymmetric.libcrypto_version_info, file=sys.stderr)


sign_key = b"""
-----BEGIN PRIVATE KEY-----
MIIEuwIBADALBgkqhkiG9w0BAQoEggSnMIIEowIBAAKCAQEArHkxb5OQOh5BiPS1
Y+EIANr7OZVyqi7fSw69mjIxjgRqnz/WjG3wuBTLcDG1vuXpu7Ojg/eADo1ZUdZL
lgSPe1mUmF8rFUW6E0j6GBUBJS4mSnqLKjjdqKPs1RjtAX3HD6juCc5un/oyeGcj
MWh1+ixuYoRgIduHpmc5kSLxpjwy2c2ZUYWR+15uKbi7Uqrn8CzmghhPF7u1t5jJ
k+mbRLU+NwqjKP+k02rrHKfsENBu+2yhugCfgUHGz51fT4N3lgbK41WB4vO54kIJ
4SaN2UDElmvJeYJVl1rPXLMVz+1ChfrIFdaVxFEF7Yn82+cgoU8ejrqejyea3/M9
5/2k0QIDAQABAoIBAD85pDozPX1YpwyGLFKDZTQvEkZPNzwUt61jp3S1rr3Rd6aO
N9+9068fjF5CEs56qN6yoSAY5Dwxa8tYw9eoL1L4CUV8KaaAK5CzQV7/oC5Zhxbp
akedlgAiq4iIvSU9TvI6Kpy0rI//n23M3TVZBlqh3AtIXJc8yzLgh1VhmnUl5Gom
i36jmW2BF1wwuqjVSKCvml/RAo2wVELwBm3WXhPxqwYGZStfonEB/MwsHGfbvFcg
MQRCClpLGKXdspam3uQvMYC0lLRED13lL1qB8pV0YFBHpNIdWlG3QYKxcwwcCoE2
OEldjRIugR9XE+t/EyQnzfkeJ3DK74h78CF7rfsCgYEAvjz8FJ85Zb2cCWCwSbBF
QR6VzSvPeHlI8z5DFFtL/nbclhUB3z12uuE2JHMwCox4HLCNYWDoOXUmWwypYJ1X
YV2VZzZyPiAuNFTQ4UHhOk6w6bChiLCjnSCet0MoV/Ukaamu49Rxq7gDy2XkIWoD
wD+9xrdD7lmP0t4YykW7IqMCgYEA6BgfUJD/SQbZkO8+YpUYC8LmFZkEg8txj7kR
wISLsLzdGxy3xh95JS96SofvfGpI5oFcMGAjX3a6vzZ2qBvQa3+IOQQWqJULdaTW
FRPPWYv9gBJDLz4SOH9pv8ZEdBbcZ5N+Y7FEBp0eTXjh1J56NonpQIRXf4ojIlvI
fXmshfsCgYBOO3gS5vPMsi/j714vv4yLXg+Oo1Cbo4zrcxRU38Kdr7XBBnyRmI4m
Bg2k6bW88M1IRxatEBQP5OxUDx3sfGf9w2V4X3yVrdgybxrDN7tupgO85oVXWATA
zjRW+wgxO7+wsDYavTfNvUvaLlmloBpQyiW5/Y2zDCPIPMuHCywM7wKBgQDdy7eZ
QYeEnRQrSkZe5UYebzl7qEhFPpUemOibBs+LrWDK+Q2yOv+FhrKiKPe2+McD6NlV
rXoAT7E06/JGwpXRNQXUHtEcd5qE6WpgqBa952bxDgLAUdwNu80uJGXkXrhwDuZ4
lL2CaIG93WhKzMvT9MVAD3iifDsJKZcWOcGiIwKBgD40p/yLSdU07L5xoJi/XhZi
lSjkwVVjelZrebI6fczcFijpIgxu0Jns21NEDQE/qPE6Mc5atCh2DenM55XXI+sa
LiSbnD7yIMXNuAS63Q+TCSDdFhTE6UHi84+1WqON8kOZrQ+2bzT014mZN+2fNXfR
jyj0su/eWkS3DmtTie1z
-----END PRIVATE KEY-----
"""

content = b"hello world\n"
sign_key = oscrypto.asymmetric.load_private_key(sign_key)
signature = oscrypto.asymmetric.rsa_pss_sign(sign_key, content, "sha256")
print(signature)

# verify.py

import sys
import oscrypto
base_path = sys.argv[1]
oscrypto.use_openssl(f"{base_path}/libcrypto.so", f"{base_path}/libssl.so")

import oscrypto.asymmetric
print("Libcrypto version:", *oscrypto._openssl.asymmetric.libcrypto_version_info, file=sys.stderr)


verify_key = b"""-----BEGIN PUBLIC KEY-----
MIIBIDALBgkqhkiG9w0BAQoDggEPADCCAQoCggEBAKx5MW+TkDoeQYj0tWPhCADa
+zmVcqou30sOvZoyMY4Eap8/1oxt8LgUy3Axtb7l6buzo4P3gA6NWVHWS5YEj3tZ
lJhfKxVFuhNI+hgVASUuJkp6iyo43aij7NUY7QF9xw+o7gnObp/6MnhnIzFodfos
bmKEYCHbh6ZnOZEi8aY8MtnNmVGFkftebim4u1Kq5/As5oIYTxe7tbeYyZPpm0S1
PjcKoyj/pNNq6xyn7BDQbvtsoboAn4FBxs+dX0+Dd5YGyuNVgeLzueJCCeEmjdlA
xJZryXmCVZdaz1yzFc/tQoX6yBXWlcRRBe2J/NvnIKFPHo66no8nmt/zPef9pNEC
AwEAAQ==
-----END PUBLIC KEY-----
"""

content = b"hello world\n"
verify_key = oscrypto.asymmetric.load_public_key(verify_key)
signature = eval(input())
result = oscrypto.asymmetric.rsa_pss_verify(verify_key, signature, content, "sha256")

See what happens when running those scripts with different combination of libcrypto:

$ python sign.py /path/to/libcrypto_v1 | python verify.py /path/to/libcrypto_v1 
Libcrypto version: 1 1 1 q
Libcrypto version: 1 1 1 q

$ python sign.py /path/to/libcrypto_v3 | python verify.py /path/to/libcrypto_v3                           
Libcrypto version: 3 0 2
Libcrypto version: 3 0 2

$ python sign.py /path/to/libcrypto_v1 | python verify.py /path/to/libcrypto_v3
Libcrypto version: 1 1 1 q
Libcrypto version: 3 0 2

$ python sign.py /path/to/libcrypto_v3 | python verify.py /path/to/libcrypto_v1
Libcrypto version: 1 1 1 q
Libcrypto version: 3 0 2
Libcrypto version: 1 1 1 q
Libcrypto version: 3 0 2
Traceback (most recent call last):
  File "/home/vinmic/repos/verify.py", line 25, in <module>
    result = oscrypto.asymmetric.rsa_pss_verify(verify_key, signature, content, "sha256")
  File "/home/vinmic/miniconda/envs/parsec/lib/python3.9/site-packages/oscrypto/_openssl/asymmetric.py", line 1160, in rsa_pss_verify
    return _verify(certificate_or_public_key, signature, data, hash_algorithm, rsa_pss_padding=True)
  File "/home/vinmic/miniconda/envs/parsec/lib/python3.9/site-packages/oscrypto/_openssl/asymmetric.py", line 1486, in _verify
    raise SignatureError('Signature is invalid')
oscrypto.errors.SignatureError: Signature is invalid

In my opinion, the bug is due to this block of code:

oscrypto/oscrypto/_openssl/asymmetric.py

Lines 1890 to 1900 in e90100e

	if libcrypto_version_info < (3, 0):
	res = libcrypto.EVP_PKEY_CTX_ctrl(
	evp_pkey_ctx_pointer,
	LibcryptoConst.EVP_PKEY_RSA,
	LibcryptoConst.EVP_PKEY_OP_SIGN | LibcryptoConst.EVP_PKEY_OP_VERIFY,
	LibcryptoConst.EVP_PKEY_CTRL_RSA_PSS_SALTLEN,
	-1,
	null()
	)
	handle_openssl_error(res)

Note how the salt length is only set if the version for libcrypto is less than 3. When libcrypto is at version 3, the default configuration is used during the signing which uses the maximum admissible length. This is not the expected behavior for oscrypto as the docs for rsa_pss_sign explicitly states:

The salt length with be the length of the hash algorithm