Crash on _PyDict_CheckConsistency with an empty instance dict about cpython

[pablogsal]

@vstinner, can you check if this patch fixes the Cython crash as well? #121866

@colesbury I can confirm it works:

Before

../python/3.13/python.exe  runtests.py -vv dealloc_raise
Python 3.13.0b3+ (heads/3.13:cd74ed0a71f, Jul 16 2024, 19:24:17) [Clang 15.0.0 (clang-1500.3.9.4)]

Running tests against Cython 3.1.0a0 746627fdca5e48db2b63c8bb4be82b33ea617549
Using Cython language level 2.
Test dependency not found: 'numpy'
Test dependency not found: 'pythran'
/Users/pgalindo3/.local/lib/python3.13/site-packages/setuptools/sandbox.py:13: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  import pkg_resources
Test dependency found: 'setuptools.sandbox' version 69.1.1
Test dependency found: 'asyncio' version 3.13.0b3+
Test dependency found: 'pstats' version 3.13.0b3+
Test dependency found: 'posix' version 3.13.0b3+
Test dependency found: 'array' version ?.?
Test dependency found: 'Cython.Coverage' version 3.1.0a0
Test dependency found: 'Cython.Coverage' version 3.1.0a0
Test dependency not found: 'IPython.testing.globalipapp'
Test dependency not found: 'jedi_BROKEN_AND_DISABLED'
Test dependency found: 'test.support' version 3.13.0b3+
Backends: c,cpp

runTest (__main__.EndToEndTest.runTest)
[-1] End-to-end dealloc_raise ... [-1] ['/Users/pgalindo3/github/cython/../python/3.13/python.exe', 'setup.py', 'build_ext', '--inplace']
Compiling dealloc_raise.pyx because it changed.
[1/1] Cythonizing dealloc_raise.pyx

/Users/pgalindo3/github/cython/Cython/Compiler/Main.py:373: FutureWarning: Cython directive 'language_level' not set, using '3' (Py3). This has changed from earlier releases! File: /Users/pgalindo3/github/cython/TEST_TMP/run/dealloc_raise/dealloc_raise.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)


[-1] ['/Users/pgalindo3/github/cython/../python/3.13/python.exe', 'test_dealloc_raise1.py']

Objects/dictobject.c:716: _PyDict_CheckConsistency: Assertion failed: mp->ma_values->valid == 1
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x102c1ccb0
object refcount : 1
object type     : 0x1026e5f50
object type name: dict
object repr     : {}

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0x0000000208024c00 (most recent call first):
  File "/Users/pgalindo3/github/cython/TEST_TMP/run/dealloc_raise/test_dealloc_raise1.py", line 10 in <module>

Extension modules: dealloc_raise (total: 1)

Final directory layout of 'dealloc_raise':
./test_dealloc_raise1.py
./dealloc_raise.pyx
./dealloc_raise.c
./dealloc_raise.cpython-313d-darwin.so
./setup.py
./test_dealloc_raise2.py
./build/temp.macosx-14.5-arm64-cpython-313-pydebug/dealloc_raise.o
./build/lib.macosx-14.5-arm64-cpython-313-pydebug/dealloc_raise.cpython-313d-darwin.so

FAIL

======================================================================
FAIL: runTest (__main__.EndToEndTest.runTest)
[-1] End-to-end dealloc_raise
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/pgalindo3/github/cython/runtests.py", line 2040, in runTest
    self.assertEqual(0, res, "non-zero exit status, last output was:\n%r\n-- stdout:%s\n-- stderr:%s\n" % (
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        ' '.join(command), out[-1], err[-1]))
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 0 != -6 : non-zero exit status, last output was:
'/Users/pgalindo3/github/cython/../python/3.13/python.exe test_dealloc_raise1.py'
-- stdout:
-- stderr:Objects/dictobject.c:716: _PyDict_CheckConsistency: Assertion failed: mp->ma_values->valid == 1
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x102c1ccb0
object refcount : 1
object type     : 0x1026e5f50
object type name: dict
object repr     : {}

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0x0000000208024c00 (most recent call first):
  File "/Users/pgalindo3/github/cython/TEST_TMP/run/dealloc_raise/test_dealloc_raise1.py", line 10 in <module>

Extension modules: dealloc_raise (total: 1)



----------------------------------------------------------------------
Ran 1 test in 1.905s

FAILED (failures=1)
Most expensive pipeline stages:
Times:
etoe-build  :     1.18 sec  (   1,  1.177 / run) - slowest: 'c:dealloc_raise(1)' (1.18s)
etoe-run    :     0.25 sec  (   1,  0.253 / run) - slowest: 'c:dealloc_raise(2)' (0.25s)
ALL DONE

After

../python/3.13/python.exe  runtests.py -vv dealloc_raise
^[[APython 3.13.0b3+ (heads/3.13-dirty:cd74ed0a71f, Jul 16 2024, 19:21:10) [Clang 15.0.0 (clang-1500.3.9.4)]

Running tests against Cython 3.1.0a0 746627fdca5e48db2b63c8bb4be82b33ea617549
Using Cython language level 2.
Test dependency not found: 'numpy'
Test dependency not found: 'pythran'
/Users/pgalindo3/.local/lib/python3.13/site-packages/setuptools/sandbox.py:13: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  import pkg_resources
Test dependency found: 'setuptools.sandbox' version 69.1.1
Test dependency found: 'asyncio' version 3.13.0b3+
Test dependency found: 'pstats' version 3.13.0b3+
Test dependency found: 'posix' version 3.13.0b3+
Test dependency found: 'array' version ?.?
Test dependency found: 'Cython.Coverage' version 3.1.0a0
Test dependency found: 'Cython.Coverage' version 3.1.0a0
Test dependency not found: 'IPython.testing.globalipapp'
Test dependency not found: 'jedi_BROKEN_AND_DISABLED'
Test dependency found: 'test.support' version 3.13.0b3+
Backends: c,cpp

runTest (__main__.EndToEndTest.runTest)
[-1] End-to-end dealloc_raise ... ok

----------------------------------------------------------------------
Ran 1 test in 3.411s

OK
Most expensive pipeline stages: 'AnalyseDeclarationsTransform': 0.42 / 8 (0.053 / run, 53.7%), 'parse': 0.11 / 8 (0.014 / run, 14.4%), 'generate_pyx_code_stage': 0.07 / 1 (0.069 / run, 8.8%), 'AnalyseExpressionsTransform': 0.02 / 8 (0.002 / run, 1.9%), 'RemoveUnreachableCode': 0.01 / 16 (0.001 / run, 1.7%), 'OptimizeBuiltinCalls': 0.01 / 8 (0.002 / run, 1.6%), 'PostParse': 0.01 / 8 (0.001 / run, 1.2%), 'MarkClosureVisitor': 0.01 / 8 (0.001 / run, 1.2%), 'InterpretCompilerDirectives': 0.01 / 8 (0.001 / run, 1.2%), 'ControlFlowAnalysis': 0.01 / 8 (0.001 / run, 1.0%)
Times:
etoe-build  :     2.33 sec  (   1,  2.331 / run) - slowest: 'c:dealloc_raise(1)' (2.33s)
etoe-run    :     1.07 sec  (   2,  0.534 / run) - slowest: 'c:dealloc_raise(2)' (0.70s), 'c:dealloc_raise(3)' (0.37s)
ALL DONE

[colesbury]

I think this is fixed now

[vstinner]

Bug also seen in the Cython test suite: #121253

[pablogsal]

This also happens when I did compile
with ./configure --with-pydebug and I'm running with PYTHONDEVMODE=1 PYTHONMALLOC=malloc

[vstinner]

See also #121863 "CMultiplyNested closure crash: Python 3.13 regression". I don't know if it's related or not, but it was also found with the Cython test suite.

[colesbury]

Repro:

class Foo:
    pass

class StrSubclass(str):
    pass

foo = Foo()
foo.attr = 2
foo.__dict__ = {}
del foo.__dict__

getattr(foo, StrSubclass("attr"))

Objects/dictobject.c:716: _PyDict_CheckConsistency: Assertion failed: mp->ma_values->valid == 1
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x7f4aa716e210
object refcount : 1
object type     : 0x55dc45c3fc20
object type name: dict
object repr     : {}

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0x00007f4aa7695740 (most recent call first):
  File "/raid/sgross/cpython/repro.py", line 14 in <module>
Aborted (core dumped)

[colesbury]

_PyObject_MaterializeManagedDict_LockHeld is called to materialize the dictionary:

• The object doesn't have a materialized dictionary, (otherwise, we return it in _PyObject_MaterializeManagedDict)
• But the inline values may still not be valid! For example, if we materialize a dictionary and then delete it, the inline values are not valid, but there is no dictionary.

[vstinner]

Bug also seen in the Cython test suite: #121253

I can reliably reproduce the error on Linux by running the Cython test dealloc_raise:

$ ~/python/3.13/python runtests.py -vv dealloc_raise
(...)
'/home/vstinner/python/3.13/python test_dealloc_raise1.py'
-- stdout:
-- stderr:Objects/dictobject.c:716: _PyDict_CheckConsistency: Assertion failed: mp->ma_values->valid == 1
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x7fc6a6b7a3f0
object refcount : 1
object type     : 0xa5a100
object type name: dict
object repr     : {}

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0x00007fc6b48e3740 (most recent call first):
  File "/home/vstinner/dev/cython/TEST_TMP/run/dealloc_raise/test_dealloc_raise1.py", line 10 in <module>
(...)

[colesbury]

@vstinner, can you check if this patch fixes the Cython crash as well? #121866

[pablogsal]

Bisected to:

8b541c017ea92040add608b3e0ef8dc85e9e6060 is the first bad commit
commit 8b541c017ea92040add608b3e0ef8dc85e9e6060
Author: Dino Viehland <[email protected]>
Date:   Sun Apr 21 22:57:05 2024 -0700

    gh-112075: Make instance attributes stored in inline "dict" thread safe (#114742)

    Make instance attributes stored in inline "dict" thread safe on free-threaded builds

 Include/cpython/object.h                       |   1 +
 Include/internal/pycore_dict.h                 |  19 +-
 Include/internal/pycore_object.h               |  16 +-
 Include/internal/pycore_pyatomic_ft_wrappers.h |  14 +
 Lib/test/test_class.py                         |   9 +
 Objects/dictobject.c                           | 385 ++++++++++++++++++++-----
 Objects/object.c                               |  45 ++-
 Objects/typeobject.c                           |  30 +-
 Python/bytecodes.c                             |  15 +-
 Python/executor_cases.c.h                      |  10 +-
 Python/generated_cases.c.h                     |  13 +-
 Python/specialize.c                            |   3 +-
 Tools/cases_generator/analyzer.py              |   1 +
 13 files changed, 419 insertions(+), 142 deletions(-)
bisect found first bad commit