Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash running PyO3 tests with --test-threads=1000 #124375

Closed
ngoldbaum opened this issue Sep 23, 2024 · 1 comment
Closed

Crash running PyO3 tests with --test-threads=1000 #124375

ngoldbaum opened this issue Sep 23, 2024 · 1 comment
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-free-threading type-crash A hard crash of the interpreter, possibly with a core dump

Comments

@ngoldbaum
Copy link
Contributor

ngoldbaum commented Sep 23, 2024 •
edited by bedevere-app bot
Loading

Crash report

What happened?

Unfortunately I don't know how to make this more minimal than "run the PyO3 tests".

You'll need rust 1.81 installed along with a copy of the latest version of PyO3 from github and a free-threaded Python interpreter.

Repeatedly run the PyO3 tests with UNSAFE_PYO3_BUILD_FREE_THREADED=1 cargo test --lib -- --test-threads=1000, and eventually you'll see a seg fault. If you run the test executable under a debugger, you'll see a C traceback like this, crashing on an atomic load inside the GC internals:

* thread #2, name = 'buffer::tests::test_array_buffer', stop reason = EXC_BAD_ACCESS (code=1, address=0x8)
  * frame #0: 0x0000000101a1d7d4 libpython3.13td.dylib`_Py_qbsr_goal_reached(qsbr=0x0000000000000000, goal=9) at pycore_qsbr.h:103:53
    frame #1: 0x0000000101a1d714 libpython3.13td.dylib`_Py_qsbr_poll(qsbr=0x0000000000000000, goal=9) at qsbr.c:165:9
    frame #2: 0x00000001017dd814 libpython3.13td.dylib`process_queue(head=0x0000000101d148d8, qsbr=0x0000000000000000, keep_empty=false) at obmalloc.c:1188:18
    frame #3: 0x00000001017dd958 libpython3.13td.dylib`process_interp_queue(queue=0x0000000101d148d0, qsbr=0x0000000000000000) at obmalloc.c:1218:9
    frame #4: 0x00000001017dd778 libpython3.13td.dylib`_PyMem_ProcessDelayed(tstate=0x000000013f91f610) at obmalloc.c:1237:5
    frame #5: 0x00000001019a8a98 libpython3.13td.dylib`process_delayed_frees(interp=0x0000000101cff940) at gc_free_threading.c:351:9
    frame #6: 0x00000001019a86f8 libpython3.13td.dylib`gc_collect_internal(interp=0x0000000101cff940, state=0x000000016fff6698, generation=0) at gc_free_threading.c:1109:5
    frame #7: 0x00000001019a5d20 libpython3.13td.dylib`gc_collect_main(tstate=0x0000000101d2fe40, generation=0, reason=_Py_GC_REASON_HEAP) at gc_free_threading.c:1225:5
    frame #8: 0x00000001019a6894 libpython3.13td.dylib`_Py_RunGC(tstate=0x0000000101d2fe40) at gc_free_threading.c:1684:5
    frame #9: 0x00000001019b8944 libpython3.13td.dylib`_Py_HandlePending(tstate=0x0000000101d2fe40) at ceval_gil.c:1296:9
    frame #10: 0x0000000101933f38 libpython3.13td.dylib`_PyEval_EvalFrameDefault(tstate=0x0000000101d2fe40, frame=0x0000000100cc0be8, throwflag=0) at generated_cases.c.h:846:13
    frame #11: 0x000000010192fbbc libpython3.13td.dylib`_PyEval_EvalFrame(tstate=0x0000000101d2fe40, frame=0x0000000100cc0878, throwflag=0) at pycore_ceval.h:119:16
    frame #12: 0x000000010192fb28 libpython3.13td.dylib`_PyEval_Vector(tstate=0x0000000101d2fe40, func=0x0000020000404a50, locals='0x16fff91a0', args='0x16fff9198', argcount=2, kwnames='0x16fff9188') at ceval.c:1806:12
    frame #13: 0x000000010171a8a8 libpython3.13td.dylib`_PyFunction_Vectorcall(func='0x16fff9210', stack='0x16fff9208', nargsf=2, kwnames='0x16fff91f8') at call.c:413:16
    frame #14: 0x0000000101718f4c libpython3.13td.dylib`_PyObject_VectorcallTstate(tstate=0x0000000101d2fe40, callable='0x16fff9268', args='0x16fff9260', nargsf=2, kwnames='0x16fff9250') at pycore_call.h:168:11
    frame #15: 0x000000010171bb88 libpython3.13td.dylib`object_vacall(tstate=0x0000000101d2fe40, base='0x16fff92e8', callable='0x16fff92e0', vargs="") at call.c:819:14
    frame #16: 0x000000010171b998 libpython3.13td.dylib`PyObject_CallMethodObjArgs(obj='0x16fff9380', name=0x0000000101cf2308) at call.c:880:24
    frame #17: 0x00000001019c64a8 libpython3.13td.dylib`import_find_and_load(tstate=0x0000000101d2fe40, abs_name=0x0000020000a8b770) at import.c:3675:11
    frame #18: 0x00000001019c578c libpython3.13td.dylib`PyImport_ImportModuleLevelObject(name=0x0000020000a8b770, globals=0x0000020000a922f0, locals=0x0000020000a922f0, fromlist=0x00000200000addf0, level=0) at import.c:3757:15
    frame #19: 0x0000000101950ad8 libpython3.13td.dylib`import_name(tstate=0x0000000101d2fe40, frame=0x0000000100cc0808, name=0x0000020000a8b770, fromlist=0x00000200000addf0, level=0) at ceval.c:2693:16
    frame #20: 0x000000010193f020 libpython3.13td.dylib`_PyEval_EvalFrameDefault(tstate=0x0000000101d2fe40, frame=0x0000000100cc0808, throwflag=0) at generated_cases.c.h:3201:19
    frame #21: 0x000000010192fbbc libpython3.13td.dylib`_PyEval_EvalFrame(tstate=0x0000000101d2fe40, frame=0x0000000100cc0808, throwflag=0) at pycore_ceval.h:119:16
    frame #22: 0x000000010192fb28 libpython3.13td.dylib`_PyEval_Vector(tstate=0x0000000101d2fe40, func=0x000002000079fcd0, locals=0x0000020000a922f0, args='0x16fffc048', argcount=0, kwnames='0x16fffc038') at ceval.c:1806:12
    frame #23: 0x000000010192f99c libpython3.13td.dylib`PyEval_EvalCode(co='0x16fffc0f0', globals=0x0000020000a922f0, locals=0x0000020000a922f0) at ceval.c:596:21
    frame #24: 0x000000010192a7bc libpython3.13td.dylib`builtin_exec_impl(module='0x16fffc1d0', source='0x16fffc1c8', globals=0x0000020000a922f0, locals=0x0000020000a922f0, closure='0x16fffc1b0') at bltinmodule.c:1145:17
    frame #25: 0x0000000101927920 libpython3.13td.dylib`builtin_exec(module='0x16fffc260', args='0x16fffc258', nargs=2, kwnames='0x16fffc248') at bltinmodule.c.h:556:20
    frame #26: 0x00000001017b3cd8 libpython3.13td.dylib`cfunction_vectorcall_FASTCALL_KEYWORDS(func='0x16fffc2e0', args='0x16fffc2d8', nargsf=2, kwnames='0x16fffc2c8') at methodobject.c:441:24
    frame #27: 0x000000010171a284 libpython3.13td.dylib`_PyVectorcall_Call(tstate=0x0000000101d2fe40, func=(libpython3.13td.dylib`cfunction_vectorcall_FASTCALL_KEYWORDS at methodobject.c:433), callable='0x16fffc350', tuple=0x0000020000a96790, kwargs={}) at call.c:273:16
    frame #28: 0x000000010171a52c libpython3.13td.dylib`_PyObject_Call(tstate=0x0000000101d2fe40, callable='0x16fffc3b8', args=0x0000020000a96790, kwargs={}) at call.c:348:16
    frame #29: 0x000000010171a614 libpython3.13td.dylib`PyObject_Call(callable='0x16fffc3f8', args=0x0000020000a96790, kwargs={}) at call.c:373:12
    frame #30: 0x0000000101936954 libpython3.13td.dylib`_PyEval_EvalFrameDefault(tstate=0x0000000101d2fe40, frame=0x0000000100cc0780, throwflag=0) at generated_cases.c.h:1355:26
    frame #31: 0x000000010192fbbc libpython3.13td.dylib`_PyEval_EvalFrame(tstate=0x0000000101d2fe40, frame=0x0000000100cc0508, throwflag=0) at pycore_ceval.h:119:16
    frame #32: 0x000000010192fb28 libpython3.13td.dylib`_PyEval_Vector(tstate=0x0000000101d2fe40, func=0x0000020000404a50, locals='0x16fffee20', args='0x16fffee18', argcount=2, kwnames='0x16fffee08') at ceval.c:1806:12
    frame #33: 0x000000010171a8a8 libpython3.13td.dylib`_PyFunction_Vectorcall(func='0x16fffee90', stack='0x16fffee88', nargsf=2, kwnames='0x16fffee78') at call.c:413:16
    frame #34: 0x0000000101718f4c libpython3.13td.dylib`_PyObject_VectorcallTstate(tstate=0x0000000101d2fe40, callable='0x16fffeee8', args='0x16fffeee0', nargsf=2, kwnames='0x16fffeed0') at pycore_call.h:168:11
    frame #35: 0x000000010171bb88 libpython3.13td.dylib`object_vacall(tstate=0x0000000101d2fe40, base='0x16fffef68', callable='0x16fffef60', vargs="") at call.c:819:14
    frame #36: 0x000000010171b998 libpython3.13td.dylib`PyObject_CallMethodObjArgs(obj='0x16ffff000', name=0x0000000101cf2308) at call.c:880:24
    frame #37: 0x00000001019c64a8 libpython3.13td.dylib`import_find_and_load(tstate=0x0000000101d2fe40, abs_name=0x0000020000a76b40) at import.c:3675:11
    frame #38: 0x00000001019c578c libpython3.13td.dylib`PyImport_ImportModuleLevelObject(name=0x0000020000a76b40, globals='0x16ffff200', locals='0x16ffff1f8', fromlist='0x16ffff1f0', level=0) at import.c:3757:15
    frame #39: 0x0000000101929850 libpython3.13td.dylib`builtin___import___impl(module='0x16ffff248', name=0x0000020000a76b40, globals='0x16ffff238', locals='0x16ffff230', fromlist='0x16ffff228', level=0) at bltinmodule.c:277:12
    frame #40: 0x0000000101926898 libpython3.13td.dylib`builtin___import__(module='0x16ffff2d8', args='0x16ffff2d0', nargs=1, kwnames='0x16ffff2c0') at bltinmodule.c.h:107:20
    frame #41: 0x00000001017b3cd8 libpython3.13td.dylib`cfunction_vectorcall_FASTCALL_KEYWORDS(func='0x16ffff360', args='0x16ffff358', nargsf=1, kwnames='0x16ffff348') at methodobject.c:441:24
    frame #42: 0x000000010171a284 libpython3.13td.dylib`_PyVectorcall_Call(tstate=0x0000000101d2fe40, func=(libpython3.13td.dylib`cfunction_vectorcall_FASTCALL_KEYWORDS at methodobject.c:433), callable='0x16ffff3d0', tuple=0x00000200000ad760, kwargs={}) at call.c:273:16
    frame #43: 0x000000010171a52c libpython3.13td.dylib`_PyObject_Call(tstate=0x0000000101d2fe40, callable='0x16ffff438', args=0x00000200000ad760, kwargs={}) at call.c:348:16
    frame #44: 0x000000010171a614 libpython3.13td.dylib`PyObject_Call(callable='0x16ffff478', args=0x00000200000ad760, kwargs={}) at call.c:373:12
    frame #45: 0x0000000101936954 libpython3.13td.dylib`_PyEval_EvalFrameDefault(tstate=0x0000000101d2fe40, frame=0x0000000100cc0480, throwflag=0) at generated_cases.c.h:1355:26
    frame #46: 0x000000010192fbbc libpython3.13td.dylib`_PyEval_EvalFrame(tstate=0x0000000101d2fe40, frame=0x0000000100cc0318, throwflag=0) at pycore_ceval.h:119:16
    frame #47: 0x000000010192fb28 libpython3.13td.dylib`_PyEval_Vector(tstate=0x0000000101d2fe40, func=0x0000020000404a50, locals='0x170001ea0', args='0x170001e98', argcount=2, kwnames='0x170001e88') at ceval.c:1806:12
    frame #48: 0x000000010171a8a8 libpython3.13td.dylib`_PyFunction_Vectorcall(func='0x170001f10', stack='0x170001f08', nargsf=2, kwnames='0x170001ef8') at call.c:413:16
    frame #49: 0x0000000101718f4c libpython3.13td.dylib`_PyObject_VectorcallTstate(tstate=0x0000000101d2fe40, callable='0x170001f68', args='0x170001f60', nargsf=2, kwnames='0x170001f50') at pycore_call.h:168:11
    frame #50: 0x000000010171bb88 libpython3.13td.dylib`object_vacall(tstate=0x0000000101d2fe40, base='0x170001fe8', callable='0x170001fe0', vargs="") at call.c:819:14
    frame #51: 0x000000010171b998 libpython3.13td.dylib`PyObject_CallMethodObjArgs(obj='0x170002080', name=0x0000000101cf2308) at call.c:880:24
    frame #52: 0x00000001019c64a8 libpython3.13td.dylib`import_find_and_load(tstate=0x0000000101d2fe40, abs_name=0x0000020000a76de0) at import.c:3675:11
    frame #53: 0x00000001019c578c libpython3.13td.dylib`PyImport_ImportModuleLevelObject(name=0x0000020000a76de0, globals=0x0000020000370c70, locals=0x0000020000370c70, fromlist=[], level=0) at import.c:3757:15
    frame #54: 0x0000000101929850 libpython3.13td.dylib`builtin___import___impl(module='0x1700022c8', name=0x0000020000a76de0, globals=0x0000020000370c70, locals=0x0000020000370c70, fromlist=[], level=0) at bltinmodule.c:277:12
    frame #55: 0x0000000101926898 libpython3.13td.dylib`builtin___import__(module='0x170002358', args='0x170002350', nargs=5, kwnames='0x170002340') at bltinmodule.c.h:107:20
    frame #56: 0x00000001017b3cd8 libpython3.13td.dylib`cfunction_vectorcall_FASTCALL_KEYWORDS(func='0x1700023e0', args='0x1700023d8', nargsf=5, kwnames='0x1700023c8') at methodobject.c:441:24
    frame #57: 0x0000000101718f4c libpython3.13td.dylib`_PyObject_VectorcallTstate(tstate=0x0000000101d2fe40, callable='0x170002438', args='0x170002430', nargsf=5, kwnames='0x170002420') at pycore_call.h:168:11
    frame #58: 0x000000010171afd4 libpython3.13td.dylib`_PyObject_CallFunctionVa(tstate=0x0000000101d2fe40, callable='0x1700024c8', format="OOOOi", va="\xe0m\xa7") at call.c:552:18
    frame #59: 0x000000010171ae00 libpython3.13td.dylib`PyObject_CallFunction(callable='0x170002548', format="OOOOi") at call.c:574:14
    frame #60: 0x00000001019c5418 libpython3.13td.dylib`PyImport_Import(module_name=0x0000020000a76de0) at import.c:3942:9
    frame #61: 0x00000001019c72e8 libpython3.13td.dylib`_PyImport_GetModuleAttr(modname=0x0000020000a76de0, attrname=0x0000020000a76e50) at import.c:4173:21
    frame #62: 0x00000001019c73cc libpython3.13td.dylib`_PyImport_GetModuleAttrString(modname="collections.abc", attrname="MutableSequence") at import.c:4194:24
    frame #63: 0x000000010717f76c array.cpython-313td-darwin.so`array_modexec(m='0x1700026e8') at arraymodule.c:3189:33
    frame #64: 0x00000001017b71c8 libpython3.13td.dylib`PyModule_ExecDef(module='0x170002770', def=0x000000010718c300) at moduleobject.c:489:23
    frame #65: 0x00000001019c9e70 libpython3.13td.dylib`exec_builtin_or_dynamic(mod='0x1700027a0') at import.c:808:12
    frame #66: 0x00000001019cd5f8 libpython3.13td.dylib`_imp_exec_dynamic_impl(module='0x1700027c8', mod='0x1700027c0') at import.c:4739:12
    frame #67: 0x00000001019cbe8c libpython3.13td.dylib`_imp_exec_dynamic(module='0x1700027f8', mod='0x1700027f0') at import.c.h:513:21
    frame #68: 0x00000001017b3f74 libpython3.13td.dylib`cfunction_vectorcall_O(func='0x170002870', args='0x170002868', nargsf=1, kwnames='0x170002858') at methodobject.c:512:24
    frame #69: 0x000000010171a284 libpython3.13td.dylib`_PyVectorcall_Call(tstate=0x0000000101d2fe40, func=(libpython3.13td.dylib`cfunction_vectorcall_O at methodobject.c:493), callable='0x1700028e0', tuple=('0x200000a9868',), kwargs={}) at call.c:273:16
    frame #70: 0x000000010171a52c libpython3.13td.dylib`_PyObject_Call(tstate=0x0000000101d2fe40, callable='0x170002948', args=('0x200000a9868',), kwargs={}) at call.c:348:16
    frame #71: 0x000000010171a614 libpython3.13td.dylib`PyObject_Call(callable='0x170002988', args=('0x200000a9868',), kwargs={}) at call.c:373:12
    frame #72: 0x0000000101936954 libpython3.13td.dylib`_PyEval_EvalFrameDefault(tstate=0x0000000101d2fe40, frame=0x0000000100cc0290, throwflag=0) at generated_cases.c.h:1355:26
    frame #73: 0x000000010192fbbc libpython3.13td.dylib`_PyEval_EvalFrame(tstate=0x0000000101d2fe40, frame=0x0000000100cc0020, throwflag=0) at pycore_ceval.h:119:16
    frame #74: 0x000000010192fb28 libpython3.13td.dylib`_PyEval_Vector(tstate=0x0000000101d2fe40, func=0x0000020000404a50, locals='0x1700053b0', args='0x1700053a8', argcount=2, kwnames='0x170005398') at ceval.c:1806:12
    frame #75: 0x000000010171a8a8 libpython3.13td.dylib`_PyFunction_Vectorcall(func='0x170005420', stack='0x170005418', nargsf=2, kwnames='0x170005408') at call.c:413:16
    frame #76: 0x0000000101718f4c libpython3.13td.dylib`_PyObject_VectorcallTstate(tstate=0x0000000101d2fe40, callable='0x170005478', args='0x170005470', nargsf=2, kwnames='0x170005460') at pycore_call.h:168:11
    frame #77: 0x000000010171bb88 libpython3.13td.dylib`object_vacall(tstate=0x0000000101d2fe40, base='0x1700054f8', callable='0x1700054f0', vargs="") at call.c:819:14
    frame #78: 0x000000010171b998 libpython3.13td.dylib`PyObject_CallMethodObjArgs(obj='0x170005590', name=0x0000000101cf2308) at call.c:880:24
    frame #79: 0x00000001019c64a8 libpython3.13td.dylib`import_find_and_load(tstate=0x0000000101d2fe40, abs_name=0x0000020000a85530) at import.c:3675:11
    frame #80: 0x00000001019c578c libpython3.13td.dylib`PyImport_ImportModuleLevelObject(name=0x0000020000a85530, globals=0x0000020000696df0, locals=0x0000020000696df0, fromlist=[], level=0) at import.c:3757:15
    frame #81: 0x0000000101929850 libpython3.13td.dylib`builtin___import___impl(module='0x1700057d8', name=0x0000020000a85530, globals=0x0000020000696df0, locals=0x0000020000696df0, fromlist=[], level=0) at bltinmodule.c:277:12
    frame #82: 0x0000000101926898 libpython3.13td.dylib`builtin___import__(module='0x170005868', args='0x170005860', nargs=5, kwnames='0x170005850') at bltinmodule.c.h:107:20
    frame #83: 0x00000001017b3cd8 libpython3.13td.dylib`cfunction_vectorcall_FASTCALL_KEYWORDS(func='0x1700058f0', args='0x1700058e8', nargsf=5, kwnames='0x1700058d8') at methodobject.c:441:24
    frame #84: 0x0000000101718f4c libpython3.13td.dylib`_PyObject_VectorcallTstate(tstate=0x0000000101d2fe40, callable='0x170005948', args='0x170005940', nargsf=5, kwnames='0x170005930') at pycore_call.h:168:11
    frame #85: 0x000000010171afd4 libpython3.13td.dylib`_PyObject_CallFunctionVa(tstate=0x0000000101d2fe40, callable='0x1700059d8', format="OOOOi", va="0U\xa8") at call.c:552:18
    frame #86: 0x000000010171ae00 libpython3.13td.dylib`PyObject_CallFunction(callable='0x170005a58', format="OOOOi") at call.c:574:14
    frame #87: 0x00000001019c5418 libpython3.13td.dylib`PyImport_Import(module_name=0x0000020000a85530) at import.c:3942:9
    frame #88: 0x00000001001d1720 pyo3-97f8571082c6d82b`pyo3::types::module::PyModule::import::h10020d9fbc325799((null)=(__0 = core::marker::PhantomData<(&pyo3::gil::GILGuard, pyo3::impl_::not_send::NotSend)> @ 0x0000000170005b4f), name=(data_ptr = "arrayf", length = 5)) at module.rs:91:13
    frame #89: 0x0000000100082e9c pyo3-97f8571082c6d82b`pyo3::marker::Python::import::h76b7d162714cdbe5((null)=(__0 = core::marker::PhantomData<(&pyo3::gil::GILGuard, pyo3::impl_::not_send::NotSend)> @ 0x0000000170005b8f), name=(data_ptr = "arrayf", length = 5)) at marker.rs:712:9
    frame #90: 0x000000010018110c pyo3-97f8571082c6d82b`pyo3::buffer::tests::test_array_buffer::_$u7b$$u7b$closure$u7d$$u7d$::h5080eb33d98ca1ff((null)={closure_env#0} @ 0x00000001700063be, (null)=(__0 = core::marker::PhantomData<(&pyo3::gil::GILGuard, pyo3::impl_::not_send::NotSend)> @ 0x00000001700063bf)) at buffer.rs:889:25
    frame #91: 0x000000010006b074 pyo3-97f8571082c6d82b`pyo3::marker::Python::with_gil::h5467a51b0b9ab8e3(f={closure_env#0} @ 0x000000017000679f) at marker.rs:409:9
    frame #92: 0x00000001000aacac pyo3-97f8571082c6d82b`pyo3::buffer::tests::test_array_buffer::h09873f4901cd5a78 at buffer.rs:888:9
    frame #93: 0x00000001001810c4 pyo3-97f8571082c6d82b`pyo3::buffer::tests::test_array_buffer::_$u7b$$u7b$closure$u7d$$u7d$::hcd0c85096f6a20d2((null)=0x00000001700067fe) at buffer.rs:887:27
    frame #94: 0x00000001001586bc pyo3-97f8571082c6d82b`core::ops::function::FnOnce::call_once::h518ac7d9c7bd346c((null)={closure_env#0} @ 0x00000001700067fe, (null)=<unavailable>) at function.rs:250:5
    frame #95: 0x000000010029af48 pyo3-97f8571082c6d82b`test::__rust_begin_short_backtrace::hc730144174a2f2b8 [inlined] core::ops::function::FnOnce::call_once::h0ba6e3d0adcb0fb8 at function.rs:250:5 [opt]
    frame #96: 0x000000010029af40 pyo3-97f8571082c6d82b`test::__rust_begin_short_backtrace::hc730144174a2f2b8 at lib.rs:624:18 [opt]
    frame #97: 0x000000010029a89c pyo3-97f8571082c6d82b`test::run_test::_$u7b$$u7b$closure$u7d$$u7d$::h3dade545f948edf5 [inlined] test::run_test_in_process::_$u7b$$u7b$closure$u7d$$u7d$::h9e544e0587f41be2 at lib.rs:647:60 [opt]
    frame #98: 0x000000010029a890 pyo3-97f8571082c6d82b`test::run_test::_$u7b$$u7b$closure$u7d$$u7d$::h3dade545f948edf5 [inlined] _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h8389036b3da9abb4 at unwind_safe.rs:272:9 [opt]
    frame #99: 0x000000010029a890 pyo3-97f8571082c6d82b`test::run_test::_$u7b$$u7b$closure$u7d$$u7d$::h3dade545f948edf5 [inlined] std::panicking::try::do_call::h6c15d214f8b0efc9 at panicking.rs:557:40 [opt]
    frame #100: 0x000000010029a88c pyo3-97f8571082c6d82b`test::run_test::_$u7b$$u7b$closure$u7d$$u7d$::h3dade545f948edf5 [inlined] std::panicking::try::h4dec6d151c5d7c52 at panicking.rs:521:19 [opt]
    frame #101: 0x000000010029a88c pyo3-97f8571082c6d82b`test::run_test::_$u7b$$u7b$closure$u7d$$u7d$::h3dade545f948edf5 [inlined] std::panic::catch_unwind::h08246350d14b78e5 at panic.rs:350:14 [opt]
    frame #102: 0x000000010029a88c pyo3-97f8571082c6d82b`test::run_test::_$u7b$$u7b$closure$u7d$$u7d$::h3dade545f948edf5 [inlined] test::run_test_in_process::h5176f0d7330017af at lib.rs:647:27 [opt]
    frame #103: 0x000000010029a808 pyo3-97f8571082c6d82b`test::run_test::_$u7b$$u7b$closure$u7d$$u7d$::h3dade545f948edf5 at lib.rs:568:43 [opt]
    frame #104: 0x000000010026c9c8 pyo3-97f8571082c6d82b`std::sys::backtrace::__rust_begin_short_backtrace::h927c17d76a6dfbc6 [inlined] test::run_test::_$u7b$$u7b$closure$u7d$$u7d$::h331e167c3eb94f21 at lib.rs:598:41 [opt]
    frame #105: 0x000000010026c940 pyo3-97f8571082c6d82b`std::sys::backtrace::__rust_begin_short_backtrace::h927c17d76a6dfbc6 at backtrace.rs:152:18 [opt]
    frame #106: 0x000000010026fb2c pyo3-97f8571082c6d82b`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h4b1e8564c6e52dc1 [inlined] std::thread::Builder::spawn_unchecked_::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h9a08c487421e7042 at mod.rs:538:17 [opt]
    frame #107: 0x000000010026fb24 pyo3-97f8571082c6d82b`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h4b1e8564c6e52dc1 [inlined] _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::hf6a2344f0bd24956 at unwind_safe.rs:272:9 [opt]
    frame #108: 0x000000010026fb24 pyo3-97f8571082c6d82b`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h4b1e8564c6e52dc1 [inlined] std::panicking::try::do_call::hfdb1a93845faf3ef at panicking.rs:557:40 [opt]
    frame #109: 0x000000010026fb24 pyo3-97f8571082c6d82b`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h4b1e8564c6e52dc1 [inlined] std::panicking::try::h4b35ce5ad8a162fd at panicking.rs:521:19 [opt]
    frame #110: 0x000000010026fb24 pyo3-97f8571082c6d82b`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h4b1e8564c6e52dc1 [inlined] std::panic::catch_unwind::h3bb4d3ee2986e761 at panic.rs:350:14 [opt]
    frame #111: 0x000000010026fb24 pyo3-97f8571082c6d82b`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h4b1e8564c6e52dc1 [inlined] std::thread::Builder::spawn_unchecked_::_$u7b$$u7b$closure$u7d$$u7d$::h5e36dd32c0d26255 at mod.rs:537:30 [opt]
    frame #112: 0x000000010026fab0 pyo3-97f8571082c6d82b`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h4b1e8564c6e52dc1 at function.rs:250:5 [opt]
    frame #113: 0x0000000100310d10 pyo3-97f8571082c6d82b`std::sys::pal::unix::thread::Thread::new::thread_start::h1bd1b9c95010bf71 [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h01276ebbe54a8110 at boxed.rs:2070:9 [opt]
    frame #114: 0x0000000100310d04 pyo3-97f8571082c6d82b`std::sys::pal::unix::thread::Thread::new::thread_start::h1bd1b9c95010bf71 [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::habad1faa89d23086 at boxed.rs:2070:9 [opt]
    frame #115: 0x0000000100310d00 pyo3-97f8571082c6d82b`std::sys::pal::unix::thread::Thread::new::thread_start::h1bd1b9c95010bf71 at thread.rs:108:17 [opt]
    frame #116: 0x0000000193ba1f94 libsystem_pthread.dylib`_pthread_start + 136

Not sure if it will be helpful, but the tracebacks from all of the threads are here: https://gist.github.com/ngoldbaum/d3f5bceba9554ba8347c40773446b08d

Happy to help with reproducing this if anyone has trouble getting PyO3 setup. I'm on the CPython discord in the #free-threading channel if chatting is easier.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux, macOS

Output from running 'python -VV' on the command line:

Python 3.13.0rc2 experimental free-threading build (main, Sep 23 2024, 13:34:50) [Clang 15.0.0 (clang-1500.3.9.4)]

Linked PRs
@ngoldbaum ngoldbaum added the type-crash A hard crash of the interpreter, possibly with a core dump label Sep 23, 2024
@ZeroIntensity ZeroIntensity added interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-free-threading labels Sep 23, 2024
@ngoldbaum ngoldbaum mentioned this issue Sep 23, 2024
Merged
@colesbury
Copy link
Contributor

Ok, I think I understand the problem. We loop over all the thread states in the free-threaded GC, calling _PyMem_ProcessDelayed():

cpython/Python/gc_free_threading.c

Lines 427 to 433 in 41e7992

HEAD_LOCK(&_PyRuntime);
PyThreadState *tstate = interp->threads.head;
while (tstate != NULL) {
_PyMem_ProcessDelayed(tstate);
tstate = (PyThreadState *)tstate->next;
}
HEAD_UNLOCK(&_PyRuntime);

The problem is that the thread states might not yet be initialized, so tstate->qsbr might be NULL. We should guard against that case either in _PyMem_ProcessDelayed() or in the GC.

colesbury added a commit to colesbury/cpython that referenced this issue Sep 24, 2024
@colesbury
pythongh-124375: Avoid calling _PyMem_ProcessDelayed on other threa…
bc567fc 
…d states

This fixes a crash when running the PyO3 test suite on the free-threaded
build. The `qsbr` field is initialized after the `PyThreadState` is
added to the interpreter's linked list -- it might still be NULL.

Instead, we "steal" the queue of to-be-freed memory blocks. This is
always initialized (possibly empty) and protected by the stop the world
pause.
@bedevere-app bedevere-app bot mentioned this issue Sep 24, 2024
Merged
colesbury added a commit to colesbury/cpython that referenced this issue Oct 15, 2024
@colesbury
Merge branch 'main' into pythongh-124375-py-mem-process-delayed
bbaef28
colesbury added a commit that referenced this issue Oct 15, 2024
@colesbury
gh-124375: Avoid calling _PyMem_ProcessDelayed on other thread stat…
54c6fcb 
…es (#124459)

This fixes a crash when running the PyO3 test suite on the free-threaded
build. The `qsbr` field is initialized after the `PyThreadState` is
added to the interpreter's linked list -- it might still be NULL.

Instead, we "steal" the queue of to-be-freed memory blocks. This is
always initialized (possibly empty) and protected by the stop the world
pause.
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Oct 15, 2024
@colesbury @miss-islington
pythongh-124375: Avoid calling _PyMem_ProcessDelayed on other threa…
93e51c9 
…d states (pythonGH-124459)

This fixes a crash when running the PyO3 test suite on the free-threaded
build. The `qsbr` field is initialized after the `PyThreadState` is
added to the interpreter's linked list -- it might still be NULL.

Instead, we "steal" the queue of to-be-freed memory blocks. This is
always initialized (possibly empty) and protected by the stop the world
pause.
(cherry picked from commit 54c6fcb)

Co-authored-by: Sam Gross <[email protected]>
@bedevere-app bedevere-app bot mentioned this issue Oct 15, 2024
Merged
colesbury added a commit that referenced this issue Oct 15, 2024
@miss-islington @colesbury
[3.13] gh-124375: Avoid calling _PyMem_ProcessDelayed on other thre…
4d83f6d 
…ad states (GH-124459) (#125540)

This fixes a crash when running the PyO3 test suite on the free-threaded
build. The `qsbr` field is initialized after the `PyThreadState` is
added to the interpreter's linked list -- it might still be NULL.

Instead, we "steal" the queue of to-be-freed memory blocks. This is
always initialized (possibly empty) and protected by the stop the world
pause.
(cherry picked from commit 54c6fcb)

Co-authored-by: Sam Gross <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-free-threading type-crash A hard crash of the interpreter, possibly with a core dump
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@colesbury @ngoldbaum @ZeroIntensity