Debug mode that can be easily activated without recompilation

[steve-s]

As Victor Stinner already noted elsewhere people do not use debug builds and miss out on its features. One way to make people more likely to use debugging features (apart from more marketing) is if they did not require recompilation of (C)Python and ideally even recompilation of the extension, so that one could easily test the whole thing including 3rd party deps in debug mode.

[gvanrossum]

What does "debug mode" mean to you? To me, it primarily means "turn on C asserts" which is not something that can be turned off without recompilation (or without a perf cost).

[steve-s]

Yes it means basically assertions.

On top of traditional assertions I think the API should ideally have mechanisms that enforce parts of its contract that otherwise can get some leeway. For example, instead of handing out real pointers to internal data-structures, it would tag/untag them at the API boundary, which can be performance sensitive, so maybe not something to do in some production/release mode, but it would ensure that the user code cannot take advantage (*) of this implementation detail that is not part of the API contract.

which is not something that can be turned off without recompilation (or without a perf cost).

Yes, not without a perf cost. I think that sometimes that could be justified (for extensions that are not performance sensitive or where hot code paths do not use Python API). Additionally, if there is some indirection already, like in HPy universal mode, where calls are routed via a pointer table, the additional cost is zero (we just use different pointer table, AFAIK modern CPUs should be good at predictable indirect jumps). Note that the HPy architecture also allows to compile in "CPython ABI" mode where this indirection is resolved at compile time and hence the turning on/off the debug mode at runtime does not work in this mode. I imagine that very performance sensitive extensions would chose to, for example, publish and maintain "CPython ABI" builds for the latest CPython version + HPy universal binary for all the other versions and other Python implementations.

(*) My thinking: dereferencing a pointer and reading some data from it may seem ok-ish to some people that are not used to pay too much attention to "program against specification and not implementation", but hopefully untagging a pointer (if running in debug mode and not otherwise) should look fishy to anyone.

[vstinner]

As Victor Stinner already noted elsewhere people do not use debug builds and miss out on its features

I designed the Python Development Mode around the fact that many users don't have access to a debug build of Python.

I would love that python -X dev script.py would work as if I had run python-debug -X dev script.py whereas python-debug is a debug build of Python. Problem: many Linux distributions don't ship python-debug to make packages smaller and because the majority of users don't need them (which is a good thing).

In 2022, I already proposed to remove assertions on Python release build, but the counter-argument is that (again) users don't have access to debug build. Thread: https://mail.python.org/archives/list/[email protected]/thread/RR3DJ4IZVSFKZUE5AV6OUCJ3QPYXD26Z/

It's inefficient that all users have to pay the cost of assertions at runtime when running correct C extensions.

[davidhewitt]

One possible option could be to enable users of GitHub Actions to use debug builds of Python as I would imagine that would enable a large percentage of open source projects to use debug builds for their test suite.

I haven't checked how much work it would be to get debug interpreters available there, from a cursory glance there isn't a mention of python-debug in the docs for actions/setup-python.

I'd be willing to run a portion of PyO3's test suite against the debug interpreter, although I think I'd also want to run against the release interpreter (to verify compatibility, rather than speed, unless the debug interpreter is significantly slower).

[gvanrossum]

@davidhewitt

One possible option could be to enable users of GitHub Actions to use debug builds of Python as I would imagine that would enable a large percentage of open source projects to use debug builds for their test suite.

This seems a simple matter of elbow grease. Maybe just open issues in the various projects asking for this to be enabled, linking back here for reference?

[vstinner]

Maybe just open issues in the various projects asking for this to be enabled

Like time that I checked, GHA didn't provide a debug build of Python. Did the situation change?

[mattip]

There is an open issue in the setup-python github action to add debug builds: actions/setup-python#86. I commented there asking about the status of the issue.

[mattip]

FWIW, NumPy does test a debug build using whatever apt install python3-dbg provides.