Compatibility with Python3.11 (again) #288

tacaswell · 2022-01-25T17:44:19Z

python/cpython#30590 made changes to the way exceptions are stored on the thread stucts and greenlet does not compile

$ gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/tcaswell/.virtualenvs/bleeding/include -I/home/tcaswell/.pybuild/bleedin
g/include/python3.11 -c src/greenlet/greenlet.cpp -o build/temp.linux-x86_64-3.11/src/greenlet/greenlet.o
In file included from src/greenlet/greenlet_internal.hpp:19,
                 from src/greenlet/greenlet.cpp:17:
src/greenlet/greenlet_greenlet.hpp: In member function ‘void greenlet::ExceptionState::operator<<(const PyThreadState*)’:
src/greenlet/greenlet_greenlet.hpp:640:31: error: ‘const PyThreadState’ {aka ‘const struct _ts’} has no member named ‘exc_state’; did you mean ‘_exc_state’?
  640 |     this->exc_state = tstate->exc_state;
      |                               ^~~~~~~~~
      |                               _exc_state
src/greenlet/greenlet_greenlet.hpp: In member function ‘void greenlet::ExceptionState::operator>>(PyThreadState*)’:
src/greenlet/greenlet_greenlet.hpp:645:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_state’; did you mean ‘_exc_state’?
  645 |     tstate->exc_state = this->exc_state;
      |             ^~~~~~~~~
      |             _exc_state
src/greenlet/greenlet_greenlet.hpp:647:52: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_state’; did you mean ‘_exc_state’?
  647 |         this->exc_info ? this->exc_info : &tstate->exc_state;
      |                                                    ^~~~~~~~~
      |                                                    _exc_state
src/greenlet/greenlet_greenlet.hpp: In constructor ‘greenlet::PythonState::PythonState()’:
src/greenlet/greenlet_greenlet.hpp:797:42: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘root_cframe’; did you mean ‘_root_cframe’?
  797 |     this->cframe = &PyThreadState_GET()->root_cframe;
      |                                          ^~~~~~~~~~~
      |                                          _root_cframe
src/greenlet/greenlet_greenlet.hpp: In member function ‘void greenlet::PythonState::set_new_cframe(CFrame&)’:
src/greenlet/greenlet_greenlet.hpp:919:52: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘root_cframe’; did you mean ‘_root_cframe’?
  919 |     this->cframe->previous = &PyThreadState_GET()->root_cframe;
      |                                                    ^~~~~~~~~~~
      |                                                    _root_cframe

The text was updated successfully, but these errors were encountered:

blackPantherOS · 2022-01-31T11:38:25Z

Python-3.11
Gcc-11.2.1
Cython: 0.29.26

gcc -pthread -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -g -fwrapv -O3 -Wall -O2 -g -pipe -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector --param=ssp-buffer-size=4 -O2 -g -pipe -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector --param=ssp-buffer-size=4 -fPIC -I/usr/include/python3.11 -c src/greenlet/greenlet.cpp -o build/temp.linux-x86_64-3.11/src/greenlet/greenlet.o
In file included from src/greenlet/greenlet_internal.hpp:19,
                 from src/greenlet/greenlet.cpp:17:
src/greenlet/greenlet_greenlet.hpp: In member function ‘void greenlet::PythonState::operator<<(const PyThreadState*)’:
src/greenlet/greenlet_greenlet.hpp:830:37: error: ‘const PyThreadState’ {aka ‘const struct _ts’} has no member named ‘recursion_depth’; did you mean ‘recursion_limit’?
  830 |     this->recursion_depth = tstate->recursion_depth;
      |                                     ^~~~~~~~~~~~~~~
      |                                     recursion_limit
src/greenlet/greenlet_greenlet.hpp:831:36: error: ‘const PyThreadState’ {aka ‘const struct _ts’} has no member named ‘frame’; did you mean ‘cframe’?
  831 |     this->_top_frame.steal(tstate->frame);
      |                                    ^~~~~
      |                                    cframe
src/greenlet/greenlet_greenlet.hpp: In member function ‘void greenlet::PythonState::operator>>(PyThreadState*)’:
src/greenlet/greenlet_greenlet.hpp:861:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘frame’; did you mean ‘cframe’?
  861 |     tstate->frame = this->_top_frame.relinquish_ownership();
      |             ^~~~~
      |             cframe
src/greenlet/greenlet_greenlet.hpp:862:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘recursion_depth’; did you mean ‘recursion_limit’?
  862 |     tstate->recursion_depth = this->recursion_depth;
      |             ^~~~~~~~~~~~~~~
      |             recursion_limit
src/greenlet/greenlet_greenlet.hpp: In member function ‘void greenlet::PythonState::set_initial_state(const PyThreadState*)’:
src/greenlet/greenlet_greenlet.hpp:883:37: error: ‘const PyThreadState’ {aka ‘const struct _ts’} has no member named ‘recursion_depth’; did you mean ‘recursion_limit’?
  883 |     this->recursion_depth = tstate->recursion_depth;
      |                                     ^~~~~~~~~~~~~~~
      |                                     recursion_limit
error: command '/usr/bin/gcc' failed with exit code 1

brandtbucher · 2022-02-01T17:31:46Z

We went ahead and reverted that change:

python/cpython#31038

ericsnowcurrently · 2022-02-01T21:23:39Z

We went ahead and reverted that change:

I only fixed root_cframe and exc_state in python/cpython#31038. recursion_depth was changed in GH-29524 and hasn't been sorted out, so it is probably still breaking greenlet.

brandtbucher · 2022-02-01T21:30:16Z

I only fixed root_cframe and exc_state in python/cpython#31038. recursion_depth was changed in GH-29524 and hasn't been sorted out, so it is probably still breaking greenlet.

That was part of my original fix in #280. I don't believe @blackPantherOS's failure is happening on Greenlet's master branch anymore.

tacaswell · 2022-02-01T21:35:49Z

At least locally for me it compiles and python -m unittest discover -v greenlet.tests reports success.

blackPantherOS · 2022-02-02T12:15:33Z

I only fixed root_cframe and exc_state in python/cpython#31038. recursion_depth was changed in GH-29524 and hasn't been sorted out, so it is probably still breaking greenlet.

That was part of my original fix in #280. I don't believe @blackPantherOS's failure is happening on Greenlet's master branch anymore.

Thanks, I tried latest git (2.0.0a1) as well

it clone https://github.com/python-greenlet/greenlet.git
Cloning into 'greenlet'...
remote: Enumerating objects: 3914, done.
remote: Counting objects: 100% (1804/1804), done.
remote: Compressing objects: 100% (446/446), done.
remote: Total 3914 (delta 1181), reused 1671 (delta 1085), pack-reused 2110
Receiving objects: 100% (3914/3914), 1.35 MiB | 4.15 MiB/s, done.
Resolving deltas: 100% (2530/2530), done.
└──╼ # cd greenlet/
└──╼ # python3 set
setup.cfg  setup.py   
└──╼ # python3 setup.py build
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.11
creating build/lib.linux-x86_64-3.11/greenlet
[...]

gcc -pthread -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -g -fwrapv -O3 -Wall -O2 -g -pipe -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector --param=ssp-buffer-size=4 -O2 -g -pipe -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector --param=ssp-buffer-size=4 -fPIC -I/usr/include/python3.11 -c src/greenlet/greenlet.cpp -o build/temp.linux-x86_64-3.11/src/greenlet/greenlet.o
In file included from src/greenlet/greenlet_internal.hpp:19,
                 from src/greenlet/greenlet.cpp:17:
src/greenlet/greenlet_greenlet.hpp: In member function ‘void greenlet::PythonState::operator<<(const PyThreadState*)’:
src/greenlet/greenlet_greenlet.hpp:830:37: error: ‘const PyThreadState’ {aka ‘const struct _ts’} has no member named ‘recursion_depth’; did you mean ‘recursion_limit’?
  830 |     this->recursion_depth = tstate->recursion_depth;
      |                                     ^~~~~~~~~~~~~~~
      |                                     recursion_limit
src/greenlet/greenlet_greenlet.hpp:831:36: error: ‘const PyThreadState’ {aka ‘const struct _ts’} has no member named ‘frame’; did you mean ‘cframe’?
  831 |     this->_top_frame.steal(tstate->frame);
      |                                    ^~~~~
      |                                    cframe
src/greenlet/greenlet_greenlet.hpp: In member function ‘void greenlet::PythonState::operator>>(PyThreadState*)’:
src/greenlet/greenlet_greenlet.hpp:861:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘frame’; did you mean ‘cframe’?
  861 |     tstate->frame = this->_top_frame.relinquish_ownership();
      |             ^~~~~
      |             cframe
src/greenlet/greenlet_greenlet.hpp:862:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘recursion_depth’; did you mean ‘recursion_limit’?
  862 |     tstate->recursion_depth = this->recursion_depth;
      |             ^~~~~~~~~~~~~~~
      |             recursion_limit
src/greenlet/greenlet_greenlet.hpp: In member function ‘void greenlet::PythonState::set_initial_state(const PyThreadState*)’:
src/greenlet/greenlet_greenlet.hpp:883:37: error: ‘const PyThreadState’ {aka ‘const struct _ts’} has no member named ‘recursion_depth’; did you mean ‘recursion_limit’?
  883 |     this->recursion_depth = tstate->recursion_depth;
      |                                     ^~~~~~~~~~~~~~~
      |                                     recursion_limit
error: command '/usr/bin/gcc' failed with exit code 1

tacaswell · 2022-03-01T16:37:07Z

I am now seeing a different failure

$ gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/tcaswell/.virtualenvs/bleeding/include -I/home/tcaswell/.pybuild/bleeding/include/python3.11 -c src/greenlet/greenlet.cpp -o build/temp.linux-x86_64-3.11/src/greenlet/greenlet.o
In file included from src/greenlet/greenlet_internal.hpp:19,
                 from src/greenlet/greenlet.cpp:17:
src/greenlet/greenlet_greenlet.hpp:137:9: error: ‘CFrame’ does not name a type; did you mean ‘_frame’?
  137 |         CFrame* cframe;
      |         ^~~~~~
      |         _frame
src/greenlet/greenlet_greenlet.hpp:142:9: error: ‘_interpreter_frame’ does not name a type; did you mean ‘_PyInterpreterFrame’?
  142 |         _interpreter_frame *current_frame;
      |         ^~~~~~~~~~~~~~~~~~
      |         _PyInterpreterFrame
src/greenlet/greenlet_greenlet.hpp:163:29: error: ‘CFrame’ has not been declared
  163 |         void set_new_cframe(CFrame& frame) G_NOEXCEPT;
      |                             ^~~~~~
src/greenlet/greenlet_greenlet.hpp: In constructor ‘greenlet::PythonState::PythonState()’:
src/greenlet/greenlet_greenlet.hpp:736:6: error: class ‘greenlet::PythonState’ does not have any field named ‘cframe’
  736 |     ,cframe(nullptr)
      |      ^~~~~~
src/greenlet/greenlet_greenlet.hpp:741:6: error: class ‘greenlet::PythonState’ does not have any field named ‘current_frame’
  741 |     ,current_frame(nullptr)
      |      ^~~~~~~~~~~~~
src/greenlet/greenlet_greenlet.hpp:797:11: error: ‘class greenlet::PythonState’ has no member named ‘cframe’
  797 |     this->cframe = &PyThreadState_GET()->root_cframe;
      |           ^~~~~~
src/greenlet/greenlet_greenlet.hpp: In member function ‘void greenlet::PythonState::operator<<(const PyThreadState*)’:
src/greenlet/greenlet_greenlet.hpp:817:11: error: ‘class greenlet::PythonState’ has no member named ‘cframe’
  817 |     this->cframe = tstate->cframe;
      |           ^~~~~~
src/greenlet/greenlet_greenlet.hpp:822:11: error: ‘class greenlet::PythonState’ has no member named ‘current_frame’
  822 |     this->current_frame = tstate->cframe->current_frame;
      |           ^~~~~~~~~~~~~
src/greenlet/greenlet_greenlet.hpp: In member function ‘void greenlet::PythonState::operator>>(PyThreadState*)’:
src/greenlet/greenlet_greenlet.hpp:844:28: error: ‘class greenlet::PythonState’ has no member named ‘cframe’
  844 |     tstate->cframe = this->cframe;
      |                            ^~~~~~
src/greenlet/greenlet_greenlet.hpp:855:43: error: ‘class greenlet::PythonState’ has no member named ‘current_frame’
  855 |     tstate->cframe->current_frame = this->current_frame;
      |                                           ^~~~~~~~~~~~~
src/greenlet/greenlet_greenlet.hpp: At global scope:
src/greenlet/greenlet_greenlet.hpp:910:6: error: variable or field ‘set_new_cframe’ declared void
  910 | void PythonState::set_new_cframe(CFrame& frame) G_NOEXCEPT
      |      ^~~~~~~~~~~
src/greenlet/greenlet_greenlet.hpp:910:34: error: ‘CFrame’ was not declared in this scope; did you mean ‘_frame’?
  910 | void PythonState::set_new_cframe(CFrame& frame) G_NOEXCEPT
      |                                  ^~~~~~
      |                                  _frame
src/greenlet/greenlet_greenlet.hpp:910:42: error: ‘frame’ was not declared in this scope; did you mean ‘_frame’?
  910 | void PythonState::set_new_cframe(CFrame& frame) G_NOEXCEPT
      |                                          ^~~~~
      |                                          _frame
src/greenlet/greenlet.cpp: In member function ‘virtual greenlet::Greenlet::switchstack_result_t greenlet::UserGreenlet::g_initialstub(void*)’:
src/greenlet/greenlet.cpp:1037:5: error: ‘CFrame’ was not declared in this scope; did you mean ‘_frame’?
 1037 |     CFrame trace_info;
      |     ^~~~~~
      |     _frame
src/greenlet/greenlet.cpp:1039:39: error: ‘trace_info’ was not declared in this scope
 1039 |     this->python_state.set_new_cframe(trace_info);
      |                                       ^~~~~~~~~~

I believe (but have not debugged/proven) that this is related to 18b5dd68c6b616257ae243c0b6bb965ffc885a23 / python/cpython#31530 / https://bugs.python.org/issue46836 which moved around some of the (internal) structs.

brandtbucher · 2022-03-01T17:05:57Z

Thanks for the update.

I'm working on seeing if we can revert those changes in CPython, rather than requiring another layer of compatibility macros here.

tacaswell · 2022-03-01T18:17:57Z

When these thing happen I should start with greenlet as other projects seem happier to adapt!

This is a very late dependency in my personal stack so I tend to get to it after dealing with cython/numpy/scipy/pybind11 etc.

brandtbucher · 2022-03-01T18:54:17Z

When these thing happen I should start with greenlet as other projects seem happier to adapt!

I'm not affiliated with Greenlet, but my team is interested in keeping a few projects like Greenlet working for dev versions of CPython (since we need them to benchmark our performance work).

zzzeek · 2022-05-14T14:03:27Z

Hi Greenlet -

This is really not my usual style as I am very familiar with people poking me about issues that will be fixed, "when they're fixed", but Python 3.11 is at 3.11b1 now, downstream distros are packaging things and being able to build greenlet on py311 is starting to really hold things up. Is there any workaround / patch / specific tag available so that downstream dependents like SQLAlchemy can move forward with Py 3.11? thanks for listening.

mike

zzzeek · 2022-05-14T17:07:56Z

looks like #302 does it, thanks @tacaswell

jamadden · 2022-05-14T18:14:01Z

I will work on getting that merged and released ASAP.

zzzeek · 2022-05-14T18:15:27Z

thanks! I have greenlet in our CI now with that patch so we're OK on this end. sorry for whining.

jamadden · 2022-05-14T18:16:53Z

Not at all! I know I haven't been as responsive as I used to be, or would like to be, the past few months. The occasional polite poke is appreciated.

tacaswell · 2022-05-14T21:05:20Z

Upstream CPython is aware of this issue and it is my understanding that this is considered a regression that they intend to fix.

python/cpython#92800 is tracking a related issue.

tacaswell · 2022-05-14T21:13:23Z

Also xref #301

Fixed issue where support for logging "stacklevel" implemented in :ticket:`7612` required adjustment to work with recently released Python 3.11.0b1, also repairs the unit tests which tested this feature. Install greenlet from a py311 compat patch. re: the stacklevel thing, this is going to be very inconvenient if we have to keep hardcoding numbers everywhere for every new python version Change-Id: I0c8f7293e98c0ca5cc544538284bfd1d3020cb1f References: python-greenlet/greenlet#288 Fixes: #8019

Fixed issue where support for logging "stacklevel" implemented in :ticket:`7612` required adjustment to work with recently released Python 3.11.0b1, also repairs the unit tests which tested this feature. Install greenlet from a py311 compat patch. re: the stacklevel thing, this is going to be very inconvenient if we have to keep hardcoding numbers everywhere for every new python version Change-Id: I0c8f7293e98c0ca5cc544538284bfd1d3020cb1f References: python-greenlet/greenlet#288 Fixes: #8019 (cherry picked from commit 43ff5b82dc0d91cacd625ac8943622ab340958c5)

vstinner · 2022-05-23T12:43:01Z

If I followed correctly, all issues discussed here have been fixed so I suggest closing this issue.

See the follow-up issue: "Schedule for a beta release of greenlet 2.0?" #303

tacaswell · 2022-05-23T20:33:06Z

I agree!

All breaking changes were fixed upstream.

tacaswell mentioned this issue Apr 30, 2022

py311 issues with const correctness #301

Closed

kod-kristoff mentioned this issue May 4, 2022

Enable python 3.11-dev in CI spraakbanken/karp-backend#205

Closed

zzzeek mentioned this issue May 14, 2022

python3.11: failing test_log_messages_have_correct_metadata_echo sqlalchemy/sqlalchemy#8019

Closed

vstinner mentioned this issue May 23, 2022

Schedule for a beta release of greenlet 2.0? #303

Closed

tacaswell closed this as completed May 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compatibility with Python3.11 (again) #288

Compatibility with Python3.11 (again) #288

tacaswell commented Jan 25, 2022

blackPantherOS commented Jan 31, 2022

brandtbucher commented Feb 1, 2022

ericsnowcurrently commented Feb 1, 2022

brandtbucher commented Feb 1, 2022

tacaswell commented Feb 1, 2022

blackPantherOS commented Feb 2, 2022

tacaswell commented Mar 1, 2022

brandtbucher commented Mar 1, 2022

tacaswell commented Mar 1, 2022

brandtbucher commented Mar 1, 2022

zzzeek commented May 14, 2022

zzzeek commented May 14, 2022

jamadden commented May 14, 2022

zzzeek commented May 14, 2022

jamadden commented May 14, 2022

tacaswell commented May 14, 2022

tacaswell commented May 14, 2022

vstinner commented May 23, 2022

tacaswell commented May 23, 2022

Compatibility with Python3.11 (again) #288

Compatibility with Python3.11 (again) #288

Comments

tacaswell commented Jan 25, 2022

blackPantherOS commented Jan 31, 2022

brandtbucher commented Feb 1, 2022

ericsnowcurrently commented Feb 1, 2022

brandtbucher commented Feb 1, 2022

tacaswell commented Feb 1, 2022

blackPantherOS commented Feb 2, 2022

tacaswell commented Mar 1, 2022

brandtbucher commented Mar 1, 2022

tacaswell commented Mar 1, 2022

brandtbucher commented Mar 1, 2022

zzzeek commented May 14, 2022

zzzeek commented May 14, 2022

jamadden commented May 14, 2022

zzzeek commented May 14, 2022

jamadden commented May 14, 2022

tacaswell commented May 14, 2022

tacaswell commented May 14, 2022

vstinner commented May 23, 2022

tacaswell commented May 23, 2022