newly release pypi package logged /usr/local/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject return f(*args, **kwds)

[dlamblin]

• version: 0.4.17
• Python version: cPython 3.7.0 downloaded from docker python:3.7.0-slim-stretch
• Operating System: debian stretch

Description:

My Apache Airflow FlaskAppBuilder webserver installed and ran fine 2h ago with v 0.4.16;
40 minutes ago the 0.4.17 package was published.
Making an unrelated config file change to my dockerfile triggered a rebuild with reinstall which didn't pin the greenlets package, and 0.4.17 was installed. Running the webserver then failed to serve any pages and logged an error:

/usr/local/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
  return f(*args, **kwds)
[2020-09-22 11:43:24 +0000] [628] [INFO] Starting gunicorn 20.0.4
[2020-09-22 11:43:24 +0000] [628] [INFO] Listening at: http://0.0.0.0:8443 (628)
[2020-09-22 11:43:24 +0000] [628] [INFO] Using worker: gevent
[2020-09-22 11:43:24 +0000] [631] [INFO] Booting worker with pid: 631
[2020-09-22 11:43:24 +0000] [632] [INFO] Booting worker with pid: 632
/usr/local/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
  return f(*args, **kwds)
[2020-09-22 11:43:24 +0000] [633] [INFO] Booting worker with pid: 633
/usr/local/lib/python3.7/site-packages/gunicorn/workers/ggevent.py:53: MonkeyPatchWarning: Monkey-patching ssl after ssl has already been imported may lead to errors, including RecursionError on Python 3.6. It may also silently lead to incorrect behaviour on Python 3.7. Please monkey-patch earlier. See https://github.com/gevent/gevent/issues/1016. Modules that had direct imports (NOT patched): ['urllib3.util.ssl_ (/usr/local/lib/python3.7/site-packages/urllib3/util/ssl_.py)', 'urllib3.util (/usr/local/lib/python3.7/site-packages/urllib3/util/__init__.py)'].
  monkey.patch_all()

Rebuilding with 0.4.16 and everything is fine again.

These constraints (failing to mention greenlets) were used: https://github.com/apache/airflow/blob/constraints-1.10.12/constraints-3.7.txt

[dlamblin]

OTOH maybe you aren't tight-knit with gevent; if so I filed this: gevent/gevent#1674

[snaury]

Yes, greenlet 0.4.17 adds a new context field to the greenlet object, but it shouldn't break binary compatibility, unless there's some embedding somewhere. Where is this warning logged?

[snaury]

Oh, it looks like gevent is subclassing a cdef class from greenlet. :( @jamadden FYI, today's release probably broke gevent...

[snaury]

I yanked 0.4.17 for the time being.

[jamadden]

Yup. This has come up before. I'll get a new release of gevent out built against that version ASAP. Users of older versions will want to pin their greenlet dependency.

In general, because many of the fields in the greenlet structure can be included or excluded based on C preprocessor definitions, even strict greenlet version pins can have this trouble.

[snaury]

Yup. This has come up before. I'll get a new release of gevent out built against that version ASAP.

Ah, cool! Then please un-yank 0.4.17 when it stops being an issue.

[dlamblin]

Ah, so even the same version could be built differently and lead to a similar issue. Hm. Well that's a tricky problem to have.
Thank you both for working on addressing this.

[jamadden]

gevent 20.9.0 was released at the same time greenlet 0.4.17 was un-yanked. On Python 3.7 and higher, gevent 20.9.0 is required to use (a standard build of) greenlet 0.4.17.

[oaosman84]

So is the fix to pin to 0.20.6 and 0.4.16? That works for me locally but for some reason our Heroku deployments (with the same Pipfile.lock) seems to be seeing the error.

[jamadden]

On Python 3.7, you either need gevent >= 1.4.0, < 20.9; greenlet >= 0.4.14, < 0.4.17 OR gevent >= 20.9; greenlet >= 0.4.17.

On Python 3.8 or 3.9, the minimum gevent version is 20.6.0, and the minimum greenlet version is 0.4.16.

I'd just use gevent>= 20.9; greenlet >= 0.4.17 everywhere.

[oaosman84]

I'm on python 3.8.5, gevent is 20.9.0 and greenlet is 0.4.17, and Heroku we get this error:

<frozen importlib._bootstrap>:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
<frozen importlib._bootstrap>:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
<frozen importlib._bootstrap>:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
<frozen importlib._bootstrap>:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
/app/.heroku/python/lib/python3.8/site-packages/environ/environ.py:628: UserWarning: /app/.env doesn't exist - if you're not configuring your environment separately, create one.
...
Segmentation fault

[jamadden]

@oaosman84 I would double check your version of gevent. The size 144 is what gevent compiled against greenlet 0.4.16 would expect. But the actual size of 152 is what greenlet 0.4.17 has.

If you're sure about gevent being 20.9, then check how it was built. If you're using the manylinux wheels, I wouldn't expect a problem. I just double checked the 64-bit 3.8 manylinux wheel and it's built correctly AFAICS. But if you're building or built gevent yourself, then it may not be correct. If you used pip wheel, things should be fine because that reads pyproject.toml and installs the matching version of greenlet. But if you just use python setup.py bdist_wheel or something similar, that will use the version of greenlet installed in the environment, which may not be correct.

[oaosman84]

@oaosman84 I would double check your version of gevent. The size 144 is what gevent compiled against greenlet 0.4.16 would expect. But the actual size of 152 is what greenlet 0.4.17 has.

If you're sure about gevent being 20.9, then check how it was built. If you're using the manylinux wheels, I wouldn't expect a problem. I just double checked the 64-bit 3.8 manylinux wheel and it's built correctly AFAICS. But if you're building or built gevent yourself, then it may not be correct. If you used pip wheel, things should be fine because that reads pyproject.toml and installs the matching version of greenlet. But if you just use python setup.py bdist_wheel or something similar, that will use the version of greenlet installed in the environment, which may not be correct.

Thanks, that makes sense. We're just using pipenv on Heroku, and our Pipfile.lock has greenlet at ==0.4.17 and gevent at ==20.9.0. The odd thing is it deploys on a new fresh app, but on old apps, it fails (the old apps were running 0.4.16 / 20.6). But when Heroku deploys a new version of the app it shouldn't matter what the old versions are running, but I suspect there might be something weird going on with Heroku.

[oaosman84]

What's the best way to confirm the versions that are actually running?

[jamadden]

As long as you're in the right environment, pip subcommands like pip freeze or pip list work, as would something like

$ python -c 'import gevent; print("gevent", gevent.__version__); import greenlet; print("greenlet", greenlet.__version__)'
gevent 20.9.1.dev0
greenlet 0.4.17

[oaosman84]

Thanks so much for helping with this, but I'm kind of desperate here as I can't really deploy our app right now.

So here's some fun, this is the output of the print command:

python -c 'import gevent; print("gevent", gevent.__version__); import greenlet; print("greenlet", greenlet.__version__)'
<frozen importlib._bootstrap>:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
<frozen importlib._bootstrap>:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
<frozen importlib._bootstrap>:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
<frozen importlib._bootstrap>:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
gevent 20.9.0
greenlet 0.4.17

So it does look like they're both on the right versions (20.9.0 / 0.4.17), but the binary incompatibility warning is there.

[jamadden]

All I can guess is that your gevent version was incorrectly compiled. You may need to wipe out all the local wheel caches (e.g., pip's download and build cache) and start fresh.

[doubler]

the new released greenlet==2.0.1 seams conflict with gevent==20.9.0

[wanjunling168]

the new released greenlet==2.0.1 seams conflict with gevent==20.9.0

Yes! Anybody can tell which version of gevent can work with greenlet==2.0.1?

Update: conda install -c conda-forge gevent==22.10.1 greenlet==1.1.3 is the newest combination I can found. conda version of newest gevent==22.10.1 only support greenlet >=1.1.3,<2.0

[jamadden]

gevent 22.10.2 requires greenlet 2; all previous versions of gevent require greenlet 1 or earlier.

gevent 20.12.0 and newer have lower and upper bound constraints on the version of greenlet they need, i.e., greenlet >= 0.4.17, < 2.0; platform_python_implementation=="CPython". Unfortunately, older versions of gevent (like 20.9.0) do not have the upper bound constraint and will incorrectly let you install greenlet 2, but they will not work with it.