creating new greenlets appears to leak memory rapidly in python 3.11 / linux

[zzzeek]

hi there -

This is indirect from SQLAlchemy where we use greenlet for our asyncio implementation, reported in sqlalchemy/sqlalchemy#8763.

The below script creates a greenlet and calls one switch. Running on linux under any Python 3.11 version with greenlet installed from pypi (greenlet-2.0.0.post0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl), the script below grows by about 10M per second. Under Python 3.10, memory is flat at about 10 megs resident.

import greenlet
import time


def greenlet_spawn(fn, *args, **kwargs):
    context = greenlet.greenlet(fn, greenlet.getcurrent())
    result = context.switch(*args, **kwargs)
    return result



def hi():
    time.sleep(.001)
    return "hello"

def go():
    while True:
        print(greenlet_spawn(hi))
go()

[salty-horse]

Previously reported here: gevent/gevent#1924

[jamadden]

This issue should be solved by using the just-released greenlet 2.0.1.

[zzzeek]

thanks!!!

[zzzeek]

The test program above works now however in our integration case we are still seeing rapid memory increases, unfortunately. I will attempt to devise a new test case.

[zzzeek]

hMmmmmm OK I think you are good actually, there now seems to be an entirely different memory leak, again python 3.11 only , either in the sqlite3 module or the aiosqlite module. how strange huh? I'll keep you posted but in our original original case with asyncpg (which is the important driver), memory seems to be stable.

[zzzeek]

OK we're good here, aiosqlite has some problem, strange, but i can reproduce without greenlet involved. thanks for the quick fix!

[zzzeek]

@jamadden I'm still in the disbelief stage, but it appears python 3.11 itself has a similar problem as the one here, with threading.Thread(). That is, the program below which is the threading equivalent also leaks memory at a similar rate:

from threading import Thread

while True:
    t1 = Thread()
    t1.start()

I'm going to quadruple check this before reporting to them but do you think the issue you found here might be similarly involved for Python's own Thread object?

[jamadden]

I'm going to quadruple check this before reporting to them but do you think the issue you found here might be similarly involved for Python's own Thread object?

I don't think so, for a couple reasons.

First, the thread never executes much in the way of Python bytecode, so it shouldn't be needing to allocate many of the "chunks" that were the problem.

Second, the reason they were an issue in greenlet is that we're switching chunks within the same thread, something that CPython should never be doing (the chunks should be thread local).

Lastly, comparing the greenlet allocations that showed the chunks as the source of the memory with the thread-based allocations gives a very different pattern:

High water mark of allocated heap + VM memory: 357.7 MB  at malloc stack log record index 114,933,363
     Current total allocated heap + VM memory: 357.7 MB  at malloc stack log record index 114,933,389
  Cumulative total allocated heap + VM memory: 18.52 GB
            Total heap + VM memory operations: 114,933,389  allocations:  58,369,435  deallocations:  56,563,954

Call graph:
    912525 (779M) << TOTAL >>
      892961 (307M) new_threadstate  (in Python) + 43  [0x1023f62eb]
      282 (282M) deque_new  (in Python) + 50  [0x10244a5d2]
      + 282 (282M) mach_vm_map  (in libsystem_kernel.dylib) + 0  [0x7ff8029047a5]
      74 (74.0M) new_threadstate  (in Python) + 43  [0x1023f62eb]
      + 74 (74.0M) mach_vm_map  (in libsystem_kernel.dylib) + 0  [0x7ff8029047a5]
      3 (48.0M) PyThread_start_new_thread  (in Python) + 197  [0x10240b4b5]
      + 3 (48.0M) _pthread_create  (in libsystem_pthread.dylib) + 0  [0x7ff802941f98]

[zzzeek]

OK! I hope they find this useful when they come in to look at this issue (also surprised noone has triaged it yet, they are usually pretty quick)

[pratik0907]

I'm using greenlet version 3.0.3 and still facing this issue

[salty-horse]

I'm using greenlet version 3.0.3 and still facing this issue

Have you tested with the script at the top? It runs fine for me (no memory leaks) with greenlet 3.0.3 on Python 3.12.3.

[pratik0907]

Yes,
Python version: 3.11.6
Greenlet version: 3.0.3

[salty-horse]

Python 3.11.6 is also working fine for me.

[pratik0907]

Not working for me, memory utilization keeps on increasing with every request. (Replicated in local env)

[pratik0907]

python = "^3.11"
uvicorn = {extras = ["standard"], version = "^0.25.0"}
fastapi = "^0.108.0"
pydantic = "^2.5.3"
celery = "^5.3.6"
aiohttp = "^3.9.1"
python-dotenv = "^1.0.0"
asyncpg = "^0.29.0"
python-json-logger = "^2.0.7"
tenacity = "^8.2.3"
sqlalchemy = "^2.0.24"
alembic = "^1.13.1"
ldclient-py = "^6.9.1"
sentry-sdk = "^1.39.1"
newrelic = "^9.3.0"
greenlet = "^3.0.3"

[zzzeek]

can't reproduce here with Python 3.11.6 and greenlet 3.0.3, memory stays fixed

 11447 classic   20   0  233292  10112   7296 S   2.0   0.0   0:00.73 python                                                                                                               

]$ ~/.venv311/bin/python -V
Python 3.11.6
$ ~/.venv311/bin/pip freeze | grep greenlet
greenlet==3.0.3

[greatitman]

I also have the same problem as pratik0907 。 I haved tried python3.10 3.11，the mem leak still exists。 How about you now @pratik0907