Exception ignored in: <function Connection.__del__ at 0x7f11449b2af0>

[dbobrov]

Version:
redis-py: 3.5.0
redis: 6.0.1

Description:
After commit 200e4df when exception catching was removed from __del__ methods. I faced this error after run tests in Django. With previous version of redis-py (3.4.1) everything was OK.

Exception ignored in: <function Connection.__del__ at 0x7f11449b2af0>
Traceback (most recent call last):
  File "/home/dmitry/dev/test-project/env/lib/python3.8/site-packages/redis/connection.py", line 537, in __del__
  File "/home/dmitry/dev/test-project/env/lib/python3.8/site-packages/redis/connection.py", line 667, in disconnect
TypeError: catching classes that do not inherit from BaseException is not allowed

[cormac-yobota]

👍 Seeing this issue aswell. Happens when a SIGTERM is fired to the worker.

Version:
redis-py: 3.5.1
redis: 3.0.6
python: 3.6

Having done a bit of basic investigation it seem to be the use of socket.error that's the cause.
https://github.com/andymccurdy/redis-py/blob/3.5.1/redis/connection.py#L667
Changing this to OSError seems to have resolved this acute problem. Although now I'm faced with another issue:

Exception ignored in: <bound method Connection.__del__ of Connection<host=localhost,port=6379,db=0>>
Traceback (most recent call last):
    File "/home/vagrant/env/lib/python3.6/site-packages/redis/connection.py", line 537, in __del__
    File "/home/vagrant/env/lib/python3.6/site-packages/redis/connection.py", line 664, in disconnect
TypeError: 'NoneType' object is not callable

I'll dig a bit deeper when I get a chance.

[andymccurdy]

@jdufresne any thoughts on why this might be happening?

In the first scenario, I'm not sure when socket.error would not derive from BaseException

In the second, the only attribute look up on line 664 is os.getpid, which could be that the os module has been unloaded by the GC prior to Connection.__del__ from being run.

[jdufresne]

I've been seeing this in my own project as well, I can try to dive into it.

[jplehmann]

I'm getting this also, when used in conjunction with django-redis.

[jdufresne]

I had a chance to look into this. Here is what I believe is happening:

The __del__ method is called during interpreter shutdown. As documented by Python, during interpreter shutdown global variables are deleted and set to None. This error is happening due to the os.getpid reference. As it has become None at shutdown, the result is:

Exception ignored in: <bound method Connection.__del__ of Connection<host=localhost,port=6379,db=0>>
Traceback (most recent call last):
  File ".../redis-py/redis/connection.py", line 537, in __del__
  File ".../redis-py/redis/connection.py", line 664, in disconnect
TypeError: 'NoneType' object is not callable

Here is the Python warning on this:

__del__() can be executed during interpreter shutdown. As a consequence, the global variables it needs to access (including other modules) may already have been deleted or set to None. Python guarantees that globals whose name begins with a single underscore are deleted from their module before other globals are deleted; if no other references to such globals exist, this may help in assuring that imported modules are still available at the time when the __del__() method is called.

IMO, this is a bug in the code using redis-py and not redis-py itself. The library users are responsible for closing resources they have created. In this case, a redis connection. Please add an explicit call in your code to close the redis connection rather than relying on the __del__ finalizer.

An improvement to redis-py would be to remove these finalizers entirely. This has been done in PR #1287. In Python, sockets already have a finalizer to close them. So the __del__ methods aren't doing anything that isn't already being done by the Python interpreter and stdlib. Doing this will remove the TypeError: 'NoneType' object is not callable but alos better indicate to library users where their resource mismanagement is through the emitted ResourceWarning.

[jdufresne]

I did notice this, in order to fully close the connection, the following pattern was required.

with redis.Redis(host) as conn:
    # Do stuff with conn
conn.connection_pool.disconnect()

Only then, did the exceptions/warnings stop. The context manager releases the connection back to the connection pool, but doesn't close the connection.

[dbobrov]

I've add connection close method in cache class.

class RedisCache(BaseCache):
    def __init__(self, location, params):
        # ...
        self.redis = redis.Redis(host, port, db=0)

    def close(self):
        self.redis.connection_pool.disconnect()

And call this method in tearDown method in all my app's tests.

class IndexUserClientTest(TestCase):
    def tearDown(self):
        cache.close()

This is solved my issue.

[andymccurdy]

I'm struggling with what to do here. In an ideal world, I agree with @jdufresne that users should be responsible for cleaning up the resources that they allocate. And that's where I'd like to get to. However redis-py 2.x/3.x has historically hidden much of the resource allocation and cleanup from users. This is further complicated by the use of a connection pool. In long running applications (like a web app), resource cleanup is more about returning the connection back to the pool rather than closing the socket, etc.

I think getting to a place where users retrieve client objects, pubsub objects, pipeline objects, etc. from a global manager object via context managers is the right way to go. This is a fundamental API change and probably should be a 4.0 goal. Especially since we've already stated that 3.5 will be the last 3.x release and we should most certainly not make such a change in a bugfix version like 3.5.x.

I'm leaning towards restoring the try/except clause in the __del__ methods and releasing another 3.5.x version to essentially silence these errors. This way we can focus on moving forward with 4.0 where we can actually do the right thing and remove these entirely while providing a more intuitive resource allocation model to users.

Thoughts?

[jdufresne]

I think getting to a place where users retrieve client objects, pubsub objects, pipeline objects, etc. from a global manager object via context managers is the right way to go.

+1. This sounds like a good idea to me.

[andymccurdy]

I've restored the try/except clauses in __del__ methods. I'll be releasing 3.5.3 shortly.

[chebee7i]

Is 3.5.3 coming out soon? We use a large redis cluster, so there are quite a few of these messages popping up anytime we use the library.

[andymccurdy]

3.5.3 is out with this fix.

[collinsethans]

I am facing this same issue while trying out redis-py (v: 5.2.0) on python (v: 3.13.0). A simple way to reproduce this is:

# python
# Python 3.13.0 | packaged by conda-forge | (main, Oct  8 2024, 20:04:32) [GCC 13.3.0] on linux

import redis
r_app = redis.Redis(unix_socket_path='/tmp/redis_app.sock', db=0)

# Use either 'exit' or Ctrl-D
exit
Exception ignored in: <function Redis.__del__ at 0x7f419d312840>
Traceback (most recent call last):
  File "/aux/apps_install/mc3/envs/16OCT24/lib/python3.13/site-packages/redis/client.py", line 520, in __del__
  File "/aux/apps_install/mc3/envs/16OCT24/lib/python3.13/site-packages/redis/client.py", line 535, in close
  File "/aux/apps_install/mc3/envs/16OCT24/lib/python3.13/site-packages/redis/connection.py", line 1497, in disconnect
  File "/aux/apps_install/mc3/envs/16OCT24/lib/python3.13/site-packages/redis/connection.py", line 1398, in _checkpid
TypeError: 'NoneType' object is not callable

Packages:

$ conda list | grep redis
21:hiredis                   3.0.0                    pypi_0    pypi
46:redis                     5.2.0                    pypi_0    pypi