Celery error: [Errno 111] Connection refused

[ampil]

I am trying to setup email alerts & reporting in Superset.
But could not start celery, it returns an error kombu.exceptions.OperationalError: [Errno 111] Connection refused.

There is a similar error on celery repo: celery/kombu#1582
But I am not sure where to apply the fix

from .celery import app as CRM_CELERY_APP

__all__ = ["CRM_CELERY_APP"]

Extracts from superset_config.py related to Celery / Redis configuration.

from celery.schedules import crontab

...

# Enable alerts in Superset
EMAIL_NOTIFICATIONS = True  
ALERT_REPORTS_NOTIFICATION_DRY_RUN = False
DEFAULT_FEATURE_FLAGS: dict[str, bool] = {
    "ALERT_REPORTS": True,
}



# Celery Redis configuration for async query execution
REDIS_HOST = "localhost"
REDIS_PORT = "6379"

class CeleryConfig:
    broker_url = f"redis://{REDIS_HOST}:{REDIS_PORT}/0"
    imports = (
        "superset.sql_lab",
        "superset.tasks.scheduler",
    )
    result_backend = f"redis://{REDIS_HOST}:{REDIS_PORT}/0"
    worker_prefetch_multiplier = 10
    task_acks_late = True
    task_annotations = {
        "sql_lab.get_sql_results": {
            "rate_limit": "100/s",
        },
    }
    beat_schedule = {
        "reports.scheduler": {
            "task": "reports.scheduler",
            "schedule": crontab(minute="*", hour="*"),
        },
        "reports.prune_log": {
            "task": "reports.prune_log",
            "schedule": crontab(minute=0, hour=0),
        },
    }

CELERY_CONFIG = CeleryConfig

# On Redis
from flask_caching.backends.rediscache import RedisCache
RESULTS_BACKEND = RedisCache(
    host=REDIS_HOST, port=REDIS_PORT, key_prefix='superset_results')

celery inspect ping returns an error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/kombu/connection.py", line 472, in _reraise_as_library_errors
    yield
  File "/usr/local/lib/python3.9/site-packages/kombu/connection.py", line 459, in _ensure_connection
    return retry_over_time(
  File "/usr/local/lib/python3.9/site-packages/kombu/utils/functional.py", line 318, in retry_over_time
    return fun(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/kombu/connection.py", line 934, in _connection_factory
    self._connection = self._establish_connection()
  File "/usr/local/lib/python3.9/site-packages/kombu/connection.py", line 860, in _establish_connection
    conn = self.transport.establish_connection()
  File "/usr/local/lib/python3.9/site-packages/kombu/transport/pyamqp.py", line 203, in establish_connection
    conn.connect()
  File "/usr/local/lib/python3.9/site-packages/amqp/connection.py", line 323, in connect
    self.transport.connect()
  File "/usr/local/lib/python3.9/site-packages/amqp/transport.py", line 129, in connect
    self._connect(self.host, self.port, self.connect_timeout)
  File "/usr/local/lib/python3.9/site-packages/amqp/transport.py", line 184, in _connect
    self.sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/celery", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/celery/__main__.py", line 15, in main
    sys.exit(_main())
  File "/usr/local/lib/python3.9/site-packages/celery/bin/celery.py", line 236, in main
    return celery(auto_envvar_prefix="CELERY")
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/celery/bin/base.py", line 135, in caller
    return f(ctx, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/celery/bin/control.py", line 186, in inspect
    replies = inspect._request(command, **arguments)
  File "/usr/local/lib/python3.9/site-packages/celery/app/control.py", line 106, in _request
    return self._prepare(self.app.control.broadcast(
  File "/usr/local/lib/python3.9/site-packages/celery/app/control.py", line 776, in broadcast
    return self.mailbox(conn)._broadcast(
  File "/usr/local/lib/python3.9/site-packages/kombu/pidbox.py", line 330, in _broadcast
    chan = channel or self.connection.default_channel
  File "/usr/local/lib/python3.9/site-packages/kombu/connection.py", line 953, in default_channel
    self._ensure_connection(**conn_opts)
  File "/usr/local/lib/python3.9/site-packages/kombu/connection.py", line 459, in _ensure_connection
    return retry_over_time(
  File "/usr/lib64/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.9/site-packages/kombu/connection.py", line 476, in _reraise_as_library_errors
    raise ConnectionError(str(exc)) from exc
kombu.exceptions.OperationalError: [Errno 111] Connection refused

@dosu, any help?
Either on where to apply the fix or other ideas how to fix it?

[dosubot[bot]]

To resolve the kombu.exceptions.OperationalError: [Errno 111] Connection refused error when setting up Celery with Redis in Superset, ensure that your Redis server is running and accessible at the specified host and port. Here are the steps you should follow:

1. Verify Redis Server Status: Make sure that the Redis server is running. You can check this by trying to connect to Redis using a Redis client or by running a command like redis-cli ping on the server where Redis is hosted.

2. Check Configuration: Ensure that the Redis connection details in your superset_config.py are correct. Specifically, verify the REDIS_HOST and REDIS_PORT settings:

   REDIS_HOST = "localhost"
   REDIS_PORT = "6379"

3. Celery Configuration: Ensure that the broker_url and result_backend in the CeleryConfig class are correctly set to use the Redis server:

   class CeleryConfig:
       broker_url = f"redis://{REDIS_HOST}:{REDIS_PORT}/0"
       result_backend = f"redis://{REDIS_HOST}:{REDIS_PORT}/0"
       # Other configurations...

4. Network Accessibility: If Redis is running on a different server, ensure that there are no network issues preventing access from the Superset server to the Redis server. This might involve checking firewall settings or network configurations.

5. Redis Authentication: If your Redis server requires authentication, make sure to include the username and password in the Redis URL. Update your configuration to include these credentials if necessary.

By following these steps, you should be able to resolve the connection issue with Redis and get Celery running for email alerts and reporting in Superset ^[1][2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[ampil]

1. redis-cli ping returns PONG
2. Configuration is REDIS_HOST = "localhost" , REDIS_PORT = "6379"
3. Celery configuration:

    broker_url = f"redis://{REDIS_HOST}:{REDIS_PORT}/0"
    result_backend = f"redis://{REDIS_HOST}:{REDIS_PORT}/0"

4. Redis is running on the same server, port is open for IPv4

telnet localhost 6379
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.

5. Redis does not require authentication

Still, the error persists.

@dosu, any help?

[dosubot[bot]]

Since you've confirmed that Redis is running and accessible, and your configuration seems correct, let's try a few more steps to troubleshoot the issue:

1. Check Celery Worker Logs: Look at the logs for the Celery worker to see if there are any additional error messages or warnings that might provide more context about the connection issue.

2. IPv6 Configuration: The telnet output shows a connection refusal on ::1 (IPv6). Ensure that your Redis server is configured to listen on both IPv4 and IPv6, or explicitly configure it to use IPv4 by setting bind 127.0.0.1 in your Redis configuration file (redis.conf).

3. Firewall and SELinux: Double-check that there are no firewall rules or SELinux policies blocking the connection. Even though you mentioned the port is open, it's worth verifying these settings.

4. Celery Version Compatibility: Ensure that the Celery version you are using is compatible with your Redis setup. Superset specifies Celery version 5.4.0, so make sure you are using this version ^[1].

5. Restart Services: Sometimes, simply restarting the Redis server and Celery workers can resolve transient issues. Try restarting both services to see if the problem persists.

6. Network Configuration: If your server has multiple network interfaces, ensure that the correct interface is being used for the connection. You might want to specify the exact IP address instead of localhost to avoid any potential DNS resolution issues.

If these steps don't resolve the issue, consider enabling more verbose logging for Celery to gather additional information about the connection attempts.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[ampil]

I made it to work. What helped:

• migrating Superset's db from SQLite to PostgreSQL (that's a must according to the docs)
• adding new lines to the file /superset/__init__.py (see a celery issue)

from celery import app as celery_app
__all__ = ['celery_app']

After these steps, celery is functioning properly.
However, when I run celery inspect ping it still returns the same error but celery continues to work.