Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slack Socket mode does not auto-reconnect to MySQL #355

Open
smk4664 opened this issue Dec 20, 2024 · 0 comments · May be fixed by #356
Open

Slack Socket mode does not auto-reconnect to MySQL #355

smk4664 opened this issue Dec 20, 2024 · 0 comments · May be fixed by #356
Assignees

Comments

@smk4664
Copy link
Contributor

smk4664 commented Dec 20, 2024

          Thank you @smk4664 ! I can confirm it solves the issue I reported - but I stumbled on something else that might be related: chatops in slack mode is not recovering from a closed database connection. The rest of nautobot was working fine, but chatops would log the following and not respond to the command:
2024-12-19 19:19:41,160 - slack_sdk.socket_mode.aiohttp - DEBUG - Received message (type: TEXT, data: {"envelope_id":"bafaf21e-a091-4240-9241-14b5ff5f03f8","payload":{"token":"XXX","team_id":"XXX","team_domain":"XXX","channel_id":"XXX","channel_name":"XXX","user_id":"XXX","user_name":"XXX","command":"\/whatever","text":"","api_app_id":"XXX","is_enterprise_install":"false","response_url":"XXX","trigger_id":"XXX"},"type":"slash_commands","accepts_response_payload":true}, extra: , session: s_8727666920834)
2024-12-19 19:19:41,161 - slack_sdk.socket_mode.aiohttp - DEBUG - A new message enqueued (current queue size: 1, session: s_8727666920834)
2024-12-19 19:19:41,161 - slack_sdk.socket_mode.aiohttp - DEBUG - Message processing started (type: slash_commands, envelope_id: bafaf21e-a091-4240-9241-14b5ff5f03f8, session: s_8727666920834)
2024-12-19 19:19:41,161 - slack_sdk.socket_mode.aiohttp - DEBUG - Worker received a socket request.
2024-12-19 19:19:41,161 - slack_sdk.socket_mode.aiohttp - DEBUG - Received slash command.
2024-12-19 19:19:41,161 - slack_sdk.socket_mode.aiohttp - DEBUG - Sending a message: {"envelope_id": "bafaf21e-a091-4240-9241-14b5ff5f03f8"} from session: s_8727666920834
2024-12-19 19:19:41,161 - slack_sdk.socket_mode.aiohttp - DEBUG - Processing slash command.
2024-12-19 19:19:41,162 - slack_sdk.socket_mode.aiohttp - ERROR - Failed to run a request listener: (2006, 'Server has gone away'), session: s_8727666920834
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/mysql/base.py", line 75, in execute
    return self.cursor.execute(query, args)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 179, in execute
    res = self._query(mogrified_query)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 330, in _query
    db.query(q)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/connections.py", line 265, in query
    _mysql.connection.query(self, query)
MySQLdb.OperationalError: (2006, 'Server has gone away')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/nautobot/.local/lib/python3.9/site-packages/slack_sdk/socket_mode/async_client.py", line 156, in run_message_listeners
    await listener(self, request)  # type: ignore
  File "/opt/nautobot/.local/lib/python3.9/site-packages/nautobot_chatops/sockets/slack.py", line 37, in process
    await process_slash_command(client, req)
  File "/opt/nautobot/.local/lib/python3.9/site-packages/nautobot_chatops/sockets/slack.py", line 80, in process_slash_command
    return await socket_check_and_enqueue_command(registry, command, subcommand, params, context, SlackDispatcher)
  File "/usr/local/lib/python3.9/site-packages/asgiref/sync.py", line 468, in __call__
    ret = await asyncio.shield(exec_coro)
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.9/site-packages/asgiref/sync.py", line 522, in thread_handler
    return func(*args, **kwargs)
  File "/opt/nautobot/.local/lib/python3.9/site-packages/nautobot_chatops/utils.py", line 119, in socket_check_and_enqueue_command
    return check_and_enqueue_command(*args, **kwargs)
  File "/opt/nautobot/.local/lib/python3.9/site-packages/nautobot_chatops/utils.py", line 157, in check_and_enqueue_command
    command_log = create_command_log(dispatcher, registry, command, subcommand, params)
  File "/opt/nautobot/.local/lib/python3.9/site-packages/nautobot_chatops/utils.py", line 112, in create_command_log
    nautobot_user=dispatcher.user,
  File "/opt/nautobot/.local/lib/python3.9/site-packages/nautobot_chatops/dispatchers/base.py", line 52, in user
    return ChatOpsAccountLink.objects.get(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 633, in get
    num = len(clone)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 380, in __len__
    self._fetch_all()
  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 1881, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 91, in __iter__
    results = compiler.execute_sql(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 102, in execute
    return super().execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/mysql/base.py", line 75, in execute
    return self.cursor.execute(query, args)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 179, in execute
    res = self._query(mogrified_query)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 330, in _query
    db.query(q)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/connections.py", line 265, in query
    _mysql.connection.query(self, query)
django.db.utils.OperationalError: (2006, 'Server has gone away')
2024-12-19 19:19:41,163 - slack_sdk.socket_mode.aiohttp - DEBUG - Message processing completed (type: slash_commands, envelope_id: bafaf21e-a091-4240-9241-14b5ff5f03f8, session: s_8727666920834)

I've done some light reading about this, and it might be enough to replace @sync_to_async with @database_sync_to_async in sockets/slack.py and nautobot_chatops/utils.py
https://channels.readthedocs.io/en/latest/topics/databases.html#database-sync-to-async
https://stackoverflow.com/a/68938365

Originally posted by @gioccher in #348 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant