Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Creating a handful (~3) of rooms, for example to populate a new Space in Element Web, leads to running into rate limits #14312

Closed
reivilibre opened this issue Oct 27, 2022 · 6 comments · Fixed by #14314
Closed

Creating a handful (~3) of rooms, for example to populate a new Space in Element Web, leads to running into rate limits #14312

reivilibre opened this issue Oct 27, 2022 · 6 comments · Fixed by #14314
Assignees
@reivilibre
Labels
O-Occasional Affects or can be seen by some users regularly or most users rarely S-Major Major functionality / product severely impaired, no satisfactory workaround. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. X-Release-Blocker Must be resolved before making a release

Comments

@reivilibre
Copy link
Contributor

With default rate limiting values, we used to be able to create at least 3 rooms in one go. The new Space screen in Element Web depends on this (element-hq/element-web#23620) and breaks if any of those room creations are rate limited, as they are now (v1.70.0).

The following test case demonstrates this:

from http import HTTPStatus

from synapse.rest.client import room
from tests.unittest import HomeserverTestCase, override_config


class BisectCase(HomeserverTestCase):
    servlets = [room.register_servlets]
    hijack_auth = True
    user_id = "@sid1:test"

    @override_config({
        "rc_message": {"per_second": 0.2, "burst_count": 10},
    })
    def test_can_create_3_rooms(self):
        """
        Should be able to create 3 rooms in a row (e.g. rooms in a space)
        without hitting rate limits.
        """

        for _ in range(3):
            channel = self.make_request(
                "POST",
                "/createRoom",
                {},
            )
            self.assertEqual(channel.code, HTTPStatus.OK, channel.json_body)

Bisecting, the following commit is what made the above fail because of the rate limit: 8ab16a9 'Persist CreateRoom events to DB in a batch (#13800)' in v1.69.0.

We should try to revert rate limiting to behave the same as before or closely enough that this basic use case is not broken.
@reivilibre reivilibre added S-Major Major functionality / product severely impaired, no satisfactory workaround. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. O-Occasional Affects or can be seen by some users regularly or most users rarely labels Oct 27, 2022
@clokep
Copy link
Member

clokep commented Oct 27, 2022

Is #14307 attempting to fix this?

@reivilibre reivilibre self-assigned this Oct 27, 2022
@reivilibre
Copy link
Contributor Author

Is #14307 attempting to fix this?

I don't know, but it doesn't pass

@reivilibre
Copy link
Contributor Author

These places contribute to the rate limiting in the above test:

	~~~~~ Ratelimited actions:

	  File "/home/rei/work/synapse/synapse/http/server.py", line 229, in wrapped_async_request_handler
	    await h(self, request)
	  File "/home/rei/work/synapse/synapse/http/server.py", line 307, in _async_render_wrapper
	    callback_return = await self._async_render(request)
	  File "/home/rei/work/synapse/synapse/http/server.py", line 513, in _async_render
	    callback_return = await raw_callback_return
	  File "/home/rei/work/synapse/synapse/rest/client/room.py", line 161, in on_POST
	    info, _ = await self._room_creation_handler.create_room(
	  File "/home/rei/work/synapse/synapse/handlers/room.py", line 756, in create_room
	    await self.request_ratelimiter.ratelimit(requester)
	  File "/home/rei/work/synapse/synapse/api/ratelimiting.py", line 391, in ratelimit
	    tb = "".join(traceback.format_stack())
	
	~+~

	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1487, in _persist_events
	    event = await self.persist_and_notify_client_events(
	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1649, in persist_and_notify_client_events
	    await self.request_ratelimiter.ratelimit(
	  File "/home/rei/work/synapse/synapse/api/ratelimiting.py", line 391, in ratelimit
	    tb = "".join(traceback.format_stack())
	
	~+~

	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1487, in _persist_events
	    event = await self.persist_and_notify_client_events(
	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1649, in persist_and_notify_client_events
	    await self.request_ratelimiter.ratelimit(
	  File "/home/rei/work/synapse/synapse/api/ratelimiting.py", line 391, in ratelimit
	    tb = "".join(traceback.format_stack())
	
	~+~

	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1487, in _persist_events
	    event = await self.persist_and_notify_client_events(
	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1649, in persist_and_notify_client_events
	    await self.request_ratelimiter.ratelimit(
	  File "/home/rei/work/synapse/synapse/api/ratelimiting.py", line 391, in ratelimit
	    tb = "".join(traceback.format_stack())
	
	~+~

	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1487, in _persist_events
	    event = await self.persist_and_notify_client_events(
	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1649, in persist_and_notify_client_events
	    await self.request_ratelimiter.ratelimit(
	  File "/home/rei/work/synapse/synapse/api/ratelimiting.py", line 391, in ratelimit
	    tb = "".join(traceback.format_stack())
	
	~+~

	  File "/home/rei/work/synapse/synapse/http/server.py", line 229, in wrapped_async_request_handler
	    await h(self, request)
	  File "/home/rei/work/synapse/synapse/http/server.py", line 307, in _async_render_wrapper
	    callback_return = await self._async_render(request)
	  File "/home/rei/work/synapse/synapse/http/server.py", line 513, in _async_render
	    callback_return = await raw_callback_return
	  File "/home/rei/work/synapse/synapse/rest/client/room.py", line 161, in on_POST
	    info, _ = await self._room_creation_handler.create_room(
	  File "/home/rei/work/synapse/synapse/handlers/room.py", line 756, in create_room
	    await self.request_ratelimiter.ratelimit(requester)
	  File "/home/rei/work/synapse/synapse/api/ratelimiting.py", line 391, in ratelimit
	    tb = "".join(traceback.format_stack())
	
	~+~

	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1487, in _persist_events
	    event = await self.persist_and_notify_client_events(
	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1649, in persist_and_notify_client_events
	    await self.request_ratelimiter.ratelimit(
	  File "/home/rei/work/synapse/synapse/api/ratelimiting.py", line 391, in ratelimit
	    tb = "".join(traceback.format_stack())
	
	~+~

	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1487, in _persist_events
	    event = await self.persist_and_notify_client_events(
	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1649, in persist_and_notify_client_events
	    await self.request_ratelimiter.ratelimit(
	  File "/home/rei/work/synapse/synapse/api/ratelimiting.py", line 391, in ratelimit
	    tb = "".join(traceback.format_stack())
	
	~+~

	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1487, in _persist_events
	    event = await self.persist_and_notify_client_events(
	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1649, in persist_and_notify_client_events
	    await self.request_ratelimiter.ratelimit(
	  File "/home/rei/work/synapse/synapse/api/ratelimiting.py", line 391, in ratelimit
	    tb = "".join(traceback.format_stack())
	
	~+~

	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1487, in _persist_events
	    event = await self.persist_and_notify_client_events(
	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1649, in persist_and_notify_client_events
	    await self.request_ratelimiter.ratelimit(
	  File "/home/rei/work/synapse/synapse/api/ratelimiting.py", line 391, in ratelimit
	    tb = "".join(traceback.format_stack())
	
	~+~

	  File "/home/rei/work/synapse/synapse/http/server.py", line 229, in wrapped_async_request_handler
	    await h(self, request)
	  File "/home/rei/work/synapse/synapse/http/server.py", line 307, in _async_render_wrapper
	    callback_return = await self._async_render(request)
	  File "/home/rei/work/synapse/synapse/http/server.py", line 513, in _async_render
	    callback_return = await raw_callback_return
	  File "/home/rei/work/synapse/synapse/rest/client/room.py", line 161, in on_POST
	    info, _ = await self._room_creation_handler.create_room(
	  File "/home/rei/work/synapse/synapse/handlers/room.py", line 756, in create_room
	    await self.request_ratelimiter.ratelimit(requester)
	  File "/home/rei/work/synapse/synapse/api/ratelimiting.py", line 391, in ratelimit
	    tb = "".join(traceback.format_stack())
	
	~~~~~

One might argue that we shouldn't ratelimit the creation of events whilst creating a room at all, because then you easily end up with a partially-created room which is very janky. I'd be tempted to rate limit the room's creation but not the events that go inside (?)

@reivilibre
Copy link
Contributor Author

reivilibre commented Oct 27, 2022 •
edited
Loading

This is where the rate limits used to happen in v1.68.0:

	~~~~~ Ratelimited actions:
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
	    self._runCallbacks()
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 857, in _runCallbacks
	    current.result = callback(  # type: ignore[misc]
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1750, in gotResult
	    current_context.run(_inlineCallbacks, r, gen, status)
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
	    result = current_context.run(gen.send, result)
	  File "/home/rei/work/synapse/synapse/http/server.py", line 228, in wrapped_async_request_handler
	    await h(self, request)
	  File "/home/rei/work/synapse/synapse/http/server.py", line 306, in _async_render_wrapper
	    callback_return = await self._async_render(request)
	  File "/home/rei/work/synapse/synapse/http/server.py", line 512, in _async_render
	    callback_return = await raw_callback_return
	  File "/home/rei/work/synapse/synapse/rest/client/room.py", line 161, in on_POST
	    info, _ = await self._room_creation_handler.create_room(
	  File "/home/rei/work/synapse/synapse/handlers/room.py", line 759, in create_room
	    await self.request_ratelimiter.ratelimit(requester)
	  File "/home/rei/work/synapse/synapse/api/ratelimiting.py", line 391, in ratelimit
	    tb = "".join(traceback.format_stack(limit=10))
	
	~+~
	  File "/home/rei/work/synapse/synapse/util/metrics.py", line 113, in measured_func
	    r = await func(self, *args, **kwargs)
	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1327, in handle_new_client_event
	    run_in_background(
	  File "/home/rei/work/synapse/synapse/logging/context.py", line 814, in run_in_background
	    res = defer.ensureDeferred(res)
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1129, in ensureDeferred
	    return Deferred.fromCoroutine(coro)
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1105, in fromCoroutine
	    return _cancellableInlineCallbacks(coro)
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1815, in _cancellableInlineCallbacks
	    _inlineCallbacks(None, gen, status)
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
	    result = current_context.run(gen.send, result)
	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1415, in _persist_event
	    event = await self.persist_and_notify_client_event(
	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1567, in persist_and_notify_client_event
	    await self.request_ratelimiter.ratelimit(
	  File "/home/rei/work/synapse/synapse/api/ratelimiting.py", line 391, in ratelimit
	    tb = "".join(traceback.format_stack(limit=10))
	
	~+~
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
	    self._runCallbacks()
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 857, in _runCallbacks
	    current.result = callback(  # type: ignore[misc]
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1750, in gotResult
	    current_context.run(_inlineCallbacks, r, gen, status)
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
	    result = current_context.run(gen.send, result)
	  File "/home/rei/work/synapse/synapse/http/server.py", line 228, in wrapped_async_request_handler
	    await h(self, request)
	  File "/home/rei/work/synapse/synapse/http/server.py", line 306, in _async_render_wrapper
	    callback_return = await self._async_render(request)
	  File "/home/rei/work/synapse/synapse/http/server.py", line 512, in _async_render
	    callback_return = await raw_callback_return
	  File "/home/rei/work/synapse/synapse/rest/client/room.py", line 161, in on_POST
	    info, _ = await self._room_creation_handler.create_room(
	  File "/home/rei/work/synapse/synapse/handlers/room.py", line 759, in create_room
	    await self.request_ratelimiter.ratelimit(requester)
	  File "/home/rei/work/synapse/synapse/api/ratelimiting.py", line 391, in ratelimit
	    tb = "".join(traceback.format_stack(limit=10))
	
	~+~
	  File "/home/rei/work/synapse/synapse/util/metrics.py", line 113, in measured_func
	    r = await func(self, *args, **kwargs)
	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1327, in handle_new_client_event
	    run_in_background(
	  File "/home/rei/work/synapse/synapse/logging/context.py", line 814, in run_in_background
	    res = defer.ensureDeferred(res)
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1129, in ensureDeferred
	    return Deferred.fromCoroutine(coro)
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1105, in fromCoroutine
	    return _cancellableInlineCallbacks(coro)
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1815, in _cancellableInlineCallbacks
	    _inlineCallbacks(None, gen, status)
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
	    result = current_context.run(gen.send, result)
	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1415, in _persist_event
	    event = await self.persist_and_notify_client_event(
	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1567, in persist_and_notify_client_event
	    await self.request_ratelimiter.ratelimit(
	  File "/home/rei/work/synapse/synapse/api/ratelimiting.py", line 391, in ratelimit
	    tb = "".join(traceback.format_stack(limit=10))
	
	~+~
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
	    self._runCallbacks()
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 857, in _runCallbacks
	    current.result = callback(  # type: ignore[misc]
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1750, in gotResult
	    current_context.run(_inlineCallbacks, r, gen, status)
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
	    result = current_context.run(gen.send, result)
	  File "/home/rei/work/synapse/synapse/http/server.py", line 228, in wrapped_async_request_handler
	    await h(self, request)
	  File "/home/rei/work/synapse/synapse/http/server.py", line 306, in _async_render_wrapper
	    callback_return = await self._async_render(request)
	  File "/home/rei/work/synapse/synapse/http/server.py", line 512, in _async_render
	    callback_return = await raw_callback_return
	  File "/home/rei/work/synapse/synapse/rest/client/room.py", line 161, in on_POST
	    info, _ = await self._room_creation_handler.create_room(
	  File "/home/rei/work/synapse/synapse/handlers/room.py", line 759, in create_room
	    await self.request_ratelimiter.ratelimit(requester)
	  File "/home/rei/work/synapse/synapse/api/ratelimiting.py", line 391, in ratelimit
	    tb = "".join(traceback.format_stack(limit=10))
	
	~+~
	  File "/home/rei/work/synapse/synapse/util/metrics.py", line 113, in measured_func
	    r = await func(self, *args, **kwargs)
	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1327, in handle_new_client_event
	    run_in_background(
	  File "/home/rei/work/synapse/synapse/logging/context.py", line 814, in run_in_background
	    res = defer.ensureDeferred(res)
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1129, in ensureDeferred
	    return Deferred.fromCoroutine(coro)
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1105, in fromCoroutine
	    return _cancellableInlineCallbacks(coro)
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1815, in _cancellableInlineCallbacks
	    _inlineCallbacks(None, gen, status)
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
	    result = current_context.run(gen.send, result)
	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1415, in _persist_event
	    event = await self.persist_and_notify_client_event(
	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1567, in persist_and_notify_client_event
	    await self.request_ratelimiter.ratelimit(
	  File "/home/rei/work/synapse/synapse/api/ratelimiting.py", line 391, in ratelimit
	    tb = "".join(traceback.format_stack(limit=10))
	
	~+~
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
	    self._runCallbacks()
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 857, in _runCallbacks
	    current.result = callback(  # type: ignore[misc]
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1750, in gotResult
	    current_context.run(_inlineCallbacks, r, gen, status)
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
	    result = current_context.run(gen.send, result)
	  File "/home/rei/work/synapse/synapse/http/server.py", line 228, in wrapped_async_request_handler
	    await h(self, request)
	  File "/home/rei/work/synapse/synapse/http/server.py", line 306, in _async_render_wrapper
	    callback_return = await self._async_render(request)
	  File "/home/rei/work/synapse/synapse/http/server.py", line 512, in _async_render
	    callback_return = await raw_callback_return
	  File "/home/rei/work/synapse/synapse/rest/client/room.py", line 161, in on_POST
	    info, _ = await self._room_creation_handler.create_room(
	  File "/home/rei/work/synapse/synapse/handlers/room.py", line 759, in create_room
	    await self.request_ratelimiter.ratelimit(requester)
	  File "/home/rei/work/synapse/synapse/api/ratelimiting.py", line 391, in ratelimit
	    tb = "".join(traceback.format_stack(limit=10))
	
	~+~
	  File "/home/rei/work/synapse/synapse/util/metrics.py", line 113, in measured_func
	    r = await func(self, *args, **kwargs)
	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1327, in handle_new_client_event
	    run_in_background(
	  File "/home/rei/work/synapse/synapse/logging/context.py", line 814, in run_in_background
	    res = defer.ensureDeferred(res)
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1129, in ensureDeferred
	    return Deferred.fromCoroutine(coro)
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1105, in fromCoroutine
	    return _cancellableInlineCallbacks(coro)
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1815, in _cancellableInlineCallbacks
	    _inlineCallbacks(None, gen, status)
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
	    result = current_context.run(gen.send, result)
	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1415, in _persist_event
	    event = await self.persist_and_notify_client_event(
	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1567, in persist_and_notify_client_event
	    await self.request_ratelimiter.ratelimit(
	  File "/home/rei/work/synapse/synapse/api/ratelimiting.py", line 391, in ratelimit
	    tb = "".join(traceback.format_stack(limit=10))
	
	~+~
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
	    self._runCallbacks()
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 857, in _runCallbacks
	    current.result = callback(  # type: ignore[misc]
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1750, in gotResult
	    current_context.run(_inlineCallbacks, r, gen, status)
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
	    result = current_context.run(gen.send, result)
	  File "/home/rei/work/synapse/synapse/http/server.py", line 228, in wrapped_async_request_handler
	    await h(self, request)
	  File "/home/rei/work/synapse/synapse/http/server.py", line 306, in _async_render_wrapper
	    callback_return = await self._async_render(request)
	  File "/home/rei/work/synapse/synapse/http/server.py", line 512, in _async_render
	    callback_return = await raw_callback_return
	  File "/home/rei/work/synapse/synapse/rest/client/room.py", line 161, in on_POST
	    info, _ = await self._room_creation_handler.create_room(
	  File "/home/rei/work/synapse/synapse/handlers/room.py", line 759, in create_room
	    await self.request_ratelimiter.ratelimit(requester)
	  File "/home/rei/work/synapse/synapse/api/ratelimiting.py", line 391, in ratelimit
	    tb = "".join(traceback.format_stack(limit=10))
	
	~+~
	  File "/home/rei/work/synapse/synapse/util/metrics.py", line 113, in measured_func
	    r = await func(self, *args, **kwargs)
	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1327, in handle_new_client_event
	    run_in_background(
	  File "/home/rei/work/synapse/synapse/logging/context.py", line 814, in run_in_background
	    res = defer.ensureDeferred(res)
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1129, in ensureDeferred
	    return Deferred.fromCoroutine(coro)
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1105, in fromCoroutine
	    return _cancellableInlineCallbacks(coro)
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1815, in _cancellableInlineCallbacks
	    _inlineCallbacks(None, gen, status)
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
	    result = current_context.run(gen.send, result)
	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1415, in _persist_event
	    event = await self.persist_and_notify_client_event(
	  File "/home/rei/work/synapse/synapse/handlers/message.py", line 1567, in persist_and_notify_client_event
	    await self.request_ratelimiter.ratelimit(
	  File "/home/rei/work/synapse/synapse/api/ratelimiting.py", line 391, in ratelimit
	    tb = "".join(traceback.format_stack(limit=10))
	
	~+~
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
	    self._runCallbacks()
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 857, in _runCallbacks
	    current.result = callback(  # type: ignore[misc]
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1750, in gotResult
	    current_context.run(_inlineCallbacks, r, gen, status)
	  File "/home/rei/.cache/pypoetry/virtualenvs/matrix-synapse-0QZlT1cR-py3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
	    result = current_context.run(gen.send, result)
	  File "/home/rei/work/synapse/synapse/http/server.py", line 228, in wrapped_async_request_handler
	    await h(self, request)
	  File "/home/rei/work/synapse/synapse/http/server.py", line 306, in _async_render_wrapper
	    callback_return = await self._async_render(request)
	  File "/home/rei/work/synapse/synapse/http/server.py", line 512, in _async_render
	    callback_return = await raw_callback_return
	  File "/home/rei/work/synapse/synapse/rest/client/room.py", line 161, in on_POST
	    info, _ = await self._room_creation_handler.create_room(
	  File "/home/rei/work/synapse/synapse/handlers/room.py", line 759, in create_room
	    await self.request_ratelimiter.ratelimit(requester)
	  File "/home/rei/work/synapse/synapse/api/ratelimiting.py", line 391, in ratelimit
	    tb = "".join(traceback.format_stack(limit=10))
	
	~~~~~

It appears that there was 1 event per room being rate limited, plus the room creation itself (?)

@clokep
Copy link
Member

clokep commented Oct 27, 2022

One might argue that we shouldn't ratelimit the creation of events whilst creating a room at all [..] I'd be tempted to rate limit the room's creation but not the events that go inside

I think this likely makes sense.

@reivilibre reivilibre mentioned this issue Oct 27, 2022
Merged
@reivilibre reivilibre added the X-Release-Blocker Must be resolved before making a release label Oct 28, 2022
@clokep
Copy link
Member

clokep commented Oct 31, 2022

Fixed by #14314.

@clokep clokep closed this as completed Oct 31, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@reivilibre reivilibre

Labels
O-Occasional Affects or can be seen by some users regularly or most users rarely S-Major Major functionality / product severely impaired, no satisfactory workaround. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. X-Release-Blocker Must be resolved before making a release
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix room creation being rate limited too aggressively since Synapse v1.69.0. matrix-org/synapse
2 participants
@clokep @reivilibre