Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Allow setting transaction limit for db connections #10440

Merged
merged 5 commits into from
Aug 2, 2021

Conversation

hifi
Copy link
Contributor

@hifi hifi commented Jul 21, 2021

Setting the value will help PostgreSQL free up memory by recycling the connections in the connection pool.

I would not set a default value for now but allow testing this as an experimental feature.

Possibly fixes #9182

Pull Request Checklist

  • Pull request is based on the develop branch
  • Pull request includes a changelog file. The entry should:
    • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
    • Use markdown where necessary, mostly for code blocks.
    • End with either a period (.) or an exclamation mark (!).
    • Start with a capital letter.
  • Pull request includes a sign off
  • Code style is correct (run the linters)

Setting the value will help PostgreSQL free up memory by recycling
the connections in the connection pool.

Signed-off-by: Toni Spets <[email protected]>
@hifi hifi force-pushed the feature/txn_limit branch from 429c86a to 4b1300b Compare July 21, 2021 07:15
@ShadowJonathan
Copy link
Contributor

Copy-paste from some discussion in #-dev:

As far as I see it, this’ll add a bit of overhead with some transactions as the database driver needs to find a new connection, the better way to do it would be to keep a connection “reserve” at all times (newly created, and untouched), and switch to it when one connection has exceeded its transaction limit (and thus has to be closed), so a new connection gets created in the background asynchronously instead of it becoming a head-of-the-line blocker for new transactions.

a potential optimization

@hifi
Copy link
Contributor Author

hifi commented Jul 21, 2021

It's quite a bit of more work to do that. I have doubts the connection time is actually meaningful enough to warrant that (when adding up all the other delays) unless it can be proven that this has more negative effects than positive given the reconnection itself is reliable.

Testing this in scale would need to be done with good metrics anyway. I only have access to a small instance.

Copy link
Member

@richvdh richvdh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems like a sensible improvement; I'd be happy to merge this after a bit of clean-up.

I'm worried that the new code doesn't have any test coverage at all, and is therefore very likely to break in the future. Maybe you could add something to tests.storage.test_database? There is some vaguely related stuff in tests.storage.test_base, though I think it mocks things out at the wrong level.

synapse/storage/database.py Outdated Show resolved Hide resolved
synapse/storage/database.py Outdated Show resolved Hide resolved
synapse/storage/database.py Outdated Show resolved Hide resolved
synapse/storage/database.py Outdated Show resolved Hide resolved
synapse/storage/database.py Outdated Show resolved Hide resolved
synapse/config/database.py Outdated Show resolved Hide resolved
@hifi hifi marked this pull request as ready for review July 22, 2021 15:24
@hifi
Copy link
Contributor Author

hifi commented Jul 22, 2021

Did all requested changes and added a changelog file.

@hifi hifi changed the title Draft: Allow setting transaction limit for db connections Allow setting transaction limit for db connections Jul 22, 2021
@erikjohnston erikjohnston requested a review from a team July 23, 2021 16:19
@richvdh richvdh dismissed their stale review July 27, 2021 20:35

fixed

@richvdh
Copy link
Member

richvdh commented Jul 27, 2021

The code here looks good, though you'll need to run scripts-dev/generate_sample_config to update the sample config file (and thus make CI happy).

I'm still a bit dubious about merging this without the new code being used in CI at all. Did you get anywhere with looking into writing some tests for this? Maybe we could at least run a couple of SELECT 1-type commands with the config option enabled and check that it doesn't blow up?

@hifi
Copy link
Contributor Author

hifi commented Jul 28, 2021

I tried a few different methods but nothing seems to go between the right stuff without mocking the whole adbapi ConnectionPool including connections. reconnect() is called on the raw Twisted Connection so there's very little I can do otherwise to track it's being actually called.

@hifi
Copy link
Contributor Author

hifi commented Jul 28, 2021

Added trivial test to see it doesn't break.

@@ -0,0 +1,36 @@
# Copyright 2014-2021 The Matrix.org Foundation C.I.C.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't ask you to assign copyright, so this is yours:

Suggested change
# Copyright 2014-2021 The Matrix.org Foundation C.I.C.
# Copyright 2021 Toni Spets

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's fine 😄

Copy link
Member

@richvdh richvdh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks very much for your work on this!

@richvdh richvdh enabled auto-merge (squash) August 2, 2021 13:23
@richvdh richvdh merged commit ba5287f into matrix-org:develop Aug 2, 2021
reivilibre added a commit that referenced this pull request Aug 2, 2021
reivilibre added a commit that referenced this pull request Aug 2, 2021
aaronraimist added a commit to aaronraimist/synapse that referenced this pull request Aug 13, 2021
Synapse 1.40.0 (2021-08-10)
===========================

No significant changes.

Synapse 1.40.0rc3 (2021-08-09)
==============================

Features
--------

- Support [MSC3289: room version 8](matrix-org/matrix-spec-proposals#3289). ([\matrix-org#10449](matrix-org#10449))

Bugfixes
--------

- Mark the experimental room version from [MSC2716](matrix-org/matrix-spec-proposals#2716) as unstable. ([\matrix-org#10449](matrix-org#10449))

Improved Documentation
----------------------

- Fix broken links in `upgrade.md`. Contributed by @dklimpel. ([\matrix-org#10543](matrix-org#10543))

Synapse 1.40.0rc2 (2021-08-04)
==============================

Bugfixes
--------

- Fix the `PeriodicallyFlushingMemoryHandler` inhibiting application shutdown because of its background thread. ([\matrix-org#10517](matrix-org#10517))
- Fix a bug introduced in Synapse v1.40.0rc1 that could cause Synapse to respond with an error when clients would update read receipts. ([\matrix-org#10531](matrix-org#10531))

Internal Changes
----------------

- Fix release script to open the correct URL for the release. ([\matrix-org#10516](matrix-org#10516))

Synapse 1.40.0rc1 (2021-08-03)
==============================

Features
--------

- Add support for [MSC2033](matrix-org/matrix-spec-proposals#2033): `device_id` on `/account/whoami`. ([\matrix-org#9918](matrix-org#9918))
- Update support for [MSC2716 - Incrementally importing history into existing rooms](matrix-org/matrix-spec-proposals#2716). ([\matrix-org#10245](matrix-org#10245), [\matrix-org#10432](matrix-org#10432), [\matrix-org#10463](matrix-org#10463))
- Update support for [MSC3083](matrix-org/matrix-spec-proposals#3083) to consider changes in the MSC around which servers can issue join events. ([\matrix-org#10254](matrix-org#10254), [\matrix-org#10447](matrix-org#10447), [\matrix-org#10489](matrix-org#10489))
- Initial support for [MSC3244](matrix-org/matrix-spec-proposals#3244), Room version capabilities over the /capabilities API. ([\matrix-org#10283](matrix-org#10283))
- Add a buffered logging handler which periodically flushes itself. ([\matrix-org#10407](matrix-org#10407), [\matrix-org#10515](matrix-org#10515))
- Add support for https connections to a proxy server. Contributed by @Bubu and @dklimpel. ([\matrix-org#10411](matrix-org#10411))
- Support for [MSC2285 (hidden read receipts)](matrix-org/matrix-spec-proposals#2285). Contributed by @SimonBrandner. ([\matrix-org#10413](matrix-org#10413))
- Email notifications now state whether an invitation is to a room or a space. ([\matrix-org#10426](matrix-org#10426))
- Allow setting transaction limit for database connections. ([\matrix-org#10440](matrix-org#10440), [\matrix-org#10511](matrix-org#10511))
- Add `creation_ts` to "list users" admin API. ([\matrix-org#10448](matrix-org#10448))

Bugfixes
--------

- Improve character set detection in URL previews by supporting underscores (in addition to hyphens). Contributed by @srividyut. ([\matrix-org#10410](matrix-org#10410))
- Fix events being incorrectly rejected over federation if they reference auth events that the server needed to fetch. ([\matrix-org#10439](matrix-org#10439))
- Fix `synapse_federation_server_oldest_inbound_pdu_in_staging` Prometheus metric to not report a max age of 51 years when the queue is empty. ([\matrix-org#10455](matrix-org#10455))
- Fix a bug which caused an explicit assignment of power-level 0 to a user to be misinterpreted in rare circumstances. ([\matrix-org#10499](matrix-org#10499))

Improved Documentation
----------------------

- Fix hierarchy of providers on the OpenID page. ([\matrix-org#10445](matrix-org#10445))
- Consolidate development documentation to `docs/development/`. ([\matrix-org#10453](matrix-org#10453))
- Add some developer docs to explain room DAG concepts like `outliers`, `state_groups`, `depth`, etc. ([\matrix-org#10464](matrix-org#10464))
- Document how to use Complement while developing a new Synapse feature. ([\matrix-org#10483](matrix-org#10483))

Internal Changes
----------------

- Prune inbound federation queues for a room if they get too large. ([\matrix-org#10390](matrix-org#10390))
- Add type hints to `synapse.federation.transport.client` module. ([\matrix-org#10408](matrix-org#10408))
- Remove shebang line from module files. ([\matrix-org#10415](matrix-org#10415))
- Drop backwards-compatibility code that was required to support Ubuntu Xenial. ([\matrix-org#10429](matrix-org#10429))
- Use a docker image cache for the prerequisites for the debian package build. ([\matrix-org#10431](matrix-org#10431))
- Improve servlet type hints. ([\matrix-org#10437](matrix-org#10437), [\matrix-org#10438](matrix-org#10438))
- Replace usage of `or_ignore` in `simple_insert` with `simple_upsert` usage, to stop spamming postgres logs with spurious ERROR messages. ([\matrix-org#10442](matrix-org#10442))
- Update the `tests-done` Github Actions status. ([\matrix-org#10444](matrix-org#10444), [\matrix-org#10512](matrix-org#10512))
- Update type annotations to work with forthcoming Twisted 21.7.0 release. ([\matrix-org#10446](matrix-org#10446), [\matrix-org#10450](matrix-org#10450))
- Cancel redundant GHA workflows when a new commit is pushed. ([\matrix-org#10451](matrix-org#10451))
- Mitigate media repo XSS attacks on IE11 via the non-standard X-Content-Security-Policy header. ([\matrix-org#10468](matrix-org#10468))
- Additional type hints in the state handler. ([\matrix-org#10482](matrix-org#10482))
- Update syntax used to run complement tests. ([\matrix-org#10488](matrix-org#10488))
- Fix up type annotations to work with Twisted 21.7. ([\matrix-org#10490](matrix-org#10490))
- Improve type annotations for `ObservableDeferred`. ([\matrix-org#10491](matrix-org#10491))
- Extend release script to also tag and create GitHub releases. ([\matrix-org#10496](matrix-org#10496))
- Fix a bug which caused production debian packages to be incorrectly marked as 'prerelease'. ([\matrix-org#10500](matrix-org#10500))
babolivier added a commit to matrix-org/synapse-dinsic that referenced this pull request Sep 1, 2021
Synapse 1.40.0 (2021-08-10)
===========================

No significant changes.

Synapse 1.40.0rc3 (2021-08-09)
==============================

Features
--------

- Support [MSC3289: room version 8](matrix-org/matrix-spec-proposals#3289). ([\#10449](matrix-org/synapse#10449))

Bugfixes
--------

- Mark the experimental room version from [MSC2716](matrix-org/matrix-spec-proposals#2716) as unstable. ([\#10449](matrix-org/synapse#10449))

Improved Documentation
----------------------

- Fix broken links in `upgrade.md`. Contributed by @dklimpel. ([\#10543](matrix-org/synapse#10543))

Synapse 1.40.0rc2 (2021-08-04)
==============================

Bugfixes
--------

- Fix the `PeriodicallyFlushingMemoryHandler` inhibiting application shutdown because of its background thread. ([\#10517](matrix-org/synapse#10517))
- Fix a bug introduced in Synapse v1.40.0rc1 that could cause Synapse to respond with an error when clients would update read receipts. ([\#10531](matrix-org/synapse#10531))

Internal Changes
----------------

- Fix release script to open the correct URL for the release. ([\#10516](matrix-org/synapse#10516))

Synapse 1.40.0rc1 (2021-08-03)
==============================

Features
--------

- Add support for [MSC2033](matrix-org/matrix-spec-proposals#2033): `device_id` on `/account/whoami`. ([\#9918](matrix-org/synapse#9918))
- Update support for [MSC2716 - Incrementally importing history into existing rooms](matrix-org/matrix-spec-proposals#2716). ([\#10245](matrix-org/synapse#10245), [\#10432](matrix-org/synapse#10432), [\#10463](matrix-org/synapse#10463))
- Update support for [MSC3083](matrix-org/matrix-spec-proposals#3083) to consider changes in the MSC around which servers can issue join events. ([\#10254](matrix-org/synapse#10254), [\#10447](matrix-org/synapse#10447), [\#10489](matrix-org/synapse#10489))
- Initial support for [MSC3244](matrix-org/matrix-spec-proposals#3244), Room version capabilities over the /capabilities API. ([\#10283](matrix-org/synapse#10283))
- Add a buffered logging handler which periodically flushes itself. ([\#10407](matrix-org/synapse#10407), [\#10515](matrix-org/synapse#10515))
- Add support for https connections to a proxy server. Contributed by @Bubu and @dklimpel. ([\#10411](matrix-org/synapse#10411))
- Support for [MSC2285 (hidden read receipts)](matrix-org/matrix-spec-proposals#2285). Contributed by @SimonBrandner. ([\#10413](matrix-org/synapse#10413))
- Email notifications now state whether an invitation is to a room or a space. ([\#10426](matrix-org/synapse#10426))
- Allow setting transaction limit for database connections. ([\#10440](matrix-org/synapse#10440), [\#10511](matrix-org/synapse#10511))
- Add `creation_ts` to "list users" admin API. ([\#10448](matrix-org/synapse#10448))

Bugfixes
--------

- Improve character set detection in URL previews by supporting underscores (in addition to hyphens). Contributed by @srividyut. ([\#10410](matrix-org/synapse#10410))
- Fix events being incorrectly rejected over federation if they reference auth events that the server needed to fetch. ([\#10439](matrix-org/synapse#10439))
- Fix `synapse_federation_server_oldest_inbound_pdu_in_staging` Prometheus metric to not report a max age of 51 years when the queue is empty. ([\#10455](matrix-org/synapse#10455))
- Fix a bug which caused an explicit assignment of power-level 0 to a user to be misinterpreted in rare circumstances. ([\#10499](matrix-org/synapse#10499))

Improved Documentation
----------------------

- Fix hierarchy of providers on the OpenID page. ([\#10445](matrix-org/synapse#10445))
- Consolidate development documentation to `docs/development/`. ([\#10453](matrix-org/synapse#10453))
- Add some developer docs to explain room DAG concepts like `outliers`, `state_groups`, `depth`, etc. ([\#10464](matrix-org/synapse#10464))
- Document how to use Complement while developing a new Synapse feature. ([\#10483](matrix-org/synapse#10483))

Internal Changes
----------------

- Prune inbound federation queues for a room if they get too large. ([\#10390](matrix-org/synapse#10390))
- Add type hints to `synapse.federation.transport.client` module. ([\#10408](matrix-org/synapse#10408))
- Remove shebang line from module files. ([\#10415](matrix-org/synapse#10415))
- Drop backwards-compatibility code that was required to support Ubuntu Xenial. ([\#10429](matrix-org/synapse#10429))
- Use a docker image cache for the prerequisites for the debian package build. ([\#10431](matrix-org/synapse#10431))
- Improve servlet type hints. ([\#10437](matrix-org/synapse#10437), [\#10438](matrix-org/synapse#10438))
- Replace usage of `or_ignore` in `simple_insert` with `simple_upsert` usage, to stop spamming postgres logs with spurious ERROR messages. ([\#10442](matrix-org/synapse#10442))
- Update the `tests-done` Github Actions status. ([\#10444](matrix-org/synapse#10444), [\#10512](matrix-org/synapse#10512))
- Update type annotations to work with forthcoming Twisted 21.7.0 release. ([\#10446](matrix-org/synapse#10446), [\#10450](matrix-org/synapse#10450))
- Cancel redundant GHA workflows when a new commit is pushed. ([\#10451](matrix-org/synapse#10451))
- Mitigate media repo XSS attacks on IE11 via the non-standard X-Content-Security-Policy header. ([\#10468](matrix-org/synapse#10468))
- Additional type hints in the state handler. ([\#10482](matrix-org/synapse#10482))
- Update syntax used to run complement tests. ([\#10488](matrix-org/synapse#10488))
- Fix up type annotations to work with Twisted 21.7. ([\#10490](matrix-org/synapse#10490))
- Improve type annotations for `ObservableDeferred`. ([\#10491](matrix-org/synapse#10491))
- Extend release script to also tag and create GitHub releases. ([\#10496](matrix-org/synapse#10496))
- Fix a bug which caused production debian packages to be incorrectly marked as 'prerelease'. ([\#10500](matrix-org/synapse#10500))
Fizzadar pushed a commit to Fizzadar/synapse that referenced this pull request Oct 26, 2021
Synapse 1.40.0 (2021-08-10)
===========================

No significant changes.

Synapse 1.40.0rc3 (2021-08-09)
==============================

Features
--------

- Support [MSC3289: room version 8](matrix-org/matrix-spec-proposals#3289). ([\matrix-org#10449](matrix-org#10449))

Bugfixes
--------

- Mark the experimental room version from [MSC2716](matrix-org/matrix-spec-proposals#2716) as unstable. ([\matrix-org#10449](matrix-org#10449))

Improved Documentation
----------------------

- Fix broken links in `upgrade.md`. Contributed by @dklimpel. ([\matrix-org#10543](matrix-org#10543))

Synapse 1.40.0rc2 (2021-08-04)
==============================

Bugfixes
--------

- Fix the `PeriodicallyFlushingMemoryHandler` inhibiting application shutdown because of its background thread. ([\matrix-org#10517](matrix-org#10517))
- Fix a bug introduced in Synapse v1.40.0rc1 that could cause Synapse to respond with an error when clients would update read receipts. ([\matrix-org#10531](matrix-org#10531))

Internal Changes
----------------

- Fix release script to open the correct URL for the release. ([\matrix-org#10516](matrix-org#10516))

Synapse 1.40.0rc1 (2021-08-03)
==============================

Features
--------

- Add support for [MSC2033](matrix-org/matrix-spec-proposals#2033): `device_id` on `/account/whoami`. ([\matrix-org#9918](matrix-org#9918))
- Update support for [MSC2716 - Incrementally importing history into existing rooms](matrix-org/matrix-spec-proposals#2716). ([\matrix-org#10245](matrix-org#10245), [\matrix-org#10432](matrix-org#10432), [\matrix-org#10463](matrix-org#10463))
- Update support for [MSC3083](matrix-org/matrix-spec-proposals#3083) to consider changes in the MSC around which servers can issue join events. ([\matrix-org#10254](matrix-org#10254), [\matrix-org#10447](matrix-org#10447), [\matrix-org#10489](matrix-org#10489))
- Initial support for [MSC3244](matrix-org/matrix-spec-proposals#3244), Room version capabilities over the /capabilities API. ([\matrix-org#10283](matrix-org#10283))
- Add a buffered logging handler which periodically flushes itself. ([\matrix-org#10407](matrix-org#10407), [\matrix-org#10515](matrix-org#10515))
- Add support for https connections to a proxy server. Contributed by @Bubu and @dklimpel. ([\matrix-org#10411](matrix-org#10411))
- Support for [MSC2285 (hidden read receipts)](matrix-org/matrix-spec-proposals#2285). Contributed by @SimonBrandner. ([\matrix-org#10413](matrix-org#10413))
- Email notifications now state whether an invitation is to a room or a space. ([\matrix-org#10426](matrix-org#10426))
- Allow setting transaction limit for database connections. ([\matrix-org#10440](matrix-org#10440), [\matrix-org#10511](matrix-org#10511))
- Add `creation_ts` to "list users" admin API. ([\matrix-org#10448](matrix-org#10448))

Bugfixes
--------

- Improve character set detection in URL previews by supporting underscores (in addition to hyphens). Contributed by @srividyut. ([\matrix-org#10410](matrix-org#10410))
- Fix events being incorrectly rejected over federation if they reference auth events that the server needed to fetch. ([\matrix-org#10439](matrix-org#10439))
- Fix `synapse_federation_server_oldest_inbound_pdu_in_staging` Prometheus metric to not report a max age of 51 years when the queue is empty. ([\matrix-org#10455](matrix-org#10455))
- Fix a bug which caused an explicit assignment of power-level 0 to a user to be misinterpreted in rare circumstances. ([\matrix-org#10499](matrix-org#10499))

Improved Documentation
----------------------

- Fix hierarchy of providers on the OpenID page. ([\matrix-org#10445](matrix-org#10445))
- Consolidate development documentation to `docs/development/`. ([\matrix-org#10453](matrix-org#10453))
- Add some developer docs to explain room DAG concepts like `outliers`, `state_groups`, `depth`, etc. ([\matrix-org#10464](matrix-org#10464))
- Document how to use Complement while developing a new Synapse feature. ([\matrix-org#10483](matrix-org#10483))

Internal Changes
----------------

- Prune inbound federation queues for a room if they get too large. ([\matrix-org#10390](matrix-org#10390))
- Add type hints to `synapse.federation.transport.client` module. ([\matrix-org#10408](matrix-org#10408))
- Remove shebang line from module files. ([\matrix-org#10415](matrix-org#10415))
- Drop backwards-compatibility code that was required to support Ubuntu Xenial. ([\matrix-org#10429](matrix-org#10429))
- Use a docker image cache for the prerequisites for the debian package build. ([\matrix-org#10431](matrix-org#10431))
- Improve servlet type hints. ([\matrix-org#10437](matrix-org#10437), [\matrix-org#10438](matrix-org#10438))
- Replace usage of `or_ignore` in `simple_insert` with `simple_upsert` usage, to stop spamming postgres logs with spurious ERROR messages. ([\matrix-org#10442](matrix-org#10442))
- Update the `tests-done` Github Actions status. ([\matrix-org#10444](matrix-org#10444), [\matrix-org#10512](matrix-org#10512))
- Update type annotations to work with forthcoming Twisted 21.7.0 release. ([\matrix-org#10446](matrix-org#10446), [\matrix-org#10450](matrix-org#10450))
- Cancel redundant GHA workflows when a new commit is pushed. ([\matrix-org#10451](matrix-org#10451))
- Mitigate media repo XSS attacks on IE11 via the non-standard X-Content-Security-Policy header. ([\matrix-org#10468](matrix-org#10468))
- Additional type hints in the state handler. ([\matrix-org#10482](matrix-org#10482))
- Update syntax used to run complement tests. ([\matrix-org#10488](matrix-org#10488))
- Fix up type annotations to work with Twisted 21.7. ([\matrix-org#10490](matrix-org#10490))
- Improve type annotations for `ObservableDeferred`. ([\matrix-org#10491](matrix-org#10491))
- Extend release script to also tag and create GitHub releases. ([\matrix-org#10496](matrix-org#10496))
- Fix a bug which caused production debian packages to be incorrectly marked as 'prerelease'. ([\matrix-org#10500](matrix-org#10500))
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Synapse makes postgres to eat all ram until OOM
3 participants