Skip to content

Commit

Permalink
Merge tag 'v1.44.0'
Browse files Browse the repository at this point in the history
Synapse 1.44.0 (2021-10-05)
===========================

No significant changes since 1.44.0rc3.

Synapse 1.44.0rc3 (2021-10-04)
==============================

Bugfixes
--------

- Fix a bug introduced in Synapse v1.40.0 where changing a user's display name or avatar in a restricted room would cause an authentication error. ([\matrix-org#10933](matrix-org#10933))
- Fix `/admin/whois/{user_id}` endpoint, which was broken in v1.44.0rc1. ([\matrix-org#10968](matrix-org#10968))

Synapse 1.44.0rc2 (2021-09-30)
==============================

Bugfixes
--------

- Fix a bug introduced in v1.44.0rc1 which caused the experimental [MSC2716](matrix-org/matrix-spec-proposals#2716) `/batch_send` endpoint to return a 500 error. ([\matrix-org#10938](matrix-org#10938))
- Fix a bug introduced in v1.44.0rc1 which prevented sending presence events to application services. ([\matrix-org#10944](matrix-org#10944))

Improved Documentation
----------------------

- Minor updates to the installation instructions. ([\matrix-org#10919](matrix-org#10919))

Synapse 1.44.0rc1 (2021-09-29)
==============================

Features
--------

- Only allow the [MSC2716](matrix-org/matrix-spec-proposals#2716) `/batch_send?chunk_id=xxx` endpoint to connect to an already existing insertion event. ([\matrix-org#10776](matrix-org#10776))
- Improve oEmbed URL previews by processing the author name, photo, and video information. ([\matrix-org#10814](matrix-org#10814), [\matrix-org#10819](matrix-org#10819))
- Speed up responding with large JSON objects to requests. ([\matrix-org#10868](matrix-org#10868), [\matrix-org#10905](matrix-org#10905))
- Add a `user_may_create_room_with_invites` spam checker callback to allow modules to allow or deny a room creation request based on the invites and/or 3PID invites it includes. ([\matrix-org#10898](matrix-org#10898))

Bugfixes
--------

- Fix a long-standing bug that caused an `AssertionError` when purging history in certain rooms. Contributed by @Kokokokoka. ([\matrix-org#10690](matrix-org#10690))
- Fix a long-standing bug which caused deactivated users that were later reactivated to be missing from the user directory. ([\matrix-org#10782](matrix-org#10782))
- Fix a long-standing bug that caused unbanning a user by sending a membership event to fail. Contributed by @aaronraimist. ([\matrix-org#10807](matrix-org#10807))
- Fix a long-standing bug where logging contexts would go missing when federation requests time out. ([\matrix-org#10810](matrix-org#10810))
- Fix a long-standing bug causing an error in the deprecated `/initialSync` endpoint when using the undocumented `from` and `to` parameters. ([\matrix-org#10827](matrix-org#10827))
- Fix a bug causing the `remove_stale_pushers` background job to repeatedly fail and log errors. This bug affected Synapse servers that had been upgraded from version 1.28 or older and are using SQLite. ([\matrix-org#10843](matrix-org#10843))
- Fix a long-standing bug in Unicode support of the room search admin API breaking search for rooms with non-ASCII characters. ([\matrix-org#10859](matrix-org#10859))
- Fix a bug introduced in Synapse 1.37.0 which caused `knock` membership events which we sent to remote servers to be incorrectly stored in the local database. ([\matrix-org#10873](matrix-org#10873))
- Fix invalidating one-time key count cache after claiming keys. The bug was introduced in Synapse v1.41.0. Contributed by Tulir at Beeper. ([\matrix-org#10875](matrix-org#10875))
- Fix a long-standing bug causing application service users to be subject to MAU blocking if the MAU limit had been reached, even if configured not to be blocked. ([\matrix-org#10881](matrix-org#10881))
- Fix a long-standing bug which could cause events pulled over federation to be incorrectly rejected. ([\matrix-org#10907](matrix-org#10907))
- Fix a long-standing bug causing URL cache files to be stored in storage providers. Server admins may safely delete the `url_cache/` and `url_cache_thumbnails/` directories from any configured storage providers to reclaim space. ([\matrix-org#10911](matrix-org#10911))
- Fix a long-standing bug leading to race conditions when creating media store and config directories. ([\matrix-org#10913](matrix-org#10913))

Improved Documentation
----------------------

- Fix some crashes in the Module API example code, by adding JSON encoding/decoding. ([\matrix-org#10845](matrix-org#10845))
- Add developer documentation about experimental configuration flags. ([\matrix-org#10865](matrix-org#10865))
- Properly remove deleted files from GitHub pages when generating the documentation. ([\matrix-org#10869](matrix-org#10869))

Internal Changes
----------------

- Fix GitHub Actions config so we can run sytest on synapse from parallel branches. ([\matrix-org#10659](matrix-org#10659))
- Split out [MSC2716](matrix-org/matrix-spec-proposals#2716) meta events to their own fields in the `/batch_send` response. ([\matrix-org#10777](matrix-org#10777))
- Add missing type hints to REST servlets. ([\matrix-org#10785](matrix-org#10785), [\matrix-org#10817](matrix-org#10817))
- Simplify the internal logic which maintains the user directory database tables. ([\matrix-org#10796](matrix-org#10796))
- Use direct references to config flags. ([\matrix-org#10812](matrix-org#10812), [\matrix-org#10885](matrix-org#10885), [\matrix-org#10893](matrix-org#10893), [\matrix-org#10897](matrix-org#10897))
- Specify the type of token in generic "Invalid token" error messages. ([\matrix-org#10815](matrix-org#10815))
- Make `StateFilter` frozen so it is hashable. ([\matrix-org#10816](matrix-org#10816))
- Fix a long-standing bug where an `m.room.message` event containing a null byte would cause an internal server error. ([\matrix-org#10820](matrix-org#10820))
- Add type hints to the state database. ([\matrix-org#10823](matrix-org#10823))
- Opt out of cache expiry for `get_users_who_share_room_with_user`, to hopefully improve `/sync` performance when you
  haven't synced recently. ([\matrix-org#10826](matrix-org#10826))
- Track cache eviction rates more finely in Prometheus's monitoring. ([\matrix-org#10829](matrix-org#10829))
- Add missing type hints to `synapse.handlers`. ([\matrix-org#10831](matrix-org#10831), [\matrix-org#10856](matrix-org#10856))
- Extend the Module API to let plug-ins check whether an ID is local and to access IP + User Agent data. ([\matrix-org#10833](matrix-org#10833))
- Factor out PNG image data to a constant to be used in several tests. ([\matrix-org#10834](matrix-org#10834))
- Add a test to ensure state events sent by modules get persisted correctly. ([\matrix-org#10835](matrix-org#10835))
- Rename [MSC2716](matrix-org/matrix-spec-proposals#2716) fields and event types from `chunk` to `batch` to match the `/batch_send` endpoint. ([\matrix-org#10838](matrix-org#10838))
- Rename [MSC2716](matrix-org/matrix-spec-proposals#2716) `/batch_send` query parameter from `?prev_event` to more obvious usage with `?prev_event_id`. ([\matrix-org#10839](matrix-org#10839))
- Add type hints to `synapse.http.site`. ([\matrix-org#10867](matrix-org#10867))
- Include outlier status when we log V2 or V3 events. ([\matrix-org#10879](matrix-org#10879))
- Break down Grafana's cache expiry time series based on reason for eviction, c.f. [\matrix-org#10829](matrix-org#10829). ([\matrix-org#10880](matrix-org#10880))
- Clean up some of the federation event authentication code for clarity. ([\matrix-org#10883](matrix-org#10883), [\matrix-org#10884](matrix-org#10884), [\matrix-org#10896](matrix-org#10896), [\matrix-org#10901](matrix-org#10901))
- Allow the `.` and `~` characters when creating registration tokens as per the change to [MSC3231](matrix-org/matrix-spec-proposals#3231). ([\matrix-org#10887](matrix-org#10887))
- Clean up some unnecessary parentheses in places around the codebase. ([\matrix-org#10889](matrix-org#10889))
- Improve type hinting in the user directory code. ([\matrix-org#10891](matrix-org#10891))
- Update development testing script `test_postgresql.sh` to use a supported Python version and make re-runs quicker. ([\matrix-org#10906](matrix-org#10906))
- Document and summarize changes in schema version `61` – `64`. ([\matrix-org#10917](matrix-org#10917))
- Update release script to sign the newly created git tags. ([\matrix-org#10925](matrix-org#10925))
- Fix Debian builds due to `dh-virtualenv` no longer being able to build their docs. ([\matrix-org#10931](matrix-org#10931))
  • Loading branch information
tulir committed Oct 5, 2021
2 parents bcda5cb + b2c5e79 commit fea5128
Show file tree
Hide file tree
Showing 245 changed files with 4,039 additions and 2,225 deletions.
1 change: 0 additions & 1 deletion .github/workflows/docs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,5 @@ jobs:
uses: peaceiris/actions-gh-pages@068dc23d9710f1ba62e86896f84735d869951305 # v3.8.0
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
keep_files: true
publish_dir: ./book
destination_dir: ./${{ steps.vars.outputs.branch-version }}
1 change: 1 addition & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,7 @@ jobs:
volumes:
- ${{ github.workspace }}:/src
env:
SYTEST_BRANCH: ${{ github.head_ref }}
POSTGRES: ${{ matrix.postgres && 1}}
MULTI_POSTGRES: ${{ (matrix.postgres == 'multi-postgres') && 1}}
WORKERS: ${{ matrix.workers && 1 }}
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ __pycache__/
/.coverage*
/.mypy_cache/
/.tox
/.tox-pg-container
/build/
/coverage.*
/dist/
Expand Down
104 changes: 104 additions & 0 deletions CHANGES.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,107 @@
Synapse 1.44.0 (2021-10-05)
===========================

No significant changes since 1.44.0rc3.


Synapse 1.44.0rc3 (2021-10-04)
==============================

Bugfixes
--------

- Fix a bug introduced in Synapse v1.40.0 where changing a user's display name or avatar in a restricted room would cause an authentication error. ([\#10933](https://github.com/matrix-org/synapse/issues/10933))
- Fix `/admin/whois/{user_id}` endpoint, which was broken in v1.44.0rc1. ([\#10968](https://github.com/matrix-org/synapse/issues/10968))


Synapse 1.44.0rc2 (2021-09-30)
==============================

Bugfixes
--------

- Fix a bug introduced in v1.44.0rc1 which caused the experimental [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint to return a 500 error. ([\#10938](https://github.com/matrix-org/synapse/issues/10938))
- Fix a bug introduced in v1.44.0rc1 which prevented sending presence events to application services. ([\#10944](https://github.com/matrix-org/synapse/issues/10944))


Improved Documentation
----------------------

- Minor updates to the installation instructions. ([\#10919](https://github.com/matrix-org/synapse/issues/10919))


Synapse 1.44.0rc1 (2021-09-29)
==============================

Features
--------

- Only allow the [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send?chunk_id=xxx` endpoint to connect to an already existing insertion event. ([\#10776](https://github.com/matrix-org/synapse/issues/10776))
- Improve oEmbed URL previews by processing the author name, photo, and video information. ([\#10814](https://github.com/matrix-org/synapse/issues/10814), [\#10819](https://github.com/matrix-org/synapse/issues/10819))
- Speed up responding with large JSON objects to requests. ([\#10868](https://github.com/matrix-org/synapse/issues/10868), [\#10905](https://github.com/matrix-org/synapse/issues/10905))
- Add a `user_may_create_room_with_invites` spam checker callback to allow modules to allow or deny a room creation request based on the invites and/or 3PID invites it includes. ([\#10898](https://github.com/matrix-org/synapse/issues/10898))


Bugfixes
--------

- Fix a long-standing bug that caused an `AssertionError` when purging history in certain rooms. Contributed by @Kokokokoka. ([\#10690](https://github.com/matrix-org/synapse/issues/10690))
- Fix a long-standing bug which caused deactivated users that were later reactivated to be missing from the user directory. ([\#10782](https://github.com/matrix-org/synapse/issues/10782))
- Fix a long-standing bug that caused unbanning a user by sending a membership event to fail. Contributed by @aaronraimist. ([\#10807](https://github.com/matrix-org/synapse/issues/10807))
- Fix a long-standing bug where logging contexts would go missing when federation requests time out. ([\#10810](https://github.com/matrix-org/synapse/issues/10810))
- Fix a long-standing bug causing an error in the deprecated `/initialSync` endpoint when using the undocumented `from` and `to` parameters. ([\#10827](https://github.com/matrix-org/synapse/issues/10827))
- Fix a bug causing the `remove_stale_pushers` background job to repeatedly fail and log errors. This bug affected Synapse servers that had been upgraded from version 1.28 or older and are using SQLite. ([\#10843](https://github.com/matrix-org/synapse/issues/10843))
- Fix a long-standing bug in Unicode support of the room search admin API breaking search for rooms with non-ASCII characters. ([\#10859](https://github.com/matrix-org/synapse/issues/10859))
- Fix a bug introduced in Synapse 1.37.0 which caused `knock` membership events which we sent to remote servers to be incorrectly stored in the local database. ([\#10873](https://github.com/matrix-org/synapse/issues/10873))
- Fix invalidating one-time key count cache after claiming keys. The bug was introduced in Synapse v1.41.0. Contributed by Tulir at Beeper. ([\#10875](https://github.com/matrix-org/synapse/issues/10875))
- Fix a long-standing bug causing application service users to be subject to MAU blocking if the MAU limit had been reached, even if configured not to be blocked. ([\#10881](https://github.com/matrix-org/synapse/issues/10881))
- Fix a long-standing bug which could cause events pulled over federation to be incorrectly rejected. ([\#10907](https://github.com/matrix-org/synapse/issues/10907))
- Fix a long-standing bug causing URL cache files to be stored in storage providers. Server admins may safely delete the `url_cache/` and `url_cache_thumbnails/` directories from any configured storage providers to reclaim space. ([\#10911](https://github.com/matrix-org/synapse/issues/10911))
- Fix a long-standing bug leading to race conditions when creating media store and config directories. ([\#10913](https://github.com/matrix-org/synapse/issues/10913))


Improved Documentation
----------------------

- Fix some crashes in the Module API example code, by adding JSON encoding/decoding. ([\#10845](https://github.com/matrix-org/synapse/issues/10845))
- Add developer documentation about experimental configuration flags. ([\#10865](https://github.com/matrix-org/synapse/issues/10865))
- Properly remove deleted files from GitHub pages when generating the documentation. ([\#10869](https://github.com/matrix-org/synapse/issues/10869))


Internal Changes
----------------

- Fix GitHub Actions config so we can run sytest on synapse from parallel branches. ([\#10659](https://github.com/matrix-org/synapse/issues/10659))
- Split out [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) meta events to their own fields in the `/batch_send` response. ([\#10777](https://github.com/matrix-org/synapse/issues/10777))
- Add missing type hints to REST servlets. ([\#10785](https://github.com/matrix-org/synapse/issues/10785), [\#10817](https://github.com/matrix-org/synapse/issues/10817))
- Simplify the internal logic which maintains the user directory database tables. ([\#10796](https://github.com/matrix-org/synapse/issues/10796))
- Use direct references to config flags. ([\#10812](https://github.com/matrix-org/synapse/issues/10812), [\#10885](https://github.com/matrix-org/synapse/issues/10885), [\#10893](https://github.com/matrix-org/synapse/issues/10893), [\#10897](https://github.com/matrix-org/synapse/issues/10897))
- Specify the type of token in generic "Invalid token" error messages. ([\#10815](https://github.com/matrix-org/synapse/issues/10815))
- Make `StateFilter` frozen so it is hashable. ([\#10816](https://github.com/matrix-org/synapse/issues/10816))
- Fix a long-standing bug where an `m.room.message` event containing a null byte would cause an internal server error. ([\#10820](https://github.com/matrix-org/synapse/issues/10820))
- Add type hints to the state database. ([\#10823](https://github.com/matrix-org/synapse/issues/10823))
- Opt out of cache expiry for `get_users_who_share_room_with_user`, to hopefully improve `/sync` performance when you
haven't synced recently. ([\#10826](https://github.com/matrix-org/synapse/issues/10826))
- Track cache eviction rates more finely in Prometheus's monitoring. ([\#10829](https://github.com/matrix-org/synapse/issues/10829))
- Add missing type hints to `synapse.handlers`. ([\#10831](https://github.com/matrix-org/synapse/issues/10831), [\#10856](https://github.com/matrix-org/synapse/issues/10856))
- Extend the Module API to let plug-ins check whether an ID is local and to access IP + User Agent data. ([\#10833](https://github.com/matrix-org/synapse/issues/10833))
- Factor out PNG image data to a constant to be used in several tests. ([\#10834](https://github.com/matrix-org/synapse/issues/10834))
- Add a test to ensure state events sent by modules get persisted correctly. ([\#10835](https://github.com/matrix-org/synapse/issues/10835))
- Rename [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) fields and event types from `chunk` to `batch` to match the `/batch_send` endpoint. ([\#10838](https://github.com/matrix-org/synapse/issues/10838))
- Rename [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` query parameter from `?prev_event` to more obvious usage with `?prev_event_id`. ([\#10839](https://github.com/matrix-org/synapse/issues/10839))
- Add type hints to `synapse.http.site`. ([\#10867](https://github.com/matrix-org/synapse/issues/10867))
- Include outlier status when we log V2 or V3 events. ([\#10879](https://github.com/matrix-org/synapse/issues/10879))
- Break down Grafana's cache expiry time series based on reason for eviction, c.f. [\#10829](https://github.com/matrix-org/synapse/issues/10829). ([\#10880](https://github.com/matrix-org/synapse/issues/10880))
- Clean up some of the federation event authentication code for clarity. ([\#10883](https://github.com/matrix-org/synapse/issues/10883), [\#10884](https://github.com/matrix-org/synapse/issues/10884), [\#10896](https://github.com/matrix-org/synapse/issues/10896), [\#10901](https://github.com/matrix-org/synapse/issues/10901))
- Allow the `.` and `~` characters when creating registration tokens as per the change to [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231). ([\#10887](https://github.com/matrix-org/synapse/issues/10887))
- Clean up some unnecessary parentheses in places around the codebase. ([\#10889](https://github.com/matrix-org/synapse/issues/10889))
- Improve type hinting in the user directory code. ([\#10891](https://github.com/matrix-org/synapse/issues/10891))
- Update development testing script `test_postgresql.sh` to use a supported Python version and make re-runs quicker. ([\#10906](https://github.com/matrix-org/synapse/issues/10906))
- Document and summarize changes in schema version `61` – `64`. ([\#10917](https://github.com/matrix-org/synapse/issues/10917))
- Update release script to sign the newly created git tags. ([\#10925](https://github.com/matrix-org/synapse/issues/10925))
- Fix Debian builds due to `dh-virtualenv` no longer being able to build their docs. ([\#10931](https://github.com/matrix-org/synapse/issues/10931))


Synapse 1.43.0 (2021-09-21)
===========================

Expand Down
2 changes: 1 addition & 1 deletion README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -288,7 +288,7 @@ Quick start

Before setting up a development environment for synapse, make sure you have the
system dependencies (such as the python header files) installed - see
`Installing from source <https://matrix-org.github.io/synapse/latest/setup/installation.html#installing-from-source>`_.
`Platform-specific prerequisites <https://matrix-org.github.io/synapse/latest/setup/installation.html#platform-specific-prerequisites>`_.

To check out a synapse for development, clone the git repo into a working
directory of your choice::
Expand Down
4 changes: 2 additions & 2 deletions contrib/grafana/synapse.json
Original file line number Diff line number Diff line change
Expand Up @@ -6785,7 +6785,7 @@
"expr": "rate(synapse_util_caches_cache:evicted_size{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}[$bucket_size])",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "{{name}} {{job}}-{{index}}",
"legendFormat": "{{name}} ({{reason}}) {{job}}-{{index}}",
"refId": "A"
}
],
Expand Down Expand Up @@ -10888,5 +10888,5 @@
"timezone": "",
"title": "Synapse",
"uid": "000000012",
"version": 99
"version": 100
}
24 changes: 24 additions & 0 deletions debian/changelog
Original file line number Diff line number Diff line change
@@ -1,3 +1,27 @@
matrix-synapse-py3 (1.44.0) stable; urgency=medium

* New synapse release 1.44.0.

-- Synapse Packaging team <[email protected]> Tue, 05 Oct 2021 13:43:57 +0100

matrix-synapse-py3 (1.44.0~rc3) stable; urgency=medium

* New synapse release 1.44.0~rc3.

-- Synapse Packaging team <[email protected]> Mon, 04 Oct 2021 14:57:22 +0100

matrix-synapse-py3 (1.44.0~rc2) stable; urgency=medium

* New synapse release 1.44.0~rc2.

-- Synapse Packaging team <[email protected]> Thu, 30 Sep 2021 12:39:10 +0100

matrix-synapse-py3 (1.44.0~rc1) stable; urgency=medium

* New synapse release 1.44.0~rc1.

-- Synapse Packaging team <[email protected]> Tue, 28 Sep 2021 13:41:28 +0100

matrix-synapse-py3 (1.43.0) stable; urgency=medium

* New synapse release 1.43.0.
Expand Down
5 changes: 3 additions & 2 deletions docker/Dockerfile-dhvirtualenv
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,9 @@ RUN apt-get update -qq -o Acquire::Languages=none \
&& cd /dh-virtualenv \
&& env DEBIAN_FRONTEND=noninteractive mk-build-deps -ri -t "apt-get -y --no-install-recommends"

# build it
RUN cd /dh-virtualenv && dpkg-buildpackage -us -uc -b
# Build it. Note that building the docs doesn't work due to differences in
# Sphinx APIs across versions/distros.
RUN cd /dh-virtualenv && DEB_BUILD_OPTIONS=nodoc dpkg-buildpackage -us -uc -b

###
### Stage 1
Expand Down
24 changes: 21 additions & 3 deletions docker/Dockerfile-pgtests
Original file line number Diff line number Diff line change
@@ -1,12 +1,30 @@
# Use the Sytest image that comes with a lot of the build dependencies
# pre-installed
FROM matrixdotorg/sytest:latest
FROM matrixdotorg/sytest:bionic

# The Sytest image doesn't come with python, so install that
RUN apt-get update && apt-get -qq install -y python3 python3-dev python3-pip

# We need tox to run the tests in run_pg_tests.sh
RUN python3 -m pip install tox

ADD run_pg_tests.sh /pg_tests.sh
ENTRYPOINT /pg_tests.sh
# Initialise the db
RUN su -c '/usr/lib/postgresql/10/bin/initdb -D /var/lib/postgresql/data -E "UTF-8" --lc-collate="C.UTF-8" --lc-ctype="C.UTF-8" --username=postgres' postgres

# Add a user with our UID and GID so that files get created on the host owned
# by us, not root.
ARG UID
ARG GID
RUN groupadd --gid $GID user
RUN useradd --uid $UID --gid $GID --groups sudo --no-create-home user

# Ensure we can start postgres by sudo-ing as the postgres user.
RUN apt-get update && apt-get -qq install -y sudo
RUN echo "user ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

ADD run_pg_tests.sh /run_pg_tests.sh
# Use the "exec form" of ENTRYPOINT (https://docs.docker.com/engine/reference/builder/#entrypoint)
# so that we can `docker run` this container and pass arguments to pg_tests.sh
ENTRYPOINT ["/run_pg_tests.sh"]

USER user
7 changes: 3 additions & 4 deletions docker/run_pg_tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,10 @@ set -e
# Set PGUSER so Synapse's tests know what user to connect to the database with
export PGUSER=postgres

# Initialise & start the database
su -c '/usr/lib/postgresql/9.6/bin/initdb -D /var/lib/postgresql/data -E "UTF-8" --lc-collate="en_US.UTF-8" --lc-ctype="en_US.UTF-8" --username=postgres' postgres
su -c '/usr/lib/postgresql/9.6/bin/pg_ctl -w -D /var/lib/postgresql/data start' postgres
# Start the database
sudo -u postgres /usr/lib/postgresql/10/bin/pg_ctl -w -D /var/lib/postgresql/data start

# Run the tests
cd /src
export TRIAL_FLAGS="-j 4"
tox --workdir=/tmp -e py35-postgres
tox --workdir=./.tox-pg-container -e py36-postgres "$@"
1 change: 1 addition & 0 deletions docs/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@
- [Testing]()
- [OpenTracing](opentracing.md)
- [Database Schemas](development/database_schema.md)
- [Experimental features](development/experimental_features.md)
- [Synapse Architecture]()
- [Log Contexts](log_contexts.md)
- [Replication](replication.md)
Expand Down
47 changes: 47 additions & 0 deletions docs/development/contributing_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,53 @@ To increase the log level for the tests, set `SYNAPSE_TEST_LOG_LEVEL`:
SYNAPSE_TEST_LOG_LEVEL=DEBUG trial tests
```

### Running tests under PostgreSQL

Invoking `trial` as above will use an in-memory SQLite database. This is great for
quick development and testing. However, we recommend using a PostgreSQL database
in production (and indeed, we have some code paths specific to each database).
This means that we need to run our unit tests against PostgreSQL too. Our CI does
this automatically for pull requests and release candidates, but it's sometimes
useful to reproduce this locally.

To do so, [configure Postgres](../postgres.md) and run `trial` with the
following environment variables matching your configuration:

- `SYNAPSE_POSTGRES` to anything nonempty
- `SYNAPSE_POSTGRES_HOST`
- `SYNAPSE_POSTGRES_USER`
- `SYNAPSE_POSTGRES_PASSWORD`

For example:

```shell
export SYNAPSE_POSTGRES=1
export SYNAPSE_POSTGRES_HOST=localhost
export SYNAPSE_POSTGRES_USER=postgres
export SYNAPSE_POSTGRES_PASSWORD=mydevenvpassword
trial
```

#### Prebuilt container

Since configuring PostgreSQL can be fiddly, we can make use of a pre-made
Docker container to set up PostgreSQL and run our tests for us. To do so, run

```shell
scripts-dev/test_postgresql.sh
```

Any extra arguments to the script will be passed to `tox` and then to `trial`,
so we can run a specific test in this container with e.g.

```shell
scripts-dev/test_postgresql.sh tests.replication.test_sharded_event_persister.EventPersisterShardTestCase
```

The container creates a folder in your Synapse checkout called
`.tox-pg-container` and uses this as a tox environment. The output of any
`trial` runs goes into `_trial_temp` in your synapse source directory — the same
as running `trial` directly on your host machine.

## Run the integration tests ([Sytest](https://github.com/matrix-org/sytest)).

Expand Down
37 changes: 37 additions & 0 deletions docs/development/experimental_features.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Implementing experimental features in Synapse

It can be desirable to implement "experimental" features which are disabled by
default and must be explicitly enabled via the Synapse configuration. This is
applicable for features which:

* Are unstable in the Matrix spec (e.g. those defined by an MSC that has not yet been merged).
* Developers are not confident in their use by general Synapse administrators/users
(e.g. a feature is incomplete, buggy, performs poorly, or needs further testing).

Note that this only really applies to features which are expected to be desirable
to a broad audience. The [module infrastructure](../modules/index.md) should
instead be investigated for non-standard features.

Guarding experimental features behind configuration flags should help with some
of the following scenarios:

* Ensure that clients do not assume that unstable features exist (failing
gracefully if they do not).
* Unstable features do not become de-facto standards and can be removed
aggressively (since only those who have opted-in will be affected).
* Ease finding the implementation of unstable features in Synapse (for future
removal or stabilization).
* Ease testing a feature (or removal of feature) due to enabling/disabling without
code changes. It also becomes possible to ask for wider testing, if desired.

Experimental configuration flags should be disabled by default (requiring Synapse
administrators to explicitly opt-in), although there are situations where it makes
sense (from a product point-of-view) to enable features by default. This is
expected and not an issue.

It is not a requirement for experimental features to be behind a configuration flag,
but one should be used if unsure.

New experimental configuration flags should be added under the `experimental`
configuration key (see the `synapse.config.experimental` file) and either explain
(briefly) what is being enabled, or include the MSC number.
Loading

0 comments on commit fea5128

Please sign in to comment.