Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backup_ve…

…rsions_are_numbers
matrix-org · Nov 9, 2018 · bca3b91 · bca3b91
2 parents d44dea0 + 0f3f0a6
commit bca3b91
Show file tree

Hide file tree

Showing 140 changed files with 2,475 additions and 758 deletions.
diff --git a/.travis.yml b/.travis.yml
@@ -23,6 +23,9 @@ branches:
     - develop
     - /^release-v/
 
+# When running the tox environments that call Twisted Trial, we can pass the -j
+# flag to run the tests concurrently. We set this to 2 for CPU bound tests
+# (SQLite) and 4 for I/O bound tests (PostgreSQL).
 matrix:
   fast_finish: true
   include:
@@ -33,21 +36,21 @@ matrix:
     env: TOX_ENV="pep8,check_isort"
 
   - python: 2.7
-    env: TOX_ENV=py27
+    env: TOX_ENV=py27 TRIAL_FLAGS="-j 2"
 
   - python: 2.7
-    env: TOX_ENV=py27-old
+    env: TOX_ENV=py27-old TRIAL_FLAGS="-j 2"
 
   - python: 2.7
     env: TOX_ENV=py27-postgres TRIAL_FLAGS="-j 4"
     services:
       - postgresql
 
   - python: 3.5
-    env: TOX_ENV=py35
+    env: TOX_ENV=py35 TRIAL_FLAGS="-j 2"
 
   - python: 3.6
-    env: TOX_ENV=py36
+    env: TOX_ENV=py36 TRIAL_FLAGS="-j 2"
 
   - python: 3.6
     env: TOX_ENV=py36-postgres TRIAL_FLAGS="-j 4"

diff --git a/CHANGES.md b/CHANGES.md
@@ -1,3 +1,66 @@
+Synapse 0.33.8 (2018-11-01)
+===========================
+
+No significant changes.
+
+
+Synapse 0.33.8rc2 (2018-10-31)
+==============================
+
+Bugfixes
+--------
+
+- Searches that request profile info now no longer fail with a 500. Fixes 
+  a regression in 0.33.8rc1. ([\#4122](https://github.com/matrix-org/synapse/issues/4122))
+
+
+Synapse 0.33.8rc1 (2018-10-29)
+==============================
+
+Features
+--------
+
+- Servers with auto-join rooms will now automatically create those rooms when the first user registers ([\#3975](https://github.com/matrix-org/synapse/issues/3975))
+- Add config option to control alias creation ([\#4051](https://github.com/matrix-org/synapse/issues/4051))
+- The register_new_matrix_user script is now ported to Python 3. ([\#4085](https://github.com/matrix-org/synapse/issues/4085))
+- Configure Docker image to listen on both ipv4 and ipv6. ([\#4089](https://github.com/matrix-org/synapse/issues/4089))
+
+
+Bugfixes
+--------
+
+- Fix HTTP error response codes for federated group requests. ([\#3969](https://github.com/matrix-org/synapse/issues/3969))
+- Fix issue where Python 3 users couldn't paginate /publicRooms ([\#4046](https://github.com/matrix-org/synapse/issues/4046))
+- Fix URL previewing to work in Python 3.7 ([\#4050](https://github.com/matrix-org/synapse/issues/4050))
+- synctl will use the right python executable to run worker processes ([\#4057](https://github.com/matrix-org/synapse/issues/4057))
+- Manhole now works again on Python 3, instead of failing with a "couldn't match all kex parts" when connecting. ([\#4060](https://github.com/matrix-org/synapse/issues/4060), [\#4067](https://github.com/matrix-org/synapse/issues/4067))
+- Fix some metrics being racy and causing exceptions when polled by Prometheus. ([\#4061](https://github.com/matrix-org/synapse/issues/4061))
+- Fix bug which prevented email notifications from being sent unless an absolute path was given for `email_templates`. ([\#4068](https://github.com/matrix-org/synapse/issues/4068))
+- Correctly account for cpu usage by background threads ([\#4074](https://github.com/matrix-org/synapse/issues/4074))
+- Fix race condition where config defined reserved users were not being added to
+  the monthly active user list prior to the homeserver reactor firing up ([\#4081](https://github.com/matrix-org/synapse/issues/4081))
+- Fix bug which prevented backslashes being used in event field filters ([\#4083](https://github.com/matrix-org/synapse/issues/4083))
+
+
+Internal Changes
+----------------
+
+- Add information about the [matrix-docker-ansible-deploy](https://github.com/spantaleev/matrix-docker-ansible-deploy) playbook ([\#3698](https://github.com/matrix-org/synapse/issues/3698))
+- Add initial implementation of new state resolution algorithm ([\#3786](https://github.com/matrix-org/synapse/issues/3786))
+- Reduce database load when fetching state groups ([\#4011](https://github.com/matrix-org/synapse/issues/4011))
+- Various cleanups in the federation client code ([\#4031](https://github.com/matrix-org/synapse/issues/4031))
+- Run the CircleCI builds in docker containers ([\#4041](https://github.com/matrix-org/synapse/issues/4041))
+- Only colourise synctl output when attached to tty ([\#4049](https://github.com/matrix-org/synapse/issues/4049))
+- Refactor room alias creation code ([\#4063](https://github.com/matrix-org/synapse/issues/4063))
+- Make the Python scripts in the top-level scripts folders meet pep8 and pass flake8. ([\#4068](https://github.com/matrix-org/synapse/issues/4068))
+- The README now contains example for the Caddy web server. Contributed by steamp0rt. ([\#4072](https://github.com/matrix-org/synapse/issues/4072))
+- Add psutil as an explicit dependency ([\#4073](https://github.com/matrix-org/synapse/issues/4073))
+- Clean up threading and logcontexts in pushers ([\#4075](https://github.com/matrix-org/synapse/issues/4075))
+- Correctly manage logcontexts during startup to fix some "Unexpected logging context" warnings ([\#4076](https://github.com/matrix-org/synapse/issues/4076))
+- Give some more things logcontexts ([\#4077](https://github.com/matrix-org/synapse/issues/4077))
+- Clean up some bits of code which were flagged by the linter ([\#4082](https://github.com/matrix-org/synapse/issues/4082))
+
+
 Synapse 0.33.7 (2018-10-18)
 ===========================
 

diff --git a/changelog.d/3698.misc b/changelog.d/3698.misc
diff --git a/changelog.d/3778.misc b/changelog.d/3778.misc
@@ -0,0 +1 @@
+Fix build of Docker image with docker-compose
diff --git a/changelog.d/3786.misc b/changelog.d/3786.misc
diff --git a/changelog.d/3969.bugfix b/changelog.d/3969.bugfix
diff --git a/changelog.d/3975.feature b/changelog.d/3975.feature
diff --git a/changelog.d/4004.feature b/changelog.d/4004.feature
@@ -0,0 +1 @@
+Include flags to optionally add `m.login.terms` to the registration flow when consent tracking is enabled.
diff --git a/changelog.d/4006.misc b/changelog.d/4006.misc
@@ -0,0 +1 @@
+Delete unreferenced state groups during history purge
diff --git a/changelog.d/4011.misc b/changelog.d/4011.misc
diff --git a/changelog.d/4031.misc b/changelog.d/4031.misc
diff --git a/changelog.d/4041.misc b/changelog.d/4041.misc
diff --git a/changelog.d/4046.bugfix b/changelog.d/4046.bugfix
diff --git a/changelog.d/4049.misc b/changelog.d/4049.misc
diff --git a/changelog.d/4050.bugfix b/changelog.d/4050.bugfix
diff --git a/changelog.d/4051.feature b/changelog.d/4051.feature
diff --git a/changelog.d/4057.bugfix b/changelog.d/4057.bugfix
diff --git a/changelog.d/4060.bugfix b/changelog.d/4060.bugfix
diff --git a/changelog.d/4061.bugfix b/changelog.d/4061.bugfix
diff --git a/changelog.d/4063.misc b/changelog.d/4063.misc
diff --git a/changelog.d/4067.bugfix b/changelog.d/4067.bugfix
diff --git a/changelog.d/4068.bugfix b/changelog.d/4068.bugfix
diff --git a/changelog.d/4068.misc b/changelog.d/4068.misc
diff --git a/changelog.d/4072.misc b/changelog.d/4072.misc
diff --git a/changelog.d/4073.misc b/changelog.d/4073.misc
diff --git a/changelog.d/4074.bugfix b/changelog.d/4074.bugfix
diff --git a/changelog.d/4075.misc b/changelog.d/4075.misc
diff --git a/changelog.d/4076.misc b/changelog.d/4076.misc
diff --git a/changelog.d/4077.misc b/changelog.d/4077.misc
diff --git a/changelog.d/4081.bugfix b/changelog.d/4081.bugfix
diff --git a/changelog.d/4082.misc b/changelog.d/4082.misc
diff --git a/changelog.d/4083.bugfix b/changelog.d/4083.bugfix
diff --git a/changelog.d/4085.feature b/changelog.d/4085.feature
diff --git a/changelog.d/4089.feature b/changelog.d/4089.feature
diff --git a/changelog.d/4095.bugfix b/changelog.d/4095.bugfix
@@ -0,0 +1 @@
+Fix exceptions when using the email mailer on Python 3.
diff --git a/changelog.d/4101.feature b/changelog.d/4101.feature
@@ -0,0 +1 @@
+Support for replacing rooms with new ones
diff --git a/changelog.d/4118.removal b/changelog.d/4118.removal
@@ -0,0 +1 @@
+The obsolete and non-functional /pull federation endpoint has been removed.
diff --git a/changelog.d/4119.removal b/changelog.d/4119.removal
@@ -0,0 +1 @@
+The deprecated v1 key exchange endpoints have been removed.
diff --git a/changelog.d/4120.removal b/changelog.d/4120.removal
@@ -0,0 +1 @@
+Synapse will no longer fetch keys using the fallback deprecated v1 key exchange method and will now always use v2.
diff --git a/changelog.d/4121.misc b/changelog.d/4121.misc
@@ -0,0 +1 @@
+Log some bits about room creation
diff --git a/changelog.d/4122.bugfix b/changelog.d/4122.bugfix
@@ -0,0 +1 @@
+Searches that request profile info now no longer fail with a 500.
diff --git a/changelog.d/4123.bugfix b/changelog.d/4123.bugfix
@@ -0,0 +1 @@
+fix return code of empty key backups
diff --git a/changelog.d/4124.misc b/changelog.d/4124.misc
@@ -0,0 +1 @@
+Fix `tox` failure on old systems
diff --git a/changelog.d/4127.bugfix b/changelog.d/4127.bugfix
@@ -0,0 +1 @@
+If the typing stream ID goes backwards (as on a worker when the master restarts), the worker's typing handler will no longer erroneously report rooms containing new typing events.
diff --git a/changelog.d/4128.misc b/changelog.d/4128.misc
@@ -0,0 +1 @@
+Add STATE_V2_TEST room version
diff --git a/changelog.d/4132.bugfix b/changelog.d/4132.bugfix
@@ -0,0 +1 @@
+Fix table lock of device_lists_remote_cache which could freeze the application
diff --git a/changelog.d/4133.feature b/changelog.d/4133.feature
@@ -0,0 +1 @@
+Include flags to optionally add `m.login.terms` to the registration flow when consent tracking is enabled.
diff --git a/changelog.d/4135.bugfix b/changelog.d/4135.bugfix
@@ -0,0 +1 @@
+Fix exception when using state res v2 algorithm
diff --git a/changelog.d/4137.misc b/changelog.d/4137.misc
@@ -0,0 +1 @@
+Clean up event accesses and tests
diff --git a/changelog.d/4138.misc b/changelog.d/4138.misc
@@ -0,0 +1 @@
+The default logging config will now set an explicit log file encoding of UTF-8.
diff --git a/changelog.d/4139.misc b/changelog.d/4139.misc
@@ -0,0 +1 @@
+Add helpers functions for getting prev and auth events of an event
diff --git a/changelog.d/4140.bugfix b/changelog.d/4140.bugfix
@@ -0,0 +1 @@
+Generating the user consent URI no longer fails on Python 3.
diff --git a/changelog.d/4142.feature b/changelog.d/4142.feature
@@ -0,0 +1 @@
+Include flags to optionally add `m.login.terms` to the registration flow when consent tracking is enabled.
diff --git a/changelog.d/4149.misc b/changelog.d/4149.misc
@@ -0,0 +1 @@
+Add some tests for the HTTP pusher.
diff --git a/changelog.d/4155.misc b/changelog.d/4155.misc
@@ -0,0 +1 @@
+add purge_history.sh and purge_remote_media.sh scripts to contrib/
diff --git a/changelog.d/4156.misc b/changelog.d/4156.misc
@@ -0,0 +1 @@
+HTTP tests have been refactored to contain less boilerplate.
diff --git a/changelog.d/4157.bugfix b/changelog.d/4157.bugfix
@@ -0,0 +1 @@
+Loading URL previews from the DB cache on Postgres will no longer cause Unicode type errors when responding to the request, and URL previews will no longer fail if the remote server returns a Content-Type header with the chartype in quotes.
diff --git a/changelog.d/4161.bugfix b/changelog.d/4161.bugfix
@@ -0,0 +1 @@
+The hash_password script now works on Python 3.
diff --git a/changelog.d/4163.bugfix b/changelog.d/4163.bugfix
@@ -0,0 +1 @@
+Generating the user consent URI no longer fails on Python 3.
diff --git a/changelog.d/4164.bugfix b/changelog.d/4164.bugfix
@@ -0,0 +1 @@
+Fix noop checks when updating device keys, reducing spurious device list update notifications.
diff --git a/contrib/docker/docker-compose.yml b/contrib/docker/docker-compose.yml
@@ -6,9 +6,11 @@ version: '3'
 services:
 
   synapse:
-    build: ../..
+    build:
+        context: ../..
+        dockerfile: docker/Dockerfile
     image: docker.io/matrixdotorg/synapse:latest
-    # Since snyapse does not retry to connect to the database, restart upon
+    # Since synapse does not retry to connect to the database, restart upon
     # failure
     restart: unless-stopped
     # See the readme for a full documentation of the environment settings
@@ -47,4 +49,4 @@ services:
       # You may store the database tables in a local folder..
       - ./schemas:/var/lib/postgresql/data
       # .. or store them on some high performance storage for better results
-      # - /path/to/ssd/storage:/var/lib/postfesql/data
+      # - /path/to/ssd/storage:/var/lib/postgresql/data
diff --git a/contrib/purge_api/README.md b/contrib/purge_api/README.md
@@ -0,0 +1,16 @@
+Purge history API examples
+==========================
+
+# `purge_history.sh`
+
+A bash file, that uses the [purge history API](/docs/admin_api/README.rst) to 
+purge all messages in a list of rooms up to a certain event. You can select a 
+timeframe or a number of messages that you want to keep in the room.
+
+Just configure the variables DOMAIN, ADMIN, ROOMS_ARRAY and TIME at the top of
+the script.
+
+# `purge_remote_media.sh`
+
+A bash file, that uses the [purge history API](/docs/admin_api/README.rst) to 
+purge all old cached remote media.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Include flags to optionally add `m.login.terms` to the registration flow when consent tracking is enabled.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Delete unreferenced state groups during history purge
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Fix exceptions when using the email mailer on Python 3.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		The obsolete and non-functional /pull federation endpoint has been removed.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		The deprecated v1 key exchange endpoints have been removed.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Synapse will no longer fetch keys using the fallback deprecated v1 key exchange method and will now always use v2.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Searches that request profile info now no longer fail with a 500.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		If the typing stream ID goes backwards (as on a worker when the master restarts), the worker's typing handler will no longer erroneously report rooms containing new typing events.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Fix table lock of device_lists_remote_cache which could freeze the application
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Fix exception when using state res v2 algorithm