Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Refactor devices changed query to pull less from DB #5559

Merged
merged 7 commits into from
Jun 27, 2019
Merged

Refactor devices changed query to pull less from DB #5559

merged 7 commits into from
Jun 27, 2019

Conversation

erikjohnston
Copy link
Member

When a client asks for users whose devices have changed since a token we
used to pull all users from the database since the token, which could
easily be thousands of rows for old tokens.

This PR changes this to only check for changes for users the client is
actually interested in.

Fixes #5553

@erikjohnston erikjohnston changed the title Refactor get_user_ids_changed to pull less from DB Refactor devices changed query to pull less from DB Jun 26, 2019
@erikjohnston
Refactor get_user_ids_changed to pull less from DB
a2f6d31 
When a client asks for users whose devices have changed since a token we
used to pull *all* users from the database since the token, which could
easily be thousands of rows for old tokens.

This PR changes this to only check for changes for users the client is
actually interested in.

Fixes #5553
@erikjohnston
Newsfile
508c3ce
@erikjohnston erikjohnston force-pushed the erikj/refactor_changed_devices branch from b42ae5b to 508c3ce Compare June 26, 2019 11:03
@erikjohnston erikjohnston requested a review from a team June 26, 2019 11:04
richvdh
richvdh previously requested changes Jun 26, 2019
View reviewed changes
Copy link
Member

@richvdh richvdh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bit confused :/

changelog.d/5559.misc Outdated Show resolved Hide resolved
synapse/storage/devices.py Outdated Show resolved Hide resolved
synapse/handlers/device.py
users_who_share_room = yield self.store.get_users_who_share_room_with_user(
user_id
)

# Take the intersection of the users whose devices may have changed
# and those that actually still share a room with the user
possibly_joined = possibly_changed & users_who_share_room
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is the & here now redundant?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possibly. We do go and add entries in the loop, though they should all be new users

synapse/handlers/sync.py Outdated

if not changed and not newly_left_users:
defer.returnValue(DeviceLists(changed=[], left=newly_left_users))
changed = users_who_share_room & set(newly_joined_or_invited_users)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this used to be a union: is the change to & deliberate?

synapse/handlers/sync.py Show resolved Hide resolved
synapse/storage/devices.py Show resolved Hide resolved
synapse/storage/devices.py Outdated Show resolved Hide resolved
synapse/storage/devices.py Outdated Show resolved Hide resolved
@erikjohnston erikjohnston force-pushed the erikj/refactor_changed_devices branch from 988ae87 to 61627e2 Compare June 26, 2019 18:34
@codecov
Copy link

codecov bot commented Jun 26, 2019 •
edited
Loading

Codecov Report

Merging #5559 into develop will decrease coverage by 0.03%.
The diff coverage is 41.66%.

@@             Coverage Diff             @@
##           develop    #5559      +/-   ##
===========================================
- Coverage    63.19%   63.16%   -0.04%     
===========================================
  Files          328      328              
  Lines        35925    35932       +7     
  Branches      5908     5909       +1     
===========================================
- Hits         22703    22696       -7     
- Misses       11596    11610      +14     
  Partials      1626     1626

@erikjohnston
Copy link
Member Author

@richvdh I've shuffled the code a bit and added comments to try and explain what's going on in the /sync code. I'm fairly sure they were functionally equivalent to before, but there was a lot of moving the intersection up before function calls and the like. Hopefully it should maybe make a bit of sense now.

@erikjohnston erikjohnston force-pushed the erikj/refactor_changed_devices branch from 61627e2 to 82028d7 Compare June 26, 2019 18:41
@erikjohnston erikjohnston requested a review from a team June 27, 2019 14:18
richvdh
richvdh approved these changes Jun 27, 2019
View reviewed changes
Copy link
Member

@richvdh richvdh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm modulo a bit of nitpicking

synapse/storage/devices.py Outdated

for chunk in batch_iter(to_check, 100):
txn.execute(
sql % (",".join("?" for _ in chunk),), [from_key] + list(chunk)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

chunk is a tuple, so copying to an intermediate list is redundant:

Suggested change
sql % (",".join("?" for _ in chunk),), [from_key] + list(chunk)
sql % (",".join("?" for _ in chunk),), (from_key,) + chunk

synapse/handlers/sync.py Outdated
)

# TODO: Be more clever than this, i.e. remove users who we already
# share a room with?
# Step 2, check for newly joined rooms
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

errm, I think this is a different step 2 to that mentioned at line 1097. Perhaps you want 1a and 1b.

@erikjohnston erikjohnston merged commit e79ec03 into develop Jun 27, 2019
hawkowl added a commit that referenced this pull request Jul 5, 2019
@hawkowl
Merge tag 'v1.1.0' into shhs
a1de642 
Synapse 1.1.0 (2019-07-04)
==========================

As of v1.1.0, Synapse no longer supports Python 2, nor Postgres version 9.4.
See the [upgrade notes](UPGRADE.rst#upgrading-to-v110) for more details.

This release also deprecates the use of environment variables to configure the
docker image. See the [docker README](https://github.com/matrix-org/synapse/blob/release-v1.1.0/docker/README.md#legacy-dynamic-configuration-file-support)
for more details.

No changes since 1.1.0rc2.

Synapse 1.1.0rc2 (2019-07-03)
=============================

Bugfixes
--------

- Fix regression in 1.1rc1 where OPTIONS requests to the media repo would fail. ([\#5593](#5593))
- Removed the `SYNAPSE_SMTP_*` docker container environment variables. Using these environment variables prevented the docker container from starting in Synapse v1.0, even though they didn't actually allow any functionality anyway. ([\#5596](#5596))
- Fix a number of "Starting txn from sentinel context" warnings. ([\#5605](#5605))

Internal Changes
----------------

- Update github templates. ([\#5552](#5552))

Synapse 1.1.0rc1 (2019-07-02)
=============================

As of v1.1.0, Synapse no longer supports Python 2, nor Postgres version 9.4.
See the [upgrade notes](UPGRADE.rst#upgrading-to-v110) for more details.

Features
--------

- Added possibilty to disable local password authentication. Contributed by Daniel Hoffend. ([\#5092](#5092))
- Add monthly active users to phonehome stats. ([\#5252](#5252))
- Allow expired user to trigger renewal email sending manually. ([\#5363](#5363))
- Statistics on forward extremities per room are now exposed via Prometheus. ([\#5384](#5384), [\#5458](#5458), [\#5461](#5461))
- Add --no-daemonize option to run synapse in the foreground, per issue #4130. Contributed by Soham Gumaste. ([\#5412](#5412), [\#5587](#5587))
- Fully support SAML2 authentication. Contributed by [Alexander Trost](https://github.com/galexrt) - thank you! ([\#5422](#5422))
- Allow server admins to define implementations of extra rules for allowing or denying incoming events. ([\#5440](#5440), [\#5474](#5474), [\#5477](#5477))
- Add support for handling pagination APIs on client reader worker. ([\#5505](#5505), [\#5513](#5513), [\#5531](#5531))
- Improve help and cmdline option names for --generate-config options. ([\#5512](#5512))
- Allow configuration of the path used for ACME account keys. ([\#5516](#5516), [\#5521](#5521), [\#5522](#5522))
- Add --data-dir and --open-private-ports options. ([\#5524](#5524))
- Split public rooms directory auth config in two settings, in order to manage client auth independently from the federation part of it. Obsoletes the "restrict_public_rooms_to_local_users" configuration setting. If "restrict_public_rooms_to_local_users" is set in the config, Synapse will act as if both new options are enabled, i.e. require authentication through the client API and deny federation requests. ([\#5534](#5534))
- The minimum TLS version used for outgoing federation requests can now be set with `federation_client_minimum_tls_version`. ([\#5550](#5550))
- Optimise devices changed query to not pull unnecessary rows from the database, reducing database load. ([\#5559](#5559))
- Add new metrics for number of forward extremities being persisted and number of state groups involved in resolution. ([\#5476](#5476))

Bugfixes
--------

- Fix bug processing incoming events over federation if call to `/get_missing_events` fails. ([\#5042](#5042))
- Prevent more than one room upgrade happening simultaneously on the same room. ([\#5051](#5051))
- Fix a bug where running synapse_port_db would cause the account validity feature to fail because it didn't set the type of the email_sent column to boolean. ([\#5325](#5325))
- Warn about disabling email-based password resets when a reset occurs, and remove warning when someone attempts a phone-based reset. ([\#5387](#5387))
- Fix email notifications for unnamed rooms with multiple people. ([\#5388](#5388))
- Fix exceptions in federation reader worker caused by attempting to renew attestations, which should only happen on master worker. ([\#5389](#5389))
- Fix handling of failures fetching remote content to not log failures as exceptions. ([\#5390](#5390))
- Fix a bug where deactivated users could receive renewal emails if the account validity feature is on. ([\#5394](#5394))
- Fix missing invite state after exchanging 3PID invites over federaton. ([\#5464](#5464))
- Fix intermittent exceptions on Apple hardware. Also fix bug that caused database activity times to be under-reported in log lines. ([\#5498](#5498))
- Fix logging error when a tampered event is detected. ([\#5500](#5500))
- Fix bug where clients could tight loop calling `/sync` for a period. ([\#5507](#5507))
- Fix bug with `jinja2` preventing Synapse from starting. Users who had this problem should now simply need to run `pip install matrix-synapse`. ([\#5514](#5514))
- Fix a regression where homeservers on private IP addresses were incorrectly blacklisted. ([\#5523](#5523))
- Fixed m.login.jwt using unregistred user_id and added pyjwt>=1.6.4 as jwt conditional dependencies. Contributed by Pau Rodriguez-Estivill. ([\#5555](#5555), [\#5586](#5586))
- Fix a bug that would cause invited users to receive several emails for a single 3PID invite in case the inviter is rate limited. ([\#5576](#5576))

Updates to the Docker image
---------------------------
- Add ability to change Docker containers [timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) with the `TZ` variable. ([\#5383](#5383))
- Update docker image to use Python 3.7. ([\#5546](#5546))
- Deprecate the use of environment variables for configuration, and make the use of a static configuration the default. ([\#5561](#5561), [\#5562](#5562), [\#5566](#5566), [\#5567](#5567))
- Increase default log level for docker image to INFO. It can still be changed by editing the generated log.config file. ([\#5547](#5547))
- Send synapse logs to the docker logging system, by default. ([\#5565](#5565))
- Open the non-TLS port by default. ([\#5568](#5568))
- Fix failure to start under docker with SAML support enabled. ([\#5490](#5490))
- Use a sensible location for data files when generating a config file. ([\#5563](#5563))

Deprecations and Removals
-------------------------

- Python 2.7 is no longer a supported platform. Synapse now requires Python 3.5+ to run. ([\#5425](#5425))
- PostgreSQL 9.4 is no longer supported. Synapse requires Postgres 9.5+ or above for Postgres support. ([\#5448](#5448))
- Remove support for cpu_affinity setting. ([\#5525](#5525))

Improved Documentation
----------------------
- Improve README section on performance troubleshooting. ([\#4276](#4276))
- Add information about how to install and run `black` on the codebase to code_style.rst. ([\#5537](#5537))
- Improve install docs on choosing server_name. ([\#5558](#5558))

Internal Changes
----------------

- Add logging to 3pid invite signature verification. ([\#5015](#5015))
- Update example haproxy config to a more compatible setup. ([\#5313](#5313))
- Track deactivated accounts in the database. ([\#5378](#5378), [\#5465](#5465), [\#5493](#5493))
- Clean up code for sending federation EDUs. ([\#5381](#5381))
- Add a sponsor button to the repo. ([\#5382](#5382), [\#5386](#5386))
- Don't log non-200 responses from federation queries as exceptions. ([\#5383](#5383))
- Update Python syntax in contrib/ to Python 3. ([\#5446](#5446))
- Update federation_client dev script to support `.well-known` and work with python3. ([\#5447](#5447))
- SyTest has been moved to Buildkite. ([\#5459](#5459))
- Demo script now uses python3. ([\#5460](#5460))
- Synapse can now handle RestServlets that return coroutines. ([\#5475](#5475), [\#5585](#5585))
- The demo servers talk to each other again. ([\#5478](#5478))
- Add an EXPERIMENTAL config option to try and periodically clean up extremities by sending dummy events. ([\#5480](#5480))
- Synapse's codebase is now formatted by `black`. ([\#5482](#5482))
- Some cleanups and sanity-checking in the CPU and database metrics. ([\#5499](#5499))
- Improve email notification logging. ([\#5502](#5502))
- Fix "Unexpected entry in 'full_schemas'" log warning. ([\#5509](#5509))
- Improve logging when generating config files. ([\#5510](#5510))
- Refactor and clean up Config parser for maintainability. ([\#5511](#5511))
- Make the config clearer in that email.template_dir is relative to the Synapse's root directory, not the `synapse/` folder within it. ([\#5543](#5543))
- Update v1.0.0 release changelog to include more information about changes to password resets. ([\#5545](#5545))
- Remove non-functioning check_event_hash.py dev script. ([\#5548](#5548))
- Synapse will now only allow TLS v1.2 connections when serving federation, if it terminates TLS. As Synapse's allowed ciphers were only able to be used in TLSv1.2 before, this does not change behaviour. ([\#5550](#5550))
- Logging when running GC collection on generation 0 is now at the DEBUG level, not INFO. ([\#5557](#5557))
- Reduce the amount of stuff we send in the docker context. ([\#5564](#5564))
- Point the reverse links in the Purge History contrib scripts at the intended location. ([\#5570](#5570))
hawkowl added a commit that referenced this pull request Jul 29, 2019
@hawkowl
Squashed commit of the following:
0bfd099 
commit c2ca163
Author: Amber H. Brown <[email protected]>
Date:   Fri Jul 26 22:48:20 2019 +1000

    don't block

commit 8d9a56e
Merge: b50d8a9 4a5fb54
Author: Amber H. Brown <[email protected]>
Date:   Fri Jul 26 21:35:42 2019 +1000

    Merge branch 'shhs' of ssh://github.com/matrix-org/synapse into shhs

commit 4a5fb54
Merge: 95a0386 992333b
Author: Amber Brown <[email protected]>
Date:   Fri Jul 26 20:59:41 2019 +1000

    Merge tag 'v1.2.1' into shhs

    Synapse 1.2.1 (2019-07-26)
    ==========================

    Security update
    ---------------

    This release includes *four* security fixes:

    - Prevent an attack where a federated server could send redactions for arbitrary events in v1 and v2 rooms. ([\#5767](#5767))
    - Prevent a denial-of-service attack where cycles of redaction events would make Synapse spin infinitely. Thanks to `@lrizika:matrix.org` for identifying and responsibly disclosing this issue. ([0f2ecb9](0f2ecb961))
    - Prevent an attack where users could be joined or parted from public rooms without their consent. Thanks to @dylanger for identifying and responsibly disclosing this issue. ([\#5744](#5744))
    - Fix a vulnerability where a federated server could spoof read-receipts from
      users on other servers. Thanks to @dylanger for identifying this issue too. ([\#5743](#5743))

    Additionally, the following fix was in Synapse **1.2.0**, but was not correctly
    identified during the original release:

    - It was possible for a room moderator to send a redaction for an `m.room.create` event, which would downgrade the room to version 1. Thanks to `/dev/ponies` for identifying and responsibly disclosing this issue! ([\#5701](#5701))

commit 95a0386
Author: Amber Brown <[email protected]>
Date:   Fri Jul 26 20:27:31 2019 +1000

    don't have a circleci config

commit b50d8a9
Author: Amber H. Brown <[email protected]>
Date:   Fri Jul 26 02:26:23 2019 +1000

    fix merging forward

commit 3edf6e9
Author: Amber H. Brown <[email protected]>
Date:   Fri Jul 26 02:07:05 2019 +1000

    fix this

commit f61cdc1
Merge: 43cf234 c0a1301
Author: Amber H. Brown <[email protected]>
Date:   Fri Jul 26 01:48:50 2019 +1000

    Merge tag 'v1.2.0' into shhs

    No changes since v1.2.0rc2.

commit 43cf234
Author: Amber H. Brown <[email protected]>
Date:   Fri Jul 26 01:48:20 2019 +1000

    dockerfile update

commit b7962f5
Author: Amber H. Brown <[email protected]>
Date:   Thu Jul 18 23:23:12 2019 +1000

    add a wait

commit 9bbf2d2
Author: Amber H. Brown <[email protected]>
Date:   Wed Jul 17 04:46:20 2019 +1000

    fix

commit 5daee2e
Author: Amber H. Brown <[email protected]>
Date:   Wed Jul 17 04:41:00 2019 +1000

    fix

commit 14c8b03
Author: Amber H. Brown <[email protected]>
Date:   Wed Jul 17 04:36:27 2019 +1000

    fix

commit 7fcd6c1
Author: Amber H. Brown <[email protected]>
Date:   Wed Jul 17 04:32:50 2019 +1000

    fix

commit c43c1ad
Author: Amber H. Brown <[email protected]>
Date:   Wed Jul 17 04:28:11 2019 +1000

    fix

commit a025abe
Author: Amber H. Brown <[email protected]>
Date:   Wed Jul 17 04:02:15 2019 +1000

    try now

commit c1777f5
Author: Amber H. Brown <[email protected]>
Date:   Wed Jul 17 04:00:34 2019 +1000

    try now

commit 646292c
Author: Amber H. Brown <[email protected]>
Date:   Wed Jul 17 03:58:34 2019 +1000

    see if we can do a build!

commit a175e60
Merge: 9b3a63e 0e54342
Author: Amber H. Brown <[email protected]>
Date:   Fri Jul 5 23:49:13 2019 +1000

    Merge remote-tracking branch 'origin/develop' into shhs

commit 9b3a63e
Author: Amber H. Brown <[email protected]>
Date:   Fri Jul 5 23:36:41 2019 +1000

    linting

commit 3d89feb
Author: Amber H. Brown <[email protected]>
Date:   Fri Jul 5 23:34:24 2019 +1000

    linting

commit 400bc06
Author: Amber H. Brown <[email protected]>
Date:   Fri Jul 5 22:44:22 2019 +1000

    linting

commit a1de642
Merge: f4343c7 54283f3
Author: Amber H. Brown <[email protected]>
Date:   Fri Jul 5 19:46:11 2019 +1000

    Merge tag 'v1.1.0' into shhs

    Synapse 1.1.0 (2019-07-04)
    ==========================

    As of v1.1.0, Synapse no longer supports Python 2, nor Postgres version 9.4.
    See the [upgrade notes](UPGRADE.rst#upgrading-to-v110) for more details.

    This release also deprecates the use of environment variables to configure the
    docker image. See the [docker README](https://github.com/matrix-org/synapse/blob/release-v1.1.0/docker/README.md#legacy-dynamic-configuration-file-support)
    for more details.

    No changes since 1.1.0rc2.

    Synapse 1.1.0rc2 (2019-07-03)
    =============================

    Bugfixes
    --------

    - Fix regression in 1.1rc1 where OPTIONS requests to the media repo would fail. ([\#5593](#5593))
    - Removed the `SYNAPSE_SMTP_*` docker container environment variables. Using these environment variables prevented the docker container from starting in Synapse v1.0, even though they didn't actually allow any functionality anyway. ([\#5596](#5596))
    - Fix a number of "Starting txn from sentinel context" warnings. ([\#5605](#5605))

    Internal Changes
    ----------------

    - Update github templates. ([\#5552](#5552))

    Synapse 1.1.0rc1 (2019-07-02)
    =============================

    As of v1.1.0, Synapse no longer supports Python 2, nor Postgres version 9.4.
    See the [upgrade notes](UPGRADE.rst#upgrading-to-v110) for more details.

    Features
    --------

    - Added possibilty to disable local password authentication. Contributed by Daniel Hoffend. ([\#5092](#5092))
    - Add monthly active users to phonehome stats. ([\#5252](#5252))
    - Allow expired user to trigger renewal email sending manually. ([\#5363](#5363))
    - Statistics on forward extremities per room are now exposed via Prometheus. ([\#5384](#5384), [\#5458](#5458), [\#5461](#5461))
    - Add --no-daemonize option to run synapse in the foreground, per issue #4130. Contributed by Soham Gumaste. ([\#5412](#5412), [\#5587](#5587))
    - Fully support SAML2 authentication. Contributed by [Alexander Trost](https://github.com/galexrt) - thank you! ([\#5422](#5422))
    - Allow server admins to define implementations of extra rules for allowing or denying incoming events. ([\#5440](#5440), [\#5474](#5474), [\#5477](#5477))
    - Add support for handling pagination APIs on client reader worker. ([\#5505](#5505), [\#5513](#5513), [\#5531](#5531))
    - Improve help and cmdline option names for --generate-config options. ([\#5512](#5512))
    - Allow configuration of the path used for ACME account keys. ([\#5516](#5516), [\#5521](#5521), [\#5522](#5522))
    - Add --data-dir and --open-private-ports options. ([\#5524](#5524))
    - Split public rooms directory auth config in two settings, in order to manage client auth independently from the federation part of it. Obsoletes the "restrict_public_rooms_to_local_users" configuration setting. If "restrict_public_rooms_to_local_users" is set in the config, Synapse will act as if both new options are enabled, i.e. require authentication through the client API and deny federation requests. ([\#5534](#5534))
    - The minimum TLS version used for outgoing federation requests can now be set with `federation_client_minimum_tls_version`. ([\#5550](#5550))
    - Optimise devices changed query to not pull unnecessary rows from the database, reducing database load. ([\#5559](#5559))
    - Add new metrics for number of forward extremities being persisted and number of state groups involved in resolution. ([\#5476](#5476))

    Bugfixes
    --------

    - Fix bug processing incoming events over federation if call to `/get_missing_events` fails. ([\#5042](#5042))
    - Prevent more than one room upgrade happening simultaneously on the same room. ([\#5051](#5051))
    - Fix a bug where running synapse_port_db would cause the account validity feature to fail because it didn't set the type of the email_sent column to boolean. ([\#5325](#5325))
    - Warn about disabling email-based password resets when a reset occurs, and remove warning when someone attempts a phone-based reset. ([\#5387](#5387))
    - Fix email notifications for unnamed rooms with multiple people. ([\#5388](#5388))
    - Fix exceptions in federation reader worker caused by attempting to renew attestations, which should only happen on master worker. ([\#5389](#5389))
    - Fix handling of failures fetching remote content to not log failures as exceptions. ([\#5390](#5390))
    - Fix a bug where deactivated users could receive renewal emails if the account validity feature is on. ([\#5394](#5394))
    - Fix missing invite state after exchanging 3PID invites over federaton. ([\#5464](#5464))
    - Fix intermittent exceptions on Apple hardware. Also fix bug that caused database activity times to be under-reported in log lines. ([\#5498](#5498))
    - Fix logging error when a tampered event is detected. ([\#5500](#5500))
    - Fix bug where clients could tight loop calling `/sync` for a period. ([\#5507](#5507))
    - Fix bug with `jinja2` preventing Synapse from starting. Users who had this problem should now simply need to run `pip install matrix-synapse`. ([\#5514](#5514))
    - Fix a regression where homeservers on private IP addresses were incorrectly blacklisted. ([\#5523](#5523))
    - Fixed m.login.jwt using unregistred user_id and added pyjwt>=1.6.4 as jwt conditional dependencies. Contributed by Pau Rodriguez-Estivill. ([\#5555](#5555), [\#5586](#5586))
    - Fix a bug that would cause invited users to receive several emails for a single 3PID invite in case the inviter is rate limited. ([\#5576](#5576))

    Updates to the Docker image
    ---------------------------
    - Add ability to change Docker containers [timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) with the `TZ` variable. ([\#5383](#5383))
    - Update docker image to use Python 3.7. ([\#5546](#5546))
    - Deprecate the use of environment variables for configuration, and make the use of a static configuration the default. ([\#5561](#5561), [\#5562](#5562), [\#5566](#5566), [\#5567](#5567))
    - Increase default log level for docker image to INFO. It can still be changed by editing the generated log.config file. ([\#5547](#5547))
    - Send synapse logs to the docker logging system, by default. ([\#5565](#5565))
    - Open the non-TLS port by default. ([\#5568](#5568))
    - Fix failure to start under docker with SAML support enabled. ([\#5490](#5490))
    - Use a sensible location for data files when generating a config file. ([\#5563](#5563))

    Deprecations and Removals
    -------------------------

    - Python 2.7 is no longer a supported platform. Synapse now requires Python 3.5+ to run. ([\#5425](#5425))
    - PostgreSQL 9.4 is no longer supported. Synapse requires Postgres 9.5+ or above for Postgres support. ([\#5448](#5448))
    - Remove support for cpu_affinity setting. ([\#5525](#5525))

    Improved Documentation
    ----------------------
    - Improve README section on performance troubleshooting. ([\#4276](#4276))
    - Add information about how to install and run `black` on the codebase to code_style.rst. ([\#5537](#5537))
    - Improve install docs on choosing server_name. ([\#5558](#5558))

    Internal Changes
    ----------------

    - Add logging to 3pid invite signature verification. ([\#5015](#5015))
    - Update example haproxy config to a more compatible setup. ([\#5313](#5313))
    - Track deactivated accounts in the database. ([\#5378](#5378), [\#5465](#5465), [\#5493](#5493))
    - Clean up code for sending federation EDUs. ([\#5381](#5381))
    - Add a sponsor button to the repo. ([\#5382](#5382), [\#5386](#5386))
    - Don't log non-200 responses from federation queries as exceptions. ([\#5383](#5383))
    - Update Python syntax in contrib/ to Python 3. ([\#5446](#5446))
    - Update federation_client dev script to support `.well-known` and work with python3. ([\#5447](#5447))
    - SyTest has been moved to Buildkite. ([\#5459](#5459))
    - Demo script now uses python3. ([\#5460](#5460))
    - Synapse can now handle RestServlets that return coroutines. ([\#5475](#5475), [\#5585](#5585))
    - The demo servers talk to each other again. ([\#5478](#5478))
    - Add an EXPERIMENTAL config option to try and periodically clean up extremities by sending dummy events. ([\#5480](#5480))
    - Synapse's codebase is now formatted by `black`. ([\#5482](#5482))
    - Some cleanups and sanity-checking in the CPU and database metrics. ([\#5499](#5499))
    - Improve email notification logging. ([\#5502](#5502))
    - Fix "Unexpected entry in 'full_schemas'" log warning. ([\#5509](#5509))
    - Improve logging when generating config files. ([\#5510](#5510))
    - Refactor and clean up Config parser for maintainability. ([\#5511](#5511))
    - Make the config clearer in that email.template_dir is relative to the Synapse's root directory, not the `synapse/` folder within it. ([\#5543](#5543))
    - Update v1.0.0 release changelog to include more information about changes to password resets. ([\#5545](#5545))
    - Remove non-functioning check_event_hash.py dev script. ([\#5548](#5548))
    - Synapse will now only allow TLS v1.2 connections when serving federation, if it terminates TLS. As Synapse's allowed ciphers were only able to be used in TLSv1.2 before, this does not change behaviour. ([\#5550](#5550))
    - Logging when running GC collection on generation 0 is now at the DEBUG level, not INFO. ([\#5557](#5557))
    - Reduce the amount of stuff we send in the docker context. ([\#5564](#5564))
    - Point the reverse links in the Purge History contrib scripts at the intended location. ([\#5570](#5570))

commit f4343c7
Merge: 4689408 463d5a8
Author: Amber H. Brown <[email protected]>
Date:   Wed Jul 3 22:39:30 2019 +1000

    Merge remote-tracking branch 'origin/develop' into shhs

commit 4689408
Merge: bed45ab b491468
Author: Amber H. Brown <[email protected]>
Date:   Tue Jul 2 18:31:29 2019 +1000

    Merge remote-tracking branch 'origin/develop' into shhs

commit bed45ab
Author: Amber H. Brown <[email protected]>
Date:   Tue Jul 2 18:18:09 2019 +1000

    release shhs on tags

commit 0993b05
Author: Amber H. Brown <[email protected]>
Date:   Mon Jul 1 23:13:21 2019 +1000

    improve error text when room is too large

commit e001115
Author: Amber H. Brown <[email protected]>
Date:   Tue Jun 18 21:24:56 2019 +1000

    fix

commit e60aab1
Merge: e7c1171 82d9d52
Author: Amber H. Brown <[email protected]>
Date:   Tue Jun 18 21:20:13 2019 +1000

    Merge remote-tracking branch 'origin/develop' into shhs

commit e7c1171
Merge: 8fe26db c831748
Author: Amber Brown <[email protected]>
Date:   Tue Jun 4 20:41:59 2019 +1000

    Merge remote-tracking branch 'origin/master' into shhs

commit 8fe26db
Merge: c99c105 4a30e4a
Author: Amber Brown <[email protected]>
Date:   Tue May 21 14:30:47 2019 -0500

    Merge remote-tracking branch 'origin/develop' into HEAD

commit c99c105
Author: Amber Brown <[email protected]>
Date:   Mon May 20 17:01:50 2019 -0500

    SHHS - Room Join Complexity (#5072)

commit d142e51
Merge: d424ba9 24b93b9
Author: Amber Brown <[email protected]>
Date:   Mon May 20 15:43:08 2019 -0500

    Merge remote-tracking branch 'origin/develop' into shhs

commit d424ba9
Merge: a1b8767 f1e5b41
Author: Amber Brown <[email protected]>
Date:   Wed May 15 23:30:22 2019 -0500

    Merge remote-tracking branch 'origin/develop' into shhs

commit a1b8767
Merge: faee1e9 df2ebd7
Author: Amber Brown <[email protected]>
Date:   Mon May 13 15:01:58 2019 -0500

    Merge remote-tracking branch 'origin/develop' into shhs

commit faee1e9
Merge: 12875f9 d216a36
Author: Amber Brown <[email protected]>
Date:   Wed May 8 16:29:45 2019 -0500

    Merge remote-tracking branch 'origin/develop' into shhs

commit 12875f9
Merge: ed38141 c1799b0
Author: Amber Brown <[email protected]>
Date:   Wed May 1 10:55:14 2019 -0400

    Merge remote-tracking branch 'origin/develop' into shhs

commit ed38141
Author: Amber Brown <[email protected]>
Date:   Mon Apr 29 20:57:42 2019 +1000

    target better for the shhs release docker hub, pt 3

commit bd5f624
Author: Amber Brown <[email protected]>
Date:   Mon Apr 29 20:43:17 2019 +1000

    target better for the shhs release docker hub, pt 2

commit c0f57ca
Author: Amber Brown <[email protected]>
Date:   Mon Apr 29 20:36:35 2019 +1000

    target better for the shhs release docker hub

commit 1d5cf66
Author: Amber Brown <[email protected]>
Date:   Mon Apr 29 20:33:36 2019 +1000

    no media repo == no path checks

commit 25256f9
Author: Amber Brown <[email protected]>
Date:   Mon Apr 29 20:30:55 2019 +1000

    release shhs as a release

commit a32aa2c
Author: Amber Brown <[email protected]>
Date:   Mon Apr 29 20:28:40 2019 +1000

    patch up docker

commit cbc866a
Author: Amber Brown <[email protected]>
Date:   Fri Apr 26 01:40:01 2019 +1000

    Remove Python 2 from the SHHS branch CI (#5099)
@erikjohnston erikjohnston deleted the erikj/refactor_changed_devices branch January 9, 2020 15:50
anoadragon453 added a commit that referenced this pull request Feb 14, 2020
@anoadragon453
Merge pull request #5559 from matrix-org/erikj/refactor_changed_devices
9b3dc01
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@richvdh richvdh richvdh approved these changes

Assignees

@erikjohnston erikjohnston

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@erikjohnston @richvdh