Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

disable LL for incr syncs, and log incr sync stats #3840

Merged
merged 8 commits into from
Sep 11, 2018

Conversation

ara4n
Copy link
Member

@ara4n ara4n commented Sep 10, 2018

No description provided.

@ara4n ara4n changed the base branch from master to develop September 10, 2018 23:00
@ara4n ara4n changed the base branch from develop to matthew/speed-up-ll September 10, 2018 23:00
@ara4n ara4n changed the title Matthew/disable ll on incr syncs disable LL for incr syncs, and log incr sync stats Sep 10, 2018
@ara4n ara4n requested a review from erikjohnston September 10, 2018 23:00
@ara4n
Copy link
Member Author

ara4n commented Sep 10, 2018

just realised there’s a thinko on this:

  • A joins the room
  • lots of traffic goes by
  • B is in the room and does an initial sync with LL; A isn’t included due to LL
  • B doesn’t do a /members call to load the full membership in the bg
  • time goes by
  • A speaks
  • B does a gappy incr sync, and receives A’s msg
  • B doesn’t know who A is as he’s never received a member for them.

We could mitigate this by assuming all clients do bg /members calls. But iOS for instance doesn’t yet.

I think we have to include LL events in state like we do for non-gappy incr syncs already.

note for me in the morning; hold review until then

as the user might not have heard of them before, even though we’re not LLing members in incremental syncs
@ara4n
Copy link
Member Author

ara4n commented Sep 10, 2018

actually, i think ^ fixes it. r4r!

Copy link
Member

@erikjohnston erikjohnston left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Modulo fixing PEP8 due to tab vs spaces

@ara4n
Copy link
Member Author

ara4n commented Sep 11, 2018

heh, yay for github-on-phone using tabs >:(

that said, whilst writing more sytests for this i think i've found another thinko :(

@ara4n ara4n merged commit c4aa9d6 into matthew/speed-up-ll Sep 11, 2018
ara4n added a commit that referenced this pull request Sep 11, 2018
* speed up room summaries by pulling their data from room_memberships rather than room state
* disable LL for incr syncs, and log incr sync stats  (#3840)
hawkowl added a commit that referenced this pull request Sep 24, 2018
Features
--------

- Python 3.5 and 3.6 support is now in beta.
([\#3576](#3576))
- Implement `event_format` filter param in `/sync`
([\#3790](#3790))
- Add synapse_admin_mau:registered_reserved_users metric to expose
number of real reaserved users
([\#3846](#3846))

Bugfixes
--------

- Remove connection ID for replication prometheus metrics, as it creates
a large number of new series.
([\#3788](#3788))
- guest users should not be part of mau total
([\#3800](#3800))
- Bump dependency on pyopenssl 16.x, to avoid incompatibility with
recent Twisted.
([\#3804](#3804))
- Fix existing room tags not coming down sync when joining a room
([\#3810](#3810))
- Fix jwt import check
([\#3824](#3824))
- fix VOIP crashes under Python 3 (#3821)
([\#3835](#3835))
- Fix manhole so that it works with latest openssh clients
([\#3841](#3841))
- Fix outbound requests occasionally wedging, which can result in
federation breaking between servers.
([\#3845](#3845))
- Show heroes if room name/canonical alias has been deleted
([\#3851](#3851))
- Fix handling of redacted events from federation
([\#3859](#3859))
-  ([\#3874](#3874))
- Mitigate outbound federation randomly becoming wedged
([\#3875](#3875))

Internal Changes
----------------

- CircleCI tests now run on the potential merge of a PR.
([\#3704](#3704))
- http/ is now ported to Python 3.
([\#3771](#3771))
- Improve human readable error messages for threepid
registration/account update
([\#3789](#3789))
- Make /sync slightly faster by avoiding needless copies
([\#3795](#3795))
- handlers/ is now ported to Python 3.
([\#3803](#3803))
- Limit the number of PDUs/EDUs per federation transaction
([\#3805](#3805))
- Only start postgres instance for postgres tests on Travis CI
([\#3806](#3806))
- tests/ is now ported to Python 3.
([\#3808](#3808))
- crypto/ is now ported to Python 3.
([\#3822](#3822))
- rest/ is now ported to Python 3.
([\#3823](#3823))
- add some logging for the keyring queue
([\#3826](#3826))
- speed up lazy loading by 2-3x
([\#3827](#3827))
- Improved Dockerfile to remove build requirements after building
reducing the image size.
([\#3834](#3834))
- Disable lazy loading for incremental syncs for now
([\#3840](#3840))
- federation/ is now ported to Python 3.
([\#3847](#3847))
- Log when we retry outbound requests
([\#3853](#3853))
- Removed some excess logging messages.
([\#3855](#3855))
- Speed up purge history for rooms that have been previously purged
([\#3856](#3856))
- Refactor some HTTP timeout code.
([\#3857](#3857))
- Fix running merged builds on CircleCI
([\#3858](#3858))
- Fix typo in replication stream exception.
([\#3860](#3860))
- Add in flight real time metrics for Measure blocks
([\#3871](#3871))
- Disable buffering and automatic retrying in treq requests to prevent
timeouts. ([\#3872](#3872))
- mention jemalloc in the README
([\#3877](#3877))
- Remove unmaintained "nuke-room-from-db.sh" script
([\#3888](#3888))
ara4n added a commit that referenced this pull request Sep 24, 2018
Given we have disabled lazy loading for incr syncs in #3840, we can make self-LL more efficient by only doing it on initial sync.  Also adds a bounds check for if/when we change our mind, so that we don't try to include LL members on sync responses with no timeline.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants