Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

initial cut at a room summary API #3574

Merged
merged 52 commits into from
Aug 16, 2018
Merged

initial cut at a room summary API #3574

merged 52 commits into from
Aug 16, 2018

Conversation

ara4n
Copy link
Member

@ara4n ara4n commented Jul 23, 2018

this works, although considers all room membership events rather than just joined
ones as i'm failing to see a nice way to filter based on membership.

Implements matrix-org/matrix-spec-proposals#688

Builds on #3568 (in turn #3567 (in turn #3331 (in turn #2970))). (all now merged)

Sytest at matrix-org/sytest#470

ara4n added 13 commits June 4, 2018 01:04
as per the proposal; we can deduplicate redundant lazy-loaded members
which are sent in the same sync sequence. we do this heuristically
rather than requiring the client to somehow tell us which members it
has chosen to cache, by instead caching the last N members sent to
a client, and not sending them again.  For now we hardcode N to 100.
Each cache for a given (user,device) tuple is in turn cached for up to
X minutes (to avoid the caches building up).  For now we hardcode X to 30.
works, although considers all room membership events rather than just joined
ones as i'm failing to see a nice way to filter based on membership
@ara4n ara4n requested a review from a team July 23, 2018 03:11

missing_hero_event_ids = [
member_ids[hero_id]
for hero_id in summary['heros']
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be m.heros? I get a exceptions.KeyError: 'heros' when running this locally.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed it, as am 99% certain that that is what it should be, and it unblocked me locally from syncing.

@bwindels
Copy link
Contributor

m.joined_member_count seems to be irrespective of membership. While testing in a DM, upon the other user leaving, I get this from /sync for the given room. Is it also intended that the left user is still included in m.heros? I guess it could, as you won't want the room name to change, but different from what the proposal says:

{
  "!VHGOLZdMXhOiFmizWb:localhost": {
    "timeline": {
      "limited": false,
      "prev_batch": "s500_5160_4_83_232_1_1_56_1",
      "events": [
        {
          "origin_server_ts": 1532351680209,
          "sender": "@bruno5:localhost",
          "event_id": "$153235168010hSmVV:localhost",
          "unsigned": {
            "prev_content": {
              "membership": "join",
              "avatar_url": "mxc://localhost/JyLQUEVWImjlOIyJRLGxqZtD",
              "displayname": "Bruno Mars"
            },
            "prev_sender": "@bruno5:localhost",
            "replaces_state": "$15323515718rGSwF:localhost",
            "age": 81
          },
          "state_key": "@bruno5:localhost",
          "content": {
            "membership": "leave"
          },
          "membership": "leave",
          "type": "m.room.member"
        }
      ]
    },
    "summary": {
      "m.joined_member_count": 2,
      "m.heros": [
        "@bruno1:localhost",
        "@bruno5:localhost"
      ]
    }
  }
}

@ara4n
Copy link
Member Author

ara4n commented Jul 23, 2018

Yup, see #3574 (comment). There are a bunch of correctness errors here, but at least the API can be developed against even if the data it gives is a bit wrong currently. See also the other FIXMEs at https://github.com/matrix-org/synapse/pull/3574/files#diff-6fee1701e980dbc9da57a5a108ee7b65R556 for other limitations right now.

@bwindels
Copy link
Contributor

Yeah, just realized you mentioned this in the description

@ara4n ara4n assigned richvdh and unassigned ara4n Aug 12, 2018
@ara4n
Copy link
Member Author

ara4n commented Aug 12, 2018

@richvdh PTAL.

Copy link
Member

@richvdh richvdh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm modulo nitpicking about early returns.

Note that the PR is currently based on #3568 which has pending comments.

summary["m.joined_member_count"] = len(joined_user_ids)
summary["m.invited_member_count"] = len(invited_user_ids)

if not name_id and not canonical_alias_id:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we do an early return if name_id or canonical_alias_id rather than having this massive if block?

[user_id for user_id in member_ids.keys() if user_id != me]
)[0:5]

if sync_config.filter_collection.lazy_load_members():
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, make this into an early return?

@richvdh richvdh assigned ara4n and unassigned richvdh Aug 13, 2018
@ara4n
Copy link
Member Author

ara4n commented Aug 14, 2018

I'm very dubious about the early returns; it feels much less error prone to say "only do this thing if this condition is true" rather than "return early if this condition is false, as coincidentally the remainder of the method is handling that condition being true" - as:

  • the latter results in condition-specific stuff being grouped together (indentationwise) with the generic bits of the method
  • it's harder to read, as the reader has to skim for early returns
  • if folks need to add more functionality to the method, they can no longer do it at the end, as the end has ended up on some obscure conditional branch (which may not be obvious unless they skim the whole thing for early returns)
  • it feels like trying to reduce nesting for the sake of reducing nesting.

However, I've gone and done it anyway in a bid to get this merged, but i am still wondering whether we are making reviews too subjective (or making it too hard to determine the 'strength' of review feedback - and whether it's a "if i were doing this, i'd have considered doing it like this, but it's pretty subjective" or a "this is cosmetic nitpick but something that Should Be Fixed.")

@ara4n
Copy link
Member Author

ara4n commented Aug 15, 2018

@richvdh ptal

@ara4n ara4n assigned richvdh and unassigned ara4n Aug 15, 2018
@richvdh richvdh changed the base branch from matthew/members_at to develop August 15, 2018 15:43
@richvdh
Copy link
Member

richvdh commented Aug 15, 2018

I already LGTMed this; it now looks even better. Just waiting for the tests to run.

To address your points, certainly a better alternative might be to split out separate functions.

the latter results in condition-specific stuff being grouped together (indentationwise) with the generic bits of the method

yes it does; I don't really see that as a problem.

it's harder to read, as the reader has to skim for early returns

Personally I find early returns much easier to read than having to scroll up and down to figure out how the indentation matches up and which "if" clause we're leaving.

if folks need to add more functionality to the method, they can no longer do it at the end, as the end has ended up on some obscure conditional branch (which may not be obvious unless they skim the whole thing for early returns)

I think adding stuff to the end of a function without looking at what else is going on would be, er, foolhardy.

it feels like trying to reduce nesting for the sake of reducing nesting.

sure it is.

i am still wondering whether we are making reviews too subjective (or making it too hard to determine the 'strength' of review feedback - and whether it's a "if i were doing this, i'd have considered doing it like this, but it's pretty subjective" or a "this is cosmetic nitpick but something that Should Be Fixed.")

well, we discussed this in the team meeting the other day, and yes, these topics were discussed. I don't think here is the right place to reiterate that conversation, but in this case: I am coming to this as the person who is probably going to have to maintain this code (or at least review future changes to it in the future), and I do not want to have to maintain code that has massive nested if blocks that are hard to follow. Thank you for changing it.

@@ -626,6 +626,16 @@ def compute_summary(self, room_id, sync_config, batch, state, now_token):

defer.returnValue(summary)

def get_lazy_loaded_members_cache(self, cache_key):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: in future, please do add docstrings to public functions: in particular it's important to document the args and return types.

@richvdh richvdh merged commit 3f543dc into develop Aug 16, 2018
richvdh added a commit that referenced this pull request Aug 22, 2018
Features
--------

- Add support for the SNI extension to federation TLS connections. Thanks to @vojeroen! ([\#3439](#3439))
- Add /_media/r0/config ([\#3184](#3184))
- speed up /members API and add `at` and `membership` params as per MSC1227 ([\#3568](#3568))
- implement `summary` block in /sync response as per MSC688 ([\#3574](#3574))
- Add lazy-loading support to /messages as per MSC1227 ([\#3589](#3589))
- Add ability to limit number of monthly active users on the server ([\#3633](#3633))
- Support more federation endpoints on workers ([\#3653](#3653))
- Basic support for room versioning ([\#3654](#3654))
- Ability to disable client/server Synapse via conf toggle ([\#3655](#3655))
- Ability to whitelist specific threepids against monthly active user limiting ([\#3662](#3662))
- Add some metrics for the appservice and federation event sending loops ([\#3664](#3664))
- Where server is disabled, block ability for locked out users to read new messages ([\#3670](#3670))
- set admin uri via config, to be used in error messages where the user should contact the administrator ([\#3687](#3687))
- Synapse's presence functionality can now be disabled with the "use_presence" configuration option. ([\#3694](#3694))
- For resource limit blocked users, prevent writing into rooms ([\#3708](#3708))

Bugfixes
--------

- Fix occasional glitches in the synapse_event_persisted_position metric ([\#3658](#3658))
- Fix bug on deleting 3pid when using identity servers that don't support unbind API ([\#3661](#3661))
- Make the tests pass on Twisted < 18.7.0 ([\#3676](#3676))
- Don’t ship recaptcha_ajax.js, use it directly from Google ([\#3677](#3677))
- Fixes test_reap_monthly_active_users so it passes under postgres ([\#3681](#3681))
- Fix mau blocking calulation bug on login ([\#3689](#3689))
- Fix missing yield in synapse.storage.monthly_active_users.initialise_reserved_users ([\#3692](#3692))
- Improve HTTP request logging to include all requests ([\#3700](#3700))
- Avoid timing out requests while we are streaming back the response ([\#3701](#3701))
- Support more federation endpoints on workers ([\#3705](#3705), [\#3713](#3713))
- Fix "Starting db txn 'get_all_updated_receipts' from sentinel context" warning ([\#3710](#3710))
- Fix bug where `state_cache` cache factor ignored environment variables ([\#3719](#3719))
- Fix bug in v0.33.3rc1 which caused infinite loops and OOMs ([\#3723](#3723))
- Fix bug introduced in v0.33.3rc1 which made the ToS give a 500 error ([\#3732](#3732))

Deprecations and Removals
-------------------------

- The Shared-Secret registration method of the legacy v1/register REST endpoint has been removed. For a replacement, please see [the admin/register API documentation](https://github.com/matrix-org/synapse/blob/master/docs/admin_api/register_api.rst). ([\#3703](#3703))

Internal Changes
----------------

- The test suite now can run under PostgreSQL. ([\#3423](#3423))
- Refactor HTTP replication endpoints to reduce code duplication ([\#3632](#3632))
- Tests now correctly execute on Python 3. ([\#3647](#3647))
- Sytests can now be run inside a Docker container. ([\#3660](#3660))
- Port over enough to Python 3 to allow the sytests to start. ([\#3668](#3668))
- Update docker base image from alpine 3.7 to 3.8. ([\#3669](#3669))
- Rename synapse.util.async to synapse.util.async_helpers to mitigate async becoming a keyword on Python 3.7. ([\#3678](#3678))
- Synapse's tests are now formatted with the black autoformatter. ([\#3679](#3679))
- Implemented a new testing base class to reduce test boilerplate. ([\#3684](#3684))
- Rename MAU prometheus metrics ([\#3690](#3690))
- add new error type ResourceLimit ([\#3707](#3707))
- Logcontexts for replication command handlers ([\#3709](#3709))
- Update admin register API documentation to reference a real user ID. ([\#3712](#3712))
@hawkowl hawkowl deleted the matthew/room_summary branch September 20, 2018 14:01
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants