summaryInvitedMemberCount is unreliable

[dbkr]

Describe the bug
This seems to cause summaryInvitedMemberCount to incorrectly be 1 which causes getInvitedMemberCount() in turn to be 1, despite the fact that the room state has two members, both of which are joined. The summary counts seem to not get reset in some circumstances.

This originally manifested as element-hq/element-web#20753 but I'm filing a separate bug as I'll revert the change that caused it.

To Reproduce
Steps to reproduce the behavior:
On Element-web / desktop:

1. On user A, start chat with user B
2. Accept on user B's client
3. Immediately refresh on User A's client

Expected behavior
summaryInvitedMemberCount should be 0 for the room (two users, both joined).