mypy plugin to check @cached return types

[DMRobertson]

The things that are cached should be immutable to avoid callers modifying cached values (which has lead to bugs in the past). This updates our mypy plug-in to enforce this via the type system.

Complements #13755.

Errors which have been fixed

The main obstruction seems to be that we have lots of Dict[str, Any] or Dict[str, str] in caches.

Sample:

+ mypy
synapse/handlers/read_marker.py:44: error: function get_account_data_for_room_and_type of AccountDataWorkerStore is @cached, but has mutable return value Union[dict[str, Any], None]  [synapse-@cached-mutable]
synapse/appservice/__init__.py:187: error: function get_local_users_in_room of RoomMemberWorkerStore is @cached, but has mutable return value list[str]  [synapse-@cached-mutable]
synapse/appservice/__init__.py:248: error: function get_aliases_for_room of DirectoryWorkerStore is @cached, but has mutable return value list[str]  [synapse-@cached-mutable]
synapse/app/phone_stats_home.py:203: error: function get_monthly_active_count_by_service of MonthlyActiveUsersWorkerStore is @cached, but has mutable return value dict[str, int]  [synapse-@cached-mutable]
synapse/api/filtering.py:170: error: function get_user_filter of FilteringWorkerStore is @cached, but has mutable return value dict[str, Any]  [synapse-@cached-mutable]
synapse/storage/databases/main/receipts.py:204: error: function _get_receipts_for_user_with_orderings of ReceiptsWorkerStore is @cached, but has mutable return value dict[str, Any]  [synapse-@cached-mutable]
synapse/storage/databases/main/receipts.py:283: error: function _get_linearized_receipts_for_rooms of ReceiptsWorkerStore is @cached, but has mutable return value dict[str, list[dict[str, Any]]]  [synapse-@cached-mutable]
synapse/storage/databases/main/receipts.py:309: error: function _get_linearized_receipts_for_room of ReceiptsWorkerStore is @cached, but has mutable return value list[dict[str, Any]]  [synapse-@cached-mutable]
synapse/storage/databases/main/events_worker.py:1618: error: function _have_seen_events_dict of EventsWorkerStore is @cached, but has mutable return value dict[str, bool]  [synapse-@cached-mutable]
synapse/storage/databases/main/events_worker.py:1668: error: function _have_seen_events_dict of EventsWorkerStore is @cached, but has mutable return value dict[str, bool]  [synapse-@cached-mutable]
synapse/storage/databases/main/signatures.py:77: error: function get_event_reference_hashes of SignatureWorkerStore is @cached, but has mutable return value dict[str, dict[str, bytes]]  [synapse-@cached-mutable]
synapse/storage/databases/main/roommember.py:385: error: function get_invited_rooms_for_local_user of RoomMemberWorkerStore is @cached, but has mutable return value list[synapse.storage.roommember.RoomsForUser]  [synapse-@cached-mutable]
synapse/storage/databases/main/roommember.py:723: error: function _get_rooms_for_users of RoomMemberWorkerStore is @cached, but has mutable return value dict[str, frozenset[str]]  [synapse-@cached-mutable]
synapse/storage/databases/main/roommember.py:795: error: function get_users_in_room of RoomMemberWorkerStore is @cached, but has mutable return value list[str]  [synapse-@cached-mutable]
synapse/storage/databases/main/roommember.py:861: error: function _get_user_ids_from_membership_event_ids of RoomMemberWorkerStore is @cached, but has mutable return value dict[str, Union[str, None]]  [synapse-@cached-mutable]
synapse/storage/databases/main/roommember.py:969: error: function get_users_in_room of RoomMemberWorkerStore is @cached, but has mutable return value list[str]  [synapse-@cached-mutable]
synapse/storage/databases/main/roommember.py:1020: error: function get_current_hosts_in_room of RoomMemberWorkerStore is @cached, but has mutable return value set[str]  [synapse-@cached-mutable]
synapse/storage/databases/main/roommember.py:1098: error: function _get_joined_hosts_cache of RoomMemberWorkerStore is @cached, but has mutable return value synapse.storage.databases.main.roommember._JoinedHostsCache (non-frozen attrs class)  [synapse-@cached-mutable]
synapse/storage/databases/main/roommember.py:1098: note: Error code "synapse-@cached-mutable" not covered by "type: ignore" comment

Some outstanding TODOs:

• check that frozen attrs classes have immutable fields
• work out how to handle types written in Rust

[clokep]

Can we return Mapping in that case? Or is your point that we need to audit a bunch of stuff?

[clokep]

You grabbed my interest. Seems reasonable but I have a couple thoughts!

[DMRobertson]

(Thanks for your thoughts, Patrick. I'll need to give this and them more thought, when I get some spare time to stare at this again)

[DMRobertson]

Can we return Mapping in that case? Or is your point that we need to audit a bunch of stuff?

Yep, I think we should return a Mapping to make it clear that the caller can't mutate dictionaries that are kept as cache values.

[DMRobertson]

Exciting to see!

[clokep]

Terrifying IMO of how we use this.

[DMRobertson]

Does this slow down mypy a lot?

Testing with:

git rev-parse --short HEAD
rm -r .mypy_cache/
poetry install --all-extras --sync
mypy --version
time mypy

On this branch:

$ git rev-parse --short HEAD
51a1a5fc4
$ mypy --version
mypy 1.5.1 (compiled: yes)
$ time mypy
Success: no issues found in 851 source files

real	0m36.189s
user	0m34.196s
sys	0m1.093s

On develop:

$ git rev-parse --short HEAD
7ec0a141b
$ mypy --version
mypy 1.5.1 (compiled: yes)
$ time mypy
Success: no issues found in 851 source files

real	0m33.438s
user	0m31.706s
sys	0m0.934s

So ~3s slower, say 10% overhead. I think I'd consider that good enough as-is.

[clokep]

Yeah that sounds reasonable to me.

[clokep]

I think this probably needs a review from not @DMRobertson and not myself since we collaborated on this.

[erikjohnston]

Otherwise looks good!

[erikjohnston]

This is great! 🎉

[DMRobertson]

Aside: If we have function returning e.g. Deferred[Deferred[T]], it looks like this function doesn't loop and extract the inner T. But we shouldn't be doing that in the first place.

(Would such a type even be legitimate?)

[clokep]

(Would such a type even be legitimate?)

I think it wouldn't make sense? Please don't do that.

[clokep]

I'm not sure that's even really doable since a Deferred returning a Deferred will wait on the inner Deferred? https://docs.twistedmatrix.com/en/stable/core/howto/defer.html#chaining-deferreds

Anyway, that interaction always confuses me and I'm willing to not support it.

[DMRobertson]

<3

[DMRobertson]

Can we add a sentence explaining why there is a wrapper? It looks like the point is mainly to check that we have a proper CallableType(?)

OTOH if it's just "to have smaller functions" then let's leave it as is.

[clokep]

OTOH if it's just "to have smaller functions" then let's leave it as is.

I almost inlined it, but it seemed nice to keep them separate.

[clokep]

Thanks so much for the help @erikjohnston and @DMRobertson! Happy to see this merged! 🎉