Admin API: purge_history doesn't delete local media

[zingi]

Description

When I purge the history of a room (with the admin API) on my homeserver, the locally stored media is not deleted.

This is a severe problem for people like me, who try to maintain a small homeserver. I want to purge the history of image-heavy rooms on a regular basis, to free up memory.

In my opinion it makes no sense to keep the pictures in any way anyway (from purged events). Since there is no real reference to them anymore, they are just dangling files, which occupy uselessly memory.

Steps to reproduce

1. du -sb /opt/synapse/media_store/ returns 3,331,432,273 Bytes
2. select count(*) from local_media_repository; returns 1,577 entires
3. Upload picture to (unencrypted) room
4. du -sb /opt/synapse/media_store/ returns 3,333,367,545 Bytes
5. select count(*) from local_media_repository; returns 1,579 entries
6. Purge History of room (with previously uploaded picture) using:
   POST /_synapse/admin/v1/purge_history/<room_id>[/<event_id>]

{
    "delete_local_events": true,
    "purge_up_to_ts": 1564822558201 // timestamp after the picture upload
}

Purge finishes with no errors and returns a purge_id.
If I call the purge status query it says:

{
  "status": "complete"
}

7. du -sb /opt/synapse/media_store/ still returns 3,333,367,545 Bytes
8. select count(*) from local_media_repository; still returns 1,579 entries

---

From the purge_history admin API, I expect that it:

• deletes all corresponding message events from the DB and
• deletes all corresponding local media from the local_media_repository
  • (simply delete all corresponding stored files: free up memory)

Version information

• Homeserver: personal homeserver with ​restricted federation

If not matrix.org:

• Version: v1.2.1 avhost docker

• Install method: docker-compose up -d

• Platform: VPS with Debian 9 64bit

[neilisfragile]

The reason that it doesn't just do this today is that it is not straight forward to determine definitively if a given piece of media can be safely removed since it may be referenced by multiple rooms. For instance stickers from a sticker pack are duplicated for each room.

[zingi]

I don't quite understand the difficulty here.
You can distinguish if the media is a sticker if the msgtype is m.sticker. And an image would be m.image. Or not?

[t3chguy]

Not in encrypted rooms, if a media is in both unencrypted and encrypted it'd disappear from the encrypted when the unencrypted got purged

[cryzed]

We made synapse-purge to purge remote media cache, local media and events in all rooms (both encrypted and unencrypted) until "x seconds ago" for the Synapse server, feel free to use it. It is highly configurable and should work on all instances using a Postgres database. This also preserves only the most recent user and room avatars and will remove old ones.

[richvdh]

this is by design. See also #3479.