'Export Chat' fails for very long chats (tab crashes)

[TheFel0x]

Steps to reproduce

1. Have a chat with close to 100,000 events
2. Click on "Export Chat" (HTML, From the beginning, no attachments)
3. Wait for Element to fetch all events from the server
4. Wait for Element to process the events

Outcome

What did you expect?

The usual export of the chat.

What happened instead?

After around 70-80k processed events the tab crashes on Chrome.
Same behavior on Firefox and element-desktop.
So far I haven't found any way to export large chats.

(This could maybe be avoided if it was possible to download the export in parts or if you were able to choose a time range for the export.)

Operating system

No response

Browser information

No response

URL for webapp

No response

Application version

Element version: 1.11.26 Olm version: 3.2.12

Homeserver

No response

Will you send logs?

No

[MrBruz]

Same issue here, exporting a channel that is falling apart at the seams with ~80k messages. Crashes near the end, its also crashes the render thread in paticular

[reduktr]

@TheFel0x What did you set your attachment limit? I'm currently trying to export 85000 events on element web and has been stuck on "Generating a Zip" for over 4 hours. Wondering if this will ever finish, or if this is a lost cause.

Mine is set to maximum attachment size (2000). Using Firefox Stable. I've tried to do the same on Element-Desktop and I get a white screen crash.

[eTomte]

I concur that this could use some improvement. My biggest chat is only 18,000 events, and already I find that sometimes the export hangs, and I have to try a few times to get one that completes successfully.

[TheFel0x]

@TheFel0x What did you set your attachment limit? I'm currently trying to export 85000 events on element web and has been stuck on "Generating a Zip" for over 4 hours. Wondering if this will ever finish, or if this is a lost cause.

@NeonMinnen

When I opened this issue I had tried different size limits for attachments as well as not including attachments at all. None of them worked for the chat I was trying to export.

I assume it never successfully finished exporting for you either?

[Johennes]

We've discussed this problem internally today and while we don't have a resolution, I wanted to summarize our current thinking.

The root cause for triggering OOMs is a combination of these two issues:

1. We are paging the retrieval of events (https://github.com/matrix-org/matrix-react-sdk/blob/237038aa563431bc1e5c09dd244fd43f81416779/src/utils/exportUtils/Exporter.ts#L154) and the zip writing (https://github.com/matrix-org/matrix-react-sdk/blob/develop/src/utils/exportUtils/HtmlExport.tsx#L446) separately from each other, meaning we first get all the data (in chunks) and then write it (also in chunks)
2. Separately from that JSZip's generateAsync keeps the whole file in memory during writing (https://stuk.github.io/jszip/documentation/limitations.html)

Fixing 1. seems feasible but we're not currently sure how to fix 2. in all browsers we currently support. Specifically, Safari doesn't appear to support FileSystemWritableFileStream (https://bugs.webkit.org/show_bug.cgi?id=231706#c7).

Options we have considered so far are:

• Split the zip into several – This may not be very user friendly and would require care to ensure the isolated archives work as expected
• Cap the maximum zip size by applying measurements during paging and point at other tools for exporting larger archives – This probably requires downloading more than is exported and could still cause OOMs in some cases

[Meister1593]

I noticed also that for some reason it hangs after fetching events too, on Desktop Element
Tried to debug it in firefox with local development version (librewolf to be precise), but it has gone through it seems like and processes events. Might be also issue specific to desktop element (either in reversing the collection, or decrypting that many messages, at the time i tried to add text with progress about decryption of events, but it may be unnecessary?...)