OOM crash with too many files

[ferndot]

If there are a large number of files in a storage, Firetext crashes with Out Of Memory.

We need to:

1. Determine the point at which this problem starts.
2. Discuss mitigation strategies for this issue. (I believe that it might be fixed by [File browser] Keep directory structure #174)

[ferndot]

Originally by @Ryuno-Ki. This is the directory structure of the storage that caused the crash:

$ du -h -d 2
3.8M    ./LOST.DIR
32K ./.android_secure
21M ./DCIM/100MZLLA
302M    ./DCIM/101MZLLA
3.5G    ./DCIM/MUSIC
1.6G    ./DCIM/VIDEO
21G ./DCIM/FILES
512K    ./DCIM/FFOS-Sticky-App-Imgs
128K    ./DCIM/FFOS-Drawl-App-Imgs
1.9G    ./DCIM/IMAGES
29G ./DCIM
66M ./.gallery/previews
67M ./.gallery
320K    ./.Trash-14908/info
25M ./.Trash-14908/files
26M ./.Trash-14908
128K    ./.Trash-1000/info
32K ./.Trash-1000/files
32K ./.Trash-1000/expunged
224K    ./.Trash-1000
32K ./downloads/bluetooth
64K ./downloads
5.5M    ./screenshots
64K ./.Trash-2111/info
3.8M    ./.Trash-2111/files
3.9M    ./.Trash-2111
64K ./contacts2xml/20130601020207
224K    ./contacts2xml
32K ./.downloaded
29G .

[Ryuno-Ki]

Thanks, Hoshua,

some notes here:

1. 32K is the size of every empty directory on the card's file system.
2. The output above contains only the upper most two levels (-d 2) Further details are too intrusive to me.
3. I filled a bug against Mozilla with ID #1099090 but no reaction by now.
4. I tried to reproduce the error in order to get a adb logcat to share, but … it worked.

You must have done some magic since I tried the app last.
Mostly .html and .txt files here.
Maybe it's clever to stress test the app in tests anyway …

[Ryuno-Ki]

I'd like to emphasize that my bug against Mozilla is still valid (PDF Viewer and file browser continue crashing). Just your app is fine :)

[ferndot]

@Ryuno-Ki: thanks for filing the bug. I will try to promote it so that it gets attention.

It is great to hear that Firetext didn't crash. We have done a bit of performance work lately =)

[Ryuno-Ki]

@joshua-s Would you and your team like to share some knowledge with Mozilla Hacks?

As stated in the bugzilla bug, several apps suffer from getDeviceStorage(), so it would be helpful, to explain your approach. I can look after reaching out to Robert Nyman and point him to this issue.

[ferndot]

@Ryuno-Ki I don't actually know what is preventing us from crashing. Our IO code is certainly very messy. Our performance improvements have been moving from SVG icons to a font, encapsulating the editor in an <iframe>, and optimizing some CSS rules.

If you would like to look, our IO enumerate function is here: https://github.com/codexa/firetext/blob/develop/scripts/io.js#L114

[Ryuno-Ki]

FYI bug #1104337 was linked today.