Eager loading resources

[markusbkk]

It is my impression that scummvm_fs.js lazy loads a lot of the game resources while the game is being played, making larger, more intense games feel like they're being played from a very slow CD-Rom drive (that sure is a very novel way of experiencing retro games on modern hardware 😸)

I dunno if this is due to cache limitations within the WASM spec or a self-imposed restriction.
RN, games like Grim Fandango are unplayable due to the constant streaming.

[neuromancer]

I think I saw the same in the wetlands demo, that sometimes there is a small skip in the animations.

[markusbkk]

I think I saw the same in the wetlands demo, that sometimes there is a small skip in the animations.

It's especially harsh over a slow internet connection, and Chrome (thankfully, not Firefox) gives me BSOD flashbacks because it doesn't halt the audio so it sounds like your speakers are about to blow up.

[markusbkk]

Changing the RANGE_REQUEST_BLOCK_SIZE to 4096 * 4096 or even higher (I ran into out of memory issues past 8192 * 8192) does make most games buttery smooth with only a slight hiccup syncing the next resources every now and then.

Grim Fandango is still starved for memory and I feel there's some sort of memory leak because Firefox task manager shows a memory consumption of up to 5GB while running the Curse of Monkey Island demo. CPU usage is also often in the reds.

[chkuendig]

Thank you for reporting this. There's an inherent tradeoff here on what to load. It's not realistic to download 100s of megabytes on launch of the game, OTOH the FS layer doesn't know when what data is required. I still plan on adding smarter caching for this, (i.e. speculatively download a lot of small files in one go) and also add some asynchronous loading (i.e. download the next 5MB asynchronously if the last read was about to empty the cache for a file - this should work as most games are optimized for CD-ROM/Floppy and the data is often read sequentially).

I wanted to wait for these things until the new FS layer in emscripten is ready - but didn't have any time recently to check again if that's already the case. Looks like they unfortunately didn't make much progress - emscripten-core/emscripten#15041.

[markusbkk]

Thank you for reporting this. There's an inherent tradeoff here on what to load. It's not realistic to download 100s of megabytes on launch of the game, OTOH the FS layer doesn't know when what data is required. I still plan on adding smarter caching for this, (i.e. speculatively download a lot of small files in one go) and also add some asynchronous loading (i.e. download the next 5MB asynchronously if the last read was about to empty the cache for a file - this should work as most games are optimized for CD-ROM/Floppy and the data is often read sequentially).

I wanted to wait for these things until the new FS layer in emscripten is ready - but didn't have any time recently to check again if that's already the case. Looks like they unfortunately didn't make much progress - emscripten-core/emscripten#15041.

I see. So is this completely dependent on Emscripten FS or would you consider PRs that modify ScummVM FS?

One thing I don't quite understand, btw: Why do we have to bundle the games? Since ScummVM pretty much requires a web server anyways, couldn't we just serve the games from the file system itself?

[chkuendig]

I see. So is this completely dependent on Emscripten FS or would you consider PRs that modify ScummVM FS?

I'd still consider PRs that modify ScummVM FS. Please send them directly upstream to scummvm, the emscripten port has been merged into 2.6.0. ( https://github.com/scummvm/scummvm/tree/master/dists/emscripten )

One thing I don't quite understand, btw: Why do we have to bundle the games? Since ScummVM pretty much requires a web server anyways, couldn't we just serve the games from the file system itself?

This depends on the goal of this port. My idea was always to provide a version which can be hosted centrally (ideally on scummvm.org) without users having to self-host. In that case, we would obv. need to have a way to load games. One way would be for users to bring their own (if we extend ScummVM FS with Dropbox/Google Drive support), the other is to already bundle a few games so it works at least as a demo page.