Preloading Not Working in a VERY Odd Way which Leaves Stuff Loaded Between Executions...?!

[Pixelmusement]

Godot version:
3.3.stable.official

OS/device including version:
Windows 10 Pro 2004 - Build 19041.423

Issue description:
So I tried loading up my project in the latest stable build of Godot and while most everything is functionally working fine, preloading of assets seems... broken in a way that's extremely difficult to describe... I'm going to try my best though.

So, what's happening at first after loading a project and running it is that NOTHING is preloading. When the game first starts up, every single thing which appears on screen causes a huge delay of several frames to load in, including environment effects like the sun, suggesting this might be shader related,

HOWEVER, when you stop running the project, go back to the editor, then run the project again, certain things will STILL be loaded, even things which are NOT being preloaded, and thus they won't trigger those same delays, however, it seems to be entirely random what stays loaded and what doesn't, with more things persisting with each run until you ultimately get to the point where every time you run your game, everything is already loaded into memory, regardless of whatever is being preloaded or not.

If you quit the editor to the project list and go back in, stuff stays loaded. ONLY if you quit the editor ENTIRELY will everything get unloaded, leading to those massive delays again the next time you load the editor.

Suffice to say, this is making it impossible to know what's preloading properly or not, plus is completely disrupting the ability to have a stable framerate when starting a game; the loading delays are several frames long, longer than I remember them being without preloading back in 3.2.

For reference, I'm using GLES3. No idea if this is happening on GLES2. I also tested to see if the nVidia "Shader Cache" feature was to blame but no, turning that off had no effect on this issue. I have an 8-Core AMD FX-8350 4 GHz CPU, a GTX 1070 video card, and 16 GB of RAM, so I doubt my specs are the issue either.

Steps to reproduce:

1. Load ANY project which has enough content to warrant preloading.
2. Run the project and experience all the delays from things loading only when they come on screen.
3. Quit back into the editor.
4. Run the project again and notice that some things but not everything is causing delays.
5. Repeat 3 and 4 until you notice no delays anymore.
6. Quit the editor entirely and the state of everything should go back to step 1.

Minimal reproduction project:
So far on my end this happens with literally ANY project which needs to preload numerous assets and will be far more noticeable with large projects as opposed to small projects for that reason.

[akien-mga]

Could you clarify what you mean by preloading exactly? Do you mean loading scenes or resources with preload(), or manually instancing scenes or resources at scene start?

Examples of the problematic code or scene setup would be great, or ideally a reproduction project (I understand that it needs to have a lot of assets to be reproducible but maybe a base scene setup for this "preloading" logic can be shared together with instructions to duplicate assets to create this "large project" condition?).

[Calinou]

Is this a 2D or 3D project? Shader compilation issues are mainly an issue in 3D and GLES3, but the issue has already been reported here. You can work around this by displaying objects that require shader compilation under the ground for one frame, then hiding them. There's also a shader caching pull request that should improve the situation noticeably once it's merged.

[Pixelmusement]

Could you clarify what you mean by preloading exactly? Do you mean loading scenes or resources with preload(), or manually instancing scenes or resources at scene start?

Using "preload()" or the ResourcePreloader object. Neither seem to be doing anything.

Is this a 2D or 3D project? Shader compilation issues are mainly an issue in 3D and GLES3, but the issue has already been reported here. You can work around this by displaying objects that require shader compilation under the ground for one frame, then hiding them. There's also a shader caching pull request that should improve the situation noticeably once it's merged.

Yes, it's in 3D. Stuff has to actually visually show up on screen before the delay happens, so loading objects into the scene off-screen doesn't help. Again though, once loaded in one run of the program, quitting to the editor and running again has some things still loaded and some not. It's very weird and was definitely not behaving this way in 3.2.

[akien-mga]

If it's shader related, it might also be hardware/driver related, which can make it difficult for contributors to reproduce (and thus fix) the issue.

Here are some ideas of information which could be helpful to gather to narrow it down further:

• Try your project(s) again with 3.2.3-stable to confirm that the behave did indeed change with 3.3-stable.
• If the regression is confirmed, try intermediate releases (3.2.4 betas and RCs, 3.3 RCs - they're contiguous, 3.2.4 was renamed to 3.3) to identify the first release where the regression happened. This can make it much easier to narrow down what change may be causing this issue.
• Try your project(s) on other hardware to see if the issue is reproducible or specific to your hardware and/or OS/drivers combination.

[Pixelmusement]

So I rolled back to 3.2.3 to see if I was just crazy or not and can confirm a couple things.

1. Preloading is definitely working fine in 3.2.3 (well, for as many things as it was working for, particle shaders not-withstanding) so something broke between 3.2.3 and 3.3.
2. The behaviour of NON-preloaded objects, things which are not preloaded at all and thus don't load in or compile shaders until they first appear on screen, is identical, sometimes staying loaded between runs and sometimes not.

I think what might be going on here is I ran into two problems simultaneously. Part of it absolutely seems related to the stuff Calinou mentioned just a short while ago, but there was definitely a problem introduced with preloading between 3.2.3 and 3.3 given that rolling back to 3.2.3 put my game back into the state I expected it to be in.

And yeah, I just looked and saw there's Win64 builds of Godot for each beta and RC so I'll give them all a shot starting in the middle and report back where the break occurs with preloading.

[Pixelmusement]

Can confirm the problem began with v3.2.4.beta4.official

v3.2.4.beta3.official does not have the preloading issue I found. Furthermore, while testing the various beta builds I noticed the shader caching issue which also seems to be a part of what I ran into was greatly minimized in beta1 through beta3, though comes right back into the forefront with beta4 onward and is also present in 3.2.3 stable. :o

[akien-mga]

Here's the changelog between 3.2.4 beta 3 and beta 4: b9b773c...b5e8b48

I don't see anything that would seem likely to affect preloading at a first glance.

BTW are you using GDScript or C#? And do you use FBX scenes?

[Pixelmusement]

I'm using GDScript as when I originally tried to get the Mono-enabled build of Godot working awhile back I ran into serious performance issues all across the board despite otherwise being the exact same build at the time.

Looking through the changelog right at the very end is a single change with a laundry list of changes all at once listed inside of it, one of which mentions changes to how materials are handled, but apart from that yeah, I don't see anything obvious which would be responsible for this. :|

[akien-mga]

Looking through the changelog right at the very end is a single change with a laundry list of changes all at once listed inside of it, one of which mentions changes to how materials are handled

If you mean de61cfe yeah that's a potentially sus commit, but it's only relevant if you're using FBX. It shouldn't impact native Godot scenes or other external scene formats.

[akien-mga]

If you have a compilation environment setup (e.g. Visual Studio), the next step could be to attempt bisecting the regression following these steps: https://docs.godotengine.org/en/latest/community/contributing/bisecting_regressions.html

That would be between the commits b9b773c (beta 3, good) and b5e8b48 (beta 4, bad), but fair warning, this can be fairly time consuming depending on how fast Godot builds on your hardware (given the specs I'd expect ~5-10 min per build, and this bisect would have ~6 steps).

Before doing that I'd double check that the regression indeed happened between beta 3 and beta 4 - when testing multiple builds it's easy enough to misidentify things, speaking from experience, and then spending hours bisecting something in the wrong range.

Alternatively, I could do Windows builds for you and we could do the bisect remotely, but that could take a while with build, upload, testing, getting a result,making a new build, etc.

[Pixelmusement]

Well... I've been doing more testing and I've now managed to get the problem to manifest in every version of Godot from 3.2.3.stable.official all the way up to 3.3.stable.official simply by altering which objects are visible and which are not and there doesn't seem to be any rhyme or reason as to what triggers this and what doesn't, except that things definitely are staying loaded/cached between executions in the editor, but not EVERYTHING.

Geeze this is so confusing... I just want to be able to preload things so that there's no skips in the framerate and this has just been an ongoing war I've had to fight every step of the way trying zillions of things to eliminate those stutters and delays, yet they always find ways to come back or solutions always find ways to actually NOT be solutions long-term and it's just completely sucked all of the fun and progress out of trying to make stuff with Godot... I really don't know if I can keep this up, let alone contribute any useful help to get it fixed right. >_<