Deleting shader_cache causes the project to sometimes freeze on startup

[Alexofp]

Tested versions

v4.4.beta3.official [06acfcc]

System information

Godot v4.4.beta3 - Windows 10 (build 19045) - Multi-window, 1 monitor - Vulkan (Forward+) - dedicated NVIDIA GeForce GTX 1060 3GB (NVIDIA; 32.0.15.7216) - Intel(R) Core(TM) i5-6400 CPU @ 2.70GHz (4 threads)

Issue description

Deleting the 'shader_cache' and 'vulkan' folders leads to the project (sometimes) freezing on load. Only seem to happen if there are multiple complicated materials involved. See the video

2025-02-09.13-43-29.mp4

Steps to reproduce

1. Create a new project (Or use my MRP)
2. Make a simple scene with a camera3d and 2 cubes with different materials. I enabled a bunch of 'heavy' options on each, like subsurface scattering and rim-lighting
3. Run the project. It might freeze on the godot logo or it might run fine
4. Delete the shader_cache and 'vulkan' folders from the %appData%/Godot/app_userdata/ShaderFreezeBeta3
5. Try running the project. It might freeze or it might run.

Minimal reproduction project (MRP)

shaderfreezebeta3_2025-02-09_13-50-19.zip

(I also included a scene that always loads fine. It starts off with both cubes invisible and makes them visible one by one)

[matheusmdx]

Could you check if this bug happens in 4.3 stable and 4.4 beta 2?

[Alexofp]

Tried Godot 4.3 beta 2, worked each time (tried about 10 times, zero freezes)
Tried beta 3 again, freezed the first time I tried

[matheusmdx]

I can't reproduce this bug on my end, maybe is hardware related.

Godot v4.4.beta3 - Windows 10 (build 19045) - Multi-window, 2 monitors - Vulkan (Forward+) - dedicated AMD Radeon RX 580 2048SP (Advanced Micro Devices, Inc.; 31.0.21921.1000) - AMD Ryzen 5 3600 6-Core Processor (12 threads)

@Alexofp Can you check if this happens with the compatibility renderer?

---

@DarioSamo Any chance this issue be a regression from #102122?

[Alexofp]

Tried these things (all on beta 3, each one at least 5 times, 2 cubes with random standard materials, default world enviroment, deleting the shader cache each time):
Forward+ renderer - freezes pretty much every time
Mobile renderer - no freeze
Compatability renderer - no freeze
Forward+ (Dx12 driver) - the editor freezes on reopening scenes (Probably because it's trying to show the scene with the cubes). Same freezing logic as Vulkan. It fully freezes the first time, then runs the second time (but takes longer than vulkan)
Mobile (Dx12 driver) - no freeze

Latest gpu drivers (but was happening before I updated them today, that's the reason why I updated them)

Tried the same on Godot Beta 2:
Forward+ renderer - no freeze
Mobile renderer - no freeze
Compatability renderer - no freeze
Forward+ (Dx12 driver) - takes about 30 seconds but it does run first-time (game and editor)
Mobile (Dx12 driver) - no freeze

If I make it so the cubes are shown one by one (spreading the shader compilations over a few frames), it loads perfectly fine on Forward+ Vulkan on beta 3

[Alexofp]

If I disable a 'Shader Cache' option in the project settings, it runs fine too (beta 3, forward+, vulkan). Takes about 5 seconds to run each time (slower than if shaders are cached) but it never freezes

[Alexofp]

Did some more testing
Artifact from this commit freezes (not saying that it's the one that caused it, it might be ones before it) ee4acfb
This one (and earlier) works perfectly d56cbd3

Somewhere between these 2 commits is the one that causes it for me. I don't have the stuff setup to compile the engine, sorry, I just used the artifacts that github gives

[brcontainer]

Never delete any data related to the project while it is open. Even in other applications unrelated to Godot, doing so can cause unexpected crashes. Every now and then I need to delete data from the user folder, and at that time I make sure the editor is completely closed, but that's just a suggestion 👍

[akien-mga]

Tested on Linux (Fedora 41) with 4.4.beta3 and an AMD Radeon RX Vega 56 GPU (using Mesa drivers), and I can't reproduce the issue (tried around 20 times).

Might be OS / GPU / driver specific.

CC @DarioSamo in case you have a hunch as indeed #102122 and #102125 sound potentially related in the above partial bisect result.

[DarioSamo]

CC @DarioSamo in case you have a hunch as indeed #102122 and #102125 sound potentially related in the above partial bisect result.

Ubershader thread priority sounds like the more likely one, but that'd probably hint that we have some situation in our Worker Thread Pool implementation that can deadlock itself to death rather than some actual problem from the change itself.

The SDFGI change sounds unlikely to me. Does the project in question use SDFGI at all?

[Alexofp]

No SDFGI, no world enviroment node at all in fact

[HungryProton]

I could reproduce this bug on Windows 10 laptop - Geforce 850M (Driver 572.16)
But I could not reproduce it on a Linux - RX 6600 (Mesa drivers)

No issues with the beta 1

[xsellier]

I cannot reproduce this bug on Linux - NVidia RTX 3070 (driver 570.86.15-1)

[Alexofp]

Still happens in 4.4 RC1. Latest drivers.
Biggest problem is that some people won't be able to launch godot 4.4 forward+ games at all anymore since they obviously don't have any shaders compiled yet

[PhairZ]

couldn't reproduce in v4.4.beta3.official [06acfcc] or v4.4.rc1.
Windows 10 (build 19045) - Multi-window, 1 monitor - Vulkan (Forward+) - dedicated NVIDIA GeForce RTX 2070 SUPER (NVIDIA; 32.0.15.6636) - 11th Gen Intel(R) Core(TM) i5-11400F @ 2.60GHz (12 threads)

[lyuma]

We found a deadlock on 4.4RC3 that might be related to shader generation:

• Scenes can't be fully loaded and Godot freezes #102877

Given it relates to WorkerThreadPool, and that shaders get cached, it could be randomly non-reproducible, so I'm willing to bet it wasn't fixed, and it's the same issue as #102877 . Thankfully that issue has stacktraces for the threads, so we now know for certain that there is a deadlock here and what the cause is.

[Alexofp]

Yeah, pretty sure its the same issue. This one is probably not needed now, that one has more info anyway

[Alexofp]

Can confirm this is fixed after #103506

[clayjohn]

Awesome! Thank you. Closing as fixed by #103506