After C# build, the editor freezes longer and longer as the project size scales up

[J-Ponzo]

Tested versions

• Reproductible in :
  • v4.3.dev6.mono.official [89850d5]
  • v4.1.2.stable.mono.official [399c9dc]
  • v4.2.stable.mono.official [46dc277]
  • v4.2.1.stable.mono.official [b09f793]
  • v4.3.beta.mono.custom_build [be56cab]

System information

Godot v4.3.dev6.mono - Windows 10.0.19045 - Vulkan (Forward+) - dedicated NVIDIA GeForce RTX 2080 (NVIDIA; 31.0.15.3623) - Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz (8 Threads)

Issue description

If the project is big enough, the editor freezes every time you re-build C#, i.e.:

• Click Build from the toolbar after source changes
• Click Play from the toolbar after source changes
• Launch in debug mode from VisualStudio
• Click Rebuild Project from the MSBuild dock

The bigger the project is (in data size and scripts number), the longer the freeze lasts.

I tried to make a minimal reproduction project in order to investigate and I came with the following :

• create an empty project
• create 32 empty scripts in the project
• create 32 folders containing ~1GB of data each (to be specific, I used enough copies of those free textures to build my 1GB folder https://opengameart.org/content/700-noise-textures, but it should work with anything I guess)
• foreach nbScrips/dataAmount combinations, rebuild the project and measure the freeze time :
  • click Rebuild Project from the MSBuild dock
  • wait for the Building .NET project... popup to disappear to start the stopwatch
  • spam-click the editor until the window turns white and be marked (Not Responding)
  • get ready to stop the stopwatch as soon as the editor stops freezing

Doing so, I got the following results :

Freeze time (s)	4GB	8GB	12GB	16GB	20GB	24GB	28GB	32GB
4 Scripts	-	-	-	-	-	-	-	-
8 Script	-	-	-	-	-	6.18	7.68	7.74
12 Scripts	-	-	-	6.37	7.35	9.02	11.33	12.36
16 Scripts	-	-	6.25	8.68	9.82	12.18	15.05	16.60
20 Scripts	-	-	7.41	10.82	12.16	15.16	19.19	20.27
24 Scripts	-	6.24	8.83	12.71	14.6	18.09	22.26	24.37
28 Scripts	-	7.53	10.29	14.79	17.3	21.12	26.33	28.61
32 Scripts	-	8.30	11.75	16.84	21.21	24.22	29.41	31.96

The empty cells mean the freeze was too short to be properly measured (the editor unfreezes before the window turns white)

Obviously I cannot attach a 32GB minimal project to the issue, and recreating it by hand is really tedious. So I made a public git repository for convenience (see MRP section).

Edit : The MRP is no longer 32GB. I managed to reproduce with a < 5GB project. For convenience, the main branch of the repo points to this lite version.

I know this kind of thing can be really tricky to solve. I stay available for any additional information, testing or anything I can do to help with this. I really hope we can find something because with my actual project, it costs almost 4 minutes every time I click play so I can't really keep on developing it.

Steps to reproduce

• 1 - clone, download or recreate by hand the MRP (https://github.com/J-Ponzo/editor-freeze-study)
  

• 2 - open it with a .NET version of godot (double-click the godot exe then import the project https://docs.godotengine.org/en/stable/tutorials/editor/project_manager.html#opening-and-importing-projects)

• 3 - wait until the project is completely loaded. i.e. the FileSystem dock should contain something. (This may take a while)
  

• 4 - create the C# solution (Project => Tools => C# => Create C# solution)
  

• 5 - rebuild the C# project form the MSBuild dock
  

• 6 - spam-click the editor until it freezes

Minimal reproduction project (MRP)

The project is too big but it is available here : https://github.com/J-Ponzo/editor-freeze-study

[Gnumaru]

The repo for the MPR is empty. Have you forgot to push?

Also, what is a "Go"? is it a measurement unit of data? or have you misspelled? I googled it but couldn't find. Did you mean gigabyte or maybe gigabit?

[J-Ponzo]

"The MPR depot is empty. Did you forget to push?" => yes, sorry, I'm working on it. I'm using a graphical git client and it's having a bit of trouble handling the 201251 files in my commit. I think I need to use the CLI

"Also, what is a "Go"?" => This is Gigabyte. We say "gigaoctet" in french, I did not translate it properly. I edit that. Thank you !

Edit : It took forever, but the MRP is finally available in the repo

[MatthiasBae]

I can confirm the issue. It started also with Version 4.3 dev 6. Did not have this issue in 4.2.1 and the project size has been the same.

[r-eckert]

I have the same problem in my project. After a little profiling it seems like most of the time during the freeze is spent in ResourceLoader::get_resource_type called by FileSystemDock::_get_entry_script_icon.
This seems to be related to a recent change in how resource icons work #77932.
Since FileSystemDock is updated for every single script file in the project this adds up to a 25 second freeze in my case.

For testing I replaced the call to ResourceLoader::get_resource_type with a simple check of the file extension and the freeze went back to just 3 seconds but I'm not familiar enough with Godot's internals yet to make a proper fix.

[J-Ponzo]

Ok, that's weird. I just checked other versions and I reproduced with :
- v4.1.2.stable.mono.official [399c9dc]
- v4.2.stable.mono.official [46dc277]
- v4.2.1.stable.mono.official [b09f793]

I tested with 24GB + 32 empty scripts and the freeze lasts about 11s with each version.

I used a different computer this time. Which explains the differences in measurements with the table in the original message. But the issue is still there.

System info : Godot v4.2.1.stable.mono - Windows 10.0.22631 - Vulkan (Forward+) - dedicated NVIDIA GeForce RTX 3070 (NVIDIA; 31.0.15.5186) - 12th Gen Intel(R) Core(TM) i9-12900 (24 Threads).

Maybe this is not the same problem as yours ? @MatthiasBae @r-eckert

[MatthiasBae]

Ok, that's weird. I just checked other versions and I reproduced with : - v4.1.2.stable.mono.official [399c9dc] - v4.2.stable.mono.official [46dc277] - v4.2.1.stable.mono.official [b09f793]

I tested with 24GB + 32 empty scripts and the freeze lasts about 11s with each version.

I used a different computer this time. Which explains the differences in measurements with the table in the original message. But the issue is still there.

System info : Godot v4.2.1.stable.mono - Windows 10.0.22631 - Vulkan (Forward+) - dedicated NVIDIA GeForce RTX 3070 (NVIDIA; 31.0.15.5186) - 12th Gen Intel(R) Core(TM) i9-12900 (24 Threads).

Maybe this is not the same problem as yours ? @MatthiasBae @r-eckert

I also tested 4.2.1 again and yes.. I do not know if I just not recognized it but suddenly it also freezes

[J-Ponzo]

Ok, that's weird. I just checked other versions and I reproduced with : - v4.1.2.stable.mono.official [399c9dc] - v4.2.stable.mono.official [46dc277] - v4.2.1.stable.mono.official [b09f793]
I tested with 24GB + 32 empty scripts and the freeze lasts about 11s with each version.
I used a different computer this time. Which explains the differences in measurements with the table in the original message. But the issue is still there.
System info : Godot v4.2.1.stable.mono - Windows 10.0.22631 - Vulkan (Forward+) - dedicated NVIDIA GeForce RTX 3070 (NVIDIA; 31.0.15.5186) - 12th Gen Intel(R) Core(TM) i9-12900 (24 Threads).
Maybe this is not the same problem as yours ? @MatthiasBae @r-eckert

I also tested 4.2.1 again and yes.. I do not know if I just not recognized it but suddenly it also freezes

I think I understood what happened and it is very interesting (I updated the reproduction steps of the main message accordingly).

Actually, when the godot loading screen disappears, the project is not really loaded. If you check the FileSystem dock at project startup, it is empty. Your project files only appear after a while (the delay seems correlated to the size of the project).

If you build before the project is completely loaded (i.e. the FileSystem dock is still empty), the freeze is not present. But if you wait enough for the project to be fully loaded, you will reproduce.

[J-Ponzo]

I just reproduced with a build from source (v4.3.beta.mono.custom_build [be56cab]).

@r-eckert I tried your solution but it did not solve the problem for me. Maybe I misunderstood what you did or maybe I did it wrong. Here is my FileSystemDock::_get_entry_script_icon method after changes :

String FileSystemDock::_get_entry_script_icon(const EditorFileSystemDirectory *p_dir, int p_file) {
	const PackedStringArray &deps = p_dir->get_file_deps(p_file);
	if (deps.is_empty()) {
		return String();
	}

	const String &script_path = deps[0]; // Assuming the first dependency is a script.
	const String extention = script_path.get_extension();
	if (script_path.is_empty() || !(extention == "cs" || extention == "gd")) {
	//if (script_path.is_empty() || !ClassDB::is_parent_class(ResourceLoader::get_resource_type(script_path), SNAME("Script"))) {
		return String();
	}

	String *cached = icon_cache.getptr(script_path);
	if (cached) {
		return *cached;
	}

	HashMap<String, String>::Iterator I;
	int script_file;
	EditorFileSystemDirectory *efsd = EditorFileSystem::get_singleton()->find_file(script_path, &script_file);
	if (efsd) {
		I = icon_cache.insert(script_path, efsd->get_file_script_class_icon_path(script_file));
	} else {
		I = icon_cache.insert(script_path, String());
	}
	return I->value;
}

Is that what you did ?

I also tried to do some profiling, but in my case the cost of FileSystemDock::_get_entry_script_icon very low.

Here's a screen of the most expensive functions of my profiling session :

And here is a .csv export of the full session :
capture.csv

I hope it could help

Edit : For the sake of precision, I would like to point out that I started the profiling session just before clicking on rebuild, and that I stopped it just after regaining focus. The test project was 12GB-Data and 32-Scripts (11.75s freeze according to the table in the problem description)

[r-eckert]

It seems like we are looking at two different problems then. I had trouble getting that huge test project to work consistently so I simply assumed that we had the same problem.
Since your problem also happens in 4.2 it can't be related to that icon code. Guess I'll have to open another issue for that.

I tried to profile your test project but with that size it takes forever to import everything and after each editor restart it takes a quite while to load. Maybe a more minimal test project would help get more people on board to test this, though I do understand that the size is an important factor in the problem itself.

Sorry for hijacking your issue with an unrelated one.

[J-Ponzo]

Yes, I understand this is not easy to handle. I had a very hard time investigating on it in the first place, and even after figuring out how to reproduce on a separate project, gathering accurate information to create the issue was a nightmare.

I tried to guess the minimal size so that somebody with a really powerful CPU can still reproduce. But you are right, maybe it's not the best strategy, I'll try to find something more manageable.

That said, 32GB is relatively small in comparison to the average production I had the opportunity to work on. Our projects are more about 100-200GB and we are just a small indie studio. In my opinion, the fact a 32GB project puts Godot on his knees after each editor restart is a problem too. Unlike this freeze thing, I can live with a long startup time so I did not make an issue for that. But I think I should. The underlying causes may be related, and since this is not a C# only problem, it could get more people involved.

Thank you for trying to profile the MRP and don't be sorry. It was a promising lead you shared. :)

[J-Ponzo]

In order to ease the testing process, I added multiple branches to the MRP repo corresponding to different amount of data / number of scripts.

The main branch (the one you retrieve when following the reproduction steps) is now 4GB of data / 128 empty scripts which should be way more convenient than the 32GB of data / 32 empty scripts of the original MRP.

It still reproduces the freeze even with a good CPU (~9s on 12th Gen Intel(R) Core(TM) i9-12900 (24 Threads)). But if you have a supercomputer, feel free to increase the challenge switching to another branch.

Edit : Some metrics with the new MRP :

• zip size is 4.44 GB
• full startup (until the filesystem dock displays the project files) : 2min09s on the 1st launch (14s on the next launches)
• shutdown : 9s on the 1st launch (5s on the next launches)

[IIIZorgIII]

I made the test 3 times, with this config :

Godot v4.2.2.stable.mono - Windows 10.0.19045 - Vulkan (Forward+) - dedicated NVIDIA GeForce GTX 1080 (NVIDIA; 31.0.15.3623) - Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz (8 Threads)

For the first test:
I launched the build before the project was completely open and as a result, I first had 4m15s of freeze and Godot crashed, then I restarted the build but this time they were almost instantaneous. (the project was not completely open)
After a few seconds without touching anything I had a new freeze and a new crash after 4m20s

For the second test:
I waited for the project to open. When launching the build I had a first "freeze" of 1m40s (the screen was not frozen it seems to me) then the freeze of 25s with the white screen and Godot (which does not respond).

For the third test (done this evening):
20s of freeze

Edit: I forgot to mention this but opening and closing the project is terrible for my computer (maybe over an hour to fully open it and still force closed when exiting godot.

[J-Ponzo]

Ok, I was wrong from the very start. The weight of the project has nothing to do with the freeze time. Only the number of files seems relevant. For the sake of clarity, I close this issue and open another one more accurate (and more simple) : #92485

Sorry for the misleading information (and the time you spent with this unmanageable MRP I made you to test).