Subsequent builds are slower on Windows + MSVC due to linking

[Calinou]

Godot version

4.1.dev 72b5932

System information

Windows 11 22H2 - MSVC 2022 - Intel Core i9-13900K, OS and Godot repository on a fast M.2 SSD

Issue description

Subsequent builds are slower on Windows + MSVC due to the linking step performing incremental linking, but it's counterproductive for Godot.

This is on MSVC 17.4.3 (version from early 2023):

1st build after cleaning all build files
[Time elapsed: 00:00:57.752]

2nd build after modifying scene/main/node.h
[Time elapsed: 00:01:30.433]

3rd build after modifying scene/main/node.h again
[Time elapsed: 00:01:27.513]

4th build after modifying scene/main/node.h yet again
[Time elapsed: 00:01:31.877]

5th build after removing bin/ folder and modifying scene/main/node.h another time
[Time elapsed: 00:00:26.683]

6th build after removing bin/, but not changing any other file
[Time elapsed: 00:00:05.874]

7th build after removing bin, not changing any other file and using progress=no (Windows console I/O is slow :))
[Time elapsed: 00:00:05.559]

Updating to MSVC 17.6.2 (latest version available as of June 2023) improves the situation somewhat, but it's still much slower until you remove bin/:

1st clean build
[Time elapsed: 00:00:51.550]

2nd build after modifying scene/main/node.h
[Time elapsed: 00:00:54.424]

3rd build after modifying scene/main/node.h again
[Time elapsed: 00:00:54.931]

4th build after removing bin and modifying scene/main/node.h again
[Time elapsed: 00:00:23.995]

5th build after removing bin, but not changing any other file
[Time elapsed: 00:00:05.793]

This behavior does not occur on Linux with GCC + mold. Link times are consistent regardless of whether the bin/ folder was previously erased. Removing all **/*.a files also does not impact total build time.

Steps to reproduce

Use git clean -dfxi to remove all build files before compiling (only the first time).

• Using MSVC 2022 on Windows, perform a clean build of Godot using it: scons dev_build=yes scu_build=dev.
• Modify scene/main/node.h to add a line break at the beginning of the file.
• Build again and compare the build time.
• Modify scene/main/node.h again to add a line break at the beginning of the file.
• Build again and compare the build time for a second time
• Remove the bin/ folder, modify scene/main/node.h yet another time then build again. Compare the build time.

Minimal reproduction project

N/A

[YuriSizov]

For a reference, on my Windows 11 laptop (22H2 as well) with MS Visual Studio 2022 it doesn't behave like that:

Fresh checkout (a little slow, forgot to plug in until partway through)
[Time elapsed: 00:03:47.378]

1 new line 🧛
[Time elapsed: 00:01:39.496]

2 new lines 🧛🧛
[Time elapsed: 00:01:58.228]

3 new lines 🧛🧛🧛
[Time elapsed: 00:01:33.184]

4 new lines 🧛🧛🧛🧛
[Time elapsed: 00:01:34.454]

Windows 11 (22H2, 22621.1702)
Intel Core i7-12700H, 32 GB RAM
MSVC 14.36.32532, VS 2022 (17.6.2)
Fresh checkout with no prior builds

---

I am pretty sure that this is the same behavior that I have experienced before, on VS 2019 and Windows 10, but there may have been other factors affecting the linking time and resetting it with subsequent builds. I didn't do a clean experiment like that (but I can try).

[akien-mga]

I have the same issue as Calinou using VS 2019, as I documented here: #77170 (comment)

Predator Helios 500
AMD Ryzen 7 2700 8-Core Processor, 3200 MHz, 16 GB RAM
Windows 10 Build 19044
VS 2019 16.11.26 / MSVC 14.2
Python 3.11.3 / SCons 4.5.2 / calling from git-bash.exe

When using SCU:

1st build after cleaning all build files
3m22.175s

2nd build after modifying scene/main/node.h
2m14.719s

3rd build after modifying scene/main/node.h again
2m9.989s

4th build after removing bin and modifying scene/main/node.h again
1m14.995s

5th build after removing bin again and modifying scene/main/node.h again
1m15.605s

When not using SCU:

1st build after cleaning all build files
6m42.166s

2nd build after modifying scene/main/node.h
9m17.832s

3rd build after modifying scene/main/node.h again
12m56.707s

4th build after removing bin and modifying scene/main/node.h again
3m27.323s

So my results with SCU are consistent with Calinou's.

Without SCU, the 3rd build (second incremental rebuild) is even slower than the second. I'd need to try again to confirm, and see if yet another incremental rebuild makes it even worse.

[akien-mga]

I recently removed VS 2019 and installed VS 2022. Testing again to see if the bug is still reproducible. Seems to be for me.

Not using SCU, building with scons dev_build=yes dev_mode=yes.
Adding the size of relevant files in the bin folder at each step:

• Final .exe file
• .ilk file used for the incremental linking
• .pdb file with debug symbols

1st build after cleaning all build files
7m20.413s
.exe file: 227M
.ilk file: 719M
.pdb file: 562M

2nd build after modifying scene/main/node.h
8m27.560s
.exe file: 227M
.ilk file: 1.0G during linking, 1.0G after
.pdb file: 900M

3rd build after modifying scene/main/node.h again
12m32.556s
.exe file: 227M
.ilk file: 2.0G during linking, 1.3G after
.pdb file: 958M
(roughly 9m30s spent linking)

4th build after removing bin and modifying scene/main/node.h again
3m18.172s
.exe file: 227M
.ilk file: 719M
.pdb file: 562M

I don't have in depth knowledge of how MSVC incremental linking and debug symbols work, but here's my hypothesis.
The .ilk file likely contains both past and present symbols, and gets used during the incremental linking step (that's where "Incremental LinKing" extension comes from). During each subsequent link, it increases size by including all symbols again (grows from 700M to 1.0G during first incremental relink, then up to 2.0G during the second incremental link before settling down on 1.3G), and spends an inordinate amount of doing a deduplication of those symbols to remove the ones which are duplicates. And it seems to keep old symbols, leading to increasing the final file of the file each time (700M after first link, 1.0G after first incremental relink, 1.3G after the second). Those 300M are probably the symbols of all the files affected by the change in node.h, and old symbols aren't discarded.

The .pdb file also increases in size so it likely also keeps some old symbols too. I've heard multiple reports over years of .pdb file growing exponentially.

My analysis might be a bit off, but anyway it shows that incremental linking with MSVC is just broken.

Given that a "clean" link when bin/ is deleted takes mere seconds, I see no reason to bother with it, and we can disable it with /INCREMENTAL:NO: https://learn.microsoft.com/en-us/cpp/build/reference/incremental-link-incrementally?view=msvc-170

I'll try that and report back.

[akien-mga]

Not using SCU, building with scons dev_build=yes dev_mode=yes (so comparable with times and sizes in previous comment), after adding /INCREMENTAL:NO to LINKFLAGS:

1st build after cleaning all build files
6m40.515s
.exe file: 185M
.ilk file: n/a
.pdb file: 538M

2nd build after modifying scene/main/node.h
3m15.348s
.exe file: 185M
.ilk file: n/a
.pdb file: 543M

3rd build after modifying scene/main/node.h again
3m15.245s
.exe file: 185M
.ilk file: n/a
.pdb file: 544M

4th build after removing bin and modifying scene/main/node.h again
3m15.180s
.exe file: 185M
.ilk file: n/a
.pdb file: 538M

For each build the total linking time seems to be around 10 to 12s on that machine for a dev build. Definitely not worth dealing with the bogus incremental linking.

The resulting .exe and .pdb files are smaller, and the .pdb file no longer increments size crazily (though it seems to still increment slowly after each build, though that might be a coincidence on these 3 steps).