Sporadic clang build failure due to 'llvm/CodeGen/GenVT.inc' file not found

[Zentrik]

Building for windows 32, 64 bit and aarch64 darwin by cross compiling and building the shared libraries I get the error sporadically (log at https://github.com/Zentrik/llvm_windows_tester/actions/runs/10747564963/job/29810161266, full build script described in #106859 (comment))

[02:35:10] ninja: job failed: /opt/bin/i686-w64-mingw32-libgfortran5-cxx11/i686-w64-mingw32-clang++ --target=i686-w64-mingw32 --sysroot=/opt/i686-w64-mingw32/i686-w64-mingw32/sys-root/ -DGTEST_HAS_RTTI=0 -D_DEBUG -D_FILE_OFFSET_BITS=64 -D_GLIBCXX_ASSERTIONS -D_LARGEFILE_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/workspace/build/tools/clang/tools/clang-fuzzer/handle-llvm -I/workspace/srcdir/llvm-project/clang/tools/clang-fuzzer/handle-llvm -I/workspace/srcdir/llvm-project/clang/include -I/workspace/build/tools/clang/include -I/workspace/build/include -I/workspace/srcdir/llvm-project/llvm/include -remap -D__USING_SJLJ_EXCEPTIONS__ -D__CRT__NO_INLINE -pthread -DMLIR_CAPI_ENABLE_WINDOWS_DLL_DECLSPEC -Dmlir_arm_sme_abi_stubs_EXPORTS -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -ffunction-sections -fdata-sections -fno-common -Woverloaded-virtual -Wno-nested-anon-types -O3 -DNDEBUG  -fno-exceptions -funwind-tables -fno-rtti -UNDEBUG -std=c++17 -MD -MT tools/clang/tools/clang-fuzzer/handle-llvm/CMakeFiles/obj.clangHandleLLVM.dir/handle_llvm.cpp.obj -MF tools/clang/tools/clang-fuzzer/handle-llvm/CMakeFiles/obj.clangHandleLLVM.dir/handle_llvm.cpp.obj.d -o tools/clang/tools/clang-fuzzer/handle-llvm/CMakeFiles/obj.clangHandleLLVM.dir/handle_llvm.cpp.obj -c /workspace/srcdir/llvm-project/clang/tools/clang-fuzzer/handle-llvm/handle_llvm.cpp
[02:35:10] clang-16: warning: argument unused during compilation: '-remap' [-Wunused-command-line-argument]
[02:35:10] In file included from /workspace/srcdir/llvm-project/clang/tools/clang-fuzzer/handle-llvm/handle_llvm.cpp:38:
[02:35:10] In file included from /workspace/srcdir/llvm-project/llvm/include/llvm/Passes/PassBuilder.h:19:
[02:35:10] In file included from /workspace/srcdir/llvm-project/llvm/include/llvm/CodeGen/MachinePassManager.h:26:
[02:35:10] In file included from /workspace/srcdir/llvm-project/llvm/include/llvm/CodeGen/MachineFunction.h:26:
[02:35:10] In file included from /workspace/srcdir/llvm-project/llvm/include/llvm/CodeGen/MachineBasicBlock.h:21:
[02:35:10] In file included from /workspace/srcdir/llvm-project/llvm/include/llvm/CodeGen/MachineInstr.h:24:
[02:35:10] In file included from /workspace/srcdir/llvm-project/llvm/include/llvm/CodeGen/MachineMemOperand.h:22:
[02:35:10] In file included from /workspace/srcdir/llvm-project/llvm/include/llvm/CodeGenTypes/LowLevelType.h:30:
[02:35:10] /workspace/srcdir/llvm-project/llvm/include/llvm/CodeGenTypes/MachineValueType.h:45:10: fatal error: 'llvm/CodeGen/GenVT.inc' file not found
[02:35:10] #include "llvm/CodeGen/GenVT.inc"
[02:35:10]          ^~~~~~~~~~~~~~~~~~~~~~~~
[02:35:10] 1 error generated.

I also hit this on freebsd on the 19 rc3 branch, log at https://buildkite.com/julialang/yggdrasil/builds/12323#019126e9-ad77-4ad3-b860-c9fa1ceb61f1.
This seems to have started to occur more frequently after 59f8796.

I applied the following patch,

diff --git a/clang/tools/clang-fuzzer/handle-llvm/CMakeLists.txt b/clang/tools/clang-fuzzer/handle-llvm/CMakeLists.txt
index 9962f9850f54..22656145a34b 100644
--- a/clang/tools/clang-fuzzer/handle-llvm/CMakeLists.txt
+++ b/clang/tools/clang-fuzzer/handle-llvm/CMakeLists.txt
@@ -23,5 +23,6 @@ add_clang_library(clangHandleLLVM
   handle_llvm.cpp

   DEPENDS
+  vt_gen
   intrinsics_gen
   )

but then got the following error on the darwin build (log at https://github.com/Zentrik/llvm_windows_tester/actions/runs/10750386811/job/29816468075)

[10:34:36] ninja: job failed: /opt/bin/aarch64-apple-darwin20-libgfortran5-cxx11/aarch64-apple-darwin20-clang++ --target=arm64-apple-darwin20 --sysroot=/opt/aarch64-apple-darwin20/aarch64-apple-darwin20/sys-root -DGTEST_HAS_RTTI=0 -D_DEBUG -D_GLIBCXX_ASSERTIONS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/workspace/build/tools/clang/lib/CodeGen -I/workspace/srcdir/llvm-project/clang/lib/CodeGen -I/workspace/srcdir/llvm-project/clang/include -I/workspace/build/tools/clang/include -I/workspace/build/include -I/workspace/srcdir/llvm-project/llvm/include -stdlib=libc++ -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -fno-common -Woverloaded-virtual -Wno-nested-anon-types -O3 -DNDEBUG -mmacosx-version-min=11.0  -fno-exceptions -funwind-tables -fno-rtti -UNDEBUG -std=c++17 -MD -MT tools/clang/lib/CodeGen/CMakeFiles/obj.clangCodeGen.dir/CodeGenAction.cpp.o -MF tools/clang/lib/CodeGen/CMakeFiles/obj.clangCodeGen.dir/CodeGenAction.cpp.o.d -o tools/clang/lib/CodeGen/CMakeFiles/obj.clangCodeGen.dir/CodeGenAction.cpp.o -c /workspace/srcdir/llvm-project/clang/lib/CodeGen/CodeGenAction.cpp
[10:34:36] In file included from /workspace/srcdir/llvm-project/clang/lib/CodeGen/CodeGenAction.cpp:35:
[10:34:36] In file included from /workspace/srcdir/llvm-project/llvm/include/llvm/CodeGen/MachineOptimizationRemarkEmitter.h:18:
[10:34:36] In file included from /workspace/srcdir/llvm-project/llvm/include/llvm/CodeGen/MachineFunctionPass.h:21:
[10:34:36] In file included from /workspace/srcdir/llvm-project/llvm/include/llvm/CodeGen/MachineFunction.h:26:
[10:34:36] In file included from /workspace/srcdir/llvm-project/llvm/include/llvm/CodeGen/MachineBasicBlock.h:21:
[10:34:36] In file included from /workspace/srcdir/llvm-project/llvm/include/llvm/CodeGen/MachineInstr.h:24:
[10:34:36] In file included from /workspace/srcdir/llvm-project/llvm/include/llvm/CodeGen/MachineMemOperand.h:22:
[10:34:36] In file included from /workspace/srcdir/llvm-project/llvm/include/llvm/CodeGenTypes/LowLevelType.h:30:
[10:34:36] /workspace/srcdir/llvm-project/llvm/include/llvm/CodeGenTypes/MachineValueType.h:45:10: fatal error: 'llvm/CodeGen/GenVT.inc' file not found
[10:34:36] #include "llvm/CodeGen/GenVT.inc"
[10:34:36]          ^~~~~~~~~~~~~~~~~~~~~~~~

Let me know if I'm missing any information that would be useful.

[Zentrik]

Also hit this when compiling llvm only

ninja: job failed: /opt/bin/x86_64-w64-mingw32-libgfortran5-cxx11/x86_64-w64-mingw32-clang++ --target=x86_64-w64-mingw32 --sysroot=/opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/ -DGTEST_HAS_RTTI=0 -D_DEBUG -D_FILE_OFFSET_BITS=64 -D_GLIBCXX_ASSERTIONS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/workspace/build/lib/Target/NVPTX/MCTargetDesc -I/workspace/srcdir/llvm-project/llvm/lib/Target/NVPTX/MCTargetDesc -I/workspace/srcdir/llvm-project/llvm/lib/Target/NVPTX -I/workspace/build/lib/Target/NVPTX -I/workspace/build/include -I/workspace/srcdir/llvm-project/llvm/include -isystem /workspace/destdir/include -remap -D__USING_SJLJ_EXCEPTIONS__ -D__CRT__NO_INLINE -pthread -DMLIR_CAPI_ENABLE_WINDOWS_DLL_DECLSPEC -Dmlir_arm_sme_abi_stubs_EXPORTS -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -ffunction-sections -fdata-sections -O3 -DNDEBUG -fvisibility=hidden  -fno-exceptions -funwind-tables -fno-rtti -UNDEBUG -std=c++17 -MD -MT lib/Target/NVPTX/MCTargetDesc/CMakeFiles/LLVMNVPTXDesc.dir/NVPTXInstPrinter.cpp.obj -MF lib/Target/NVPTX/MCTargetDesc/CMakeFiles/LLVMNVPTXDesc.dir/NVPTXInstPrinter.cpp.obj.d -o lib/Target/NVPTX/MCTargetDesc/CMakeFiles/LLVMNVPTXDesc.dir/NVPTXInstPrinter.cpp.obj -c /workspace/srcdir/llvm-project/llvm/lib/Target/NVPTX/MCTargetDesc/NVPTXInstPrinter.cpp
clang-16: warning: argument unused during compilation: '-remap' [-Wunused-command-line-argument]
In file included from /workspace/srcdir/llvm-project/llvm/lib/Target/NVPTX/MCTargetDesc/NVPTXInstPrinter.cpp:16:
In file included from /workspace/srcdir/llvm-project/llvm/lib/Target/NVPTX/NVPTXUtilities.h:17:
In file included from /workspace/srcdir/llvm-project/llvm/include/llvm/CodeGen/ValueTypes.h:18:
/workspace/srcdir/llvm-project/llvm/include/llvm/CodeGenTypes/MachineValueType.h:44:10: fatal error: 'llvm/CodeGen/GenVT.inc' file not found
#include "llvm/CodeGen/GenVT.inc"
         ^~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
clang-16: warning: argument unused during compilation: '-remap' [-Wunused-command-line-argument]
ninja: subcommand failed
---> ninja -j${nproc} -vv
---> ninja -j${nproc} -vv

[giordano]

We're seeing these errors in all sort of operating systems and architectures, not just those mentioned above. Note that we build with a large-ish parallelism, random failures like this usually suggests that some dependencies aren't tracked correctly in the build system.

[h-vetinari]

Seeing the same issue on linux and osx when building from 70f326c

[h-vetinari]

A fix for the llvm-only portion of this issue is in #120661

[chapuni]

How about LLVMCodeGenTypes?