CI build failure: Bash exited with code '1' failure - Could NOT find ZLIB

[ivanpovazan]

Seen when running runtime-extra-platforms on linux-arm64 Release AllSubsets_Mono_LLVMFullAot_RuntimeTests llvmfullaot

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=274202
Build error leg or test failing: Build / linux-arm64 Release AllSubsets_Mono_LLVMFullAot_RuntimeTests llvmfullaot / Build Mono LLVM AOT cross compiler
Pull request: #86266

Error Message

Fill the error message using known issues guidance.

{
  "ErrorMessage": "Could NOT find ZLIB (missing: ZLIB_LIBRARY ZLIB_INCLUDE_DIR)",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Report

Build	Definition	Step Name	Console log	Pull Request
353096	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
353048	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log	#89467
352690	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log	#89431
2230039	dotnet-runtime	Build product	Log
2229927	dotnet-runtime	Build product	Log
352326	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
2229668	dotnet-runtime	Build product	Log
2229580	dotnet-runtime	Build product	Log
2229437	dotnet-runtime	Build product	Log
351937	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
351866	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
351758	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log	#89223
351723	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
2229082	dotnet-runtime	Build product	Log
351065	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
351063	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log	#89421
351019	dotnet/runtime	Build product	Log
2228885	dotnet-runtime	Build product	Log
2228802	dotnet-runtime	Build product	Log
2228665	dotnet-runtime	Build product	Log
350495	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
350271	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
2228258	dotnet-runtime	Build product	Log
349510	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
349464	dotnet/runtime	Build product	Log
2227956	dotnet-runtime	Build product	Log
2227939	dotnet-runtime	Build product	Log
349328	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
2227837	dotnet-runtime	Build product	Log
349156	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
2227737	dotnet-runtime	Build product	Log
349144	dotnet/runtime	Build product	Log
2227669	dotnet-runtime	Build product	Log
2227649	dotnet-runtime	Build product	Log
349045	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
2227542	dotnet-runtime	Build product	Log
348654	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
2227439	dotnet-runtime	Build product	Log
348644	dotnet/runtime	Build product	Log
2227366	dotnet-runtime	Build product	Log
2227269	dotnet-runtime	Build product	Log
348291	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
347426	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
347390	dotnet/runtime	Build product	Log
2226549	dotnet-runtime	Build product	Log
2226537	dotnet-runtime	Build product	Log
346915	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
346860	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
2226244	dotnet-runtime	Build product	Log
346479	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
346114	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
346085	dotnet/runtime	Build product	Log
2225817	dotnet-runtime	Build product	Log
2225791	dotnet-runtime	Build product	Log
345752	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
345634	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
345419	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
2225456	dotnet-runtime	Build product	Log
344997	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
344698	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
344678	dotnet/runtime	Build product	Log
344129	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
343954	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log	#88723
343204	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
343172	dotnet/runtime	Build product	Log
2224068	dotnet-runtime	Build product	Log
2224038	dotnet-runtime	Build product	Log
342556	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
2223560	dotnet-runtime	Build product	Log
2223513	dotnet-runtime	Build product	Log
2223490	dotnet-runtime	Build product	Log
2223435	dotnet-runtime	Build product	Log
341622	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
2223279	dotnet-runtime	Build product	Log
2223226	dotnet-runtime	Build product	Log
341572	dotnet/runtime	Build product	Log
2223186	dotnet-runtime	Build product	Log
2223134	dotnet-runtime	Build product	Log
2223121	dotnet-runtime	Build product	Log
2223110	dotnet-runtime	Build product	Log
2223079	dotnet-runtime	Build product	Log
2223049	dotnet-runtime	Build product	Log
2223043	dotnet-runtime	Build product	Log
341305	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
2223025	dotnet-runtime	Build product	Log
2223012	dotnet-runtime	Build product	Log
2222998	dotnet-runtime	Build product	Log
2222960	dotnet-runtime	Build product	Log
2222944	dotnet-runtime	Build product	Log
2222915	dotnet-runtime	Build product	Log
2222903	dotnet-runtime	Build product	Log
341090	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
341070	dotnet/runtime	Build product	Log
2222835	dotnet-runtime	Build product	Log
340960	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
2222775	dotnet-runtime	Build product	Log
2222765	dotnet-runtime	Build product	Log
340683	dotnet/runtime	Build Mono LLVM AOT cross compiler	Log
340670	dotnet/runtime	Build product	Log
2222634	dotnet-runtime	Build product	Log
Displaying 100 of 119 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	119

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323893
Error message validated: Could NOT find ZLIB (missing: ZLIB_LIBRARY ZLIB_INCLUDE_DIR)
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:28:40 PM UTC

[ghost]

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Seen when running runtime-extra-platforms on linux-arm64 Release AllSubsets_Mono_LLVMFullAot_RuntimeTests llvmfullaot

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=274202
Build error leg or test failing: Build / linux-arm64 Release AllSubsets_Mono_LLVMFullAot_RuntimeTests llvmfullaot / Build Mono LLVM AOT cross compiler
Pull request: #86266

Error Message

Fill the error message using known issues guidance.

{
  "ErrorMessage": "Could NOT find ZLIB (missing: ZLIB_LIBRARY ZLIB_INCLUDE_DIR)",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Author:	ivanpovazan
Assignees:	-
Labels:	blocking-clean-ci, area-Infrastructure-mono, Known Build Error
Milestone:	8.0.0

[directhex]

Nothing to do with ZLIB, that's just CMake lying to you. The true build error here is clang-16: error: unable to execute command: Executable "ld" doesn't exist!, because we aren't passing a value for -fuse-ld=lld via LDFLAGS because we aren't passing a value for -DCMAKE_TOOLCHAIN_FILE= in this specific scenario. Read on for the full analysis.

Once upon a time, we had two code paths in mono.proj - Target Name="BuildMonoRuntime" which compiled a runtime against $(TargetOS)-$(TargetArchitecture), and Target Name="BuildMonoCross" which compiled a cross-compiler (runtime) against $(HostOS)-$(BuildArchitecture) with the codegen targeting $(TargetOS)-$(TargetArchitecture).

So, for example, build for android-arm on an x64 Mac and you'd get BuildMonoRuntime generating an android-arm libmonosgen.so and BuildMonoCross generating an osx-x64 mono-aot-cross. Strict 1:1 mapping (i.e. no consistent way to produce multiple cross-compilers for the same target, or any cross-compiler for a different target than the host)

In 2021 I wrote a new build system, intended to allow a many:many relationship, and part of that was changing BuildMonoCross to deal with the difference between "generate code for RID" and "compile code for RID". I kept the old method working, in order to not break anyone's workflow, but it became deprecated & basically abandoned, other than for these LLVM AOT lanes.

Fast forward to recently, when we started building everything possible with CBL-Mariner. One of the changes forced by Mariner is treating all compiles as cross-compiles - instead of "compile for the host which is always x64 or cross-compile for the OS in /crossrootfs", all architectures are cross-compiles. This is great! However, it runs into a problem with the legacy-BuildMonoCross lanes like this one, because we cannot BuildMonoRuntime using an arm64 crossrootfs and BuildMonoCross using an x64 crossrootfs - we have no Docker images containing both crossroots at once.

This is the patch to enable cross-compiler mode for legacy BuildMonoCross:

diff --git a/src/mono/mono.proj b/src/mono/mono.proj
index de64d7d3440..c187b348c1a 100644
--- a/src/mono/mono.proj
+++ b/src/mono/mono.proj
@@ -665,6 +665,12 @@
   <!-- Build AOT cross compiler (if available) -->
   <Target Name="BuildMonoCross" Condition="'$(BuildMonoAOTCrossCompiler)' == 'true'" DependsOnTargets="BuildMonoRuntime">
 
+    <!-- Legacy cross-compiler mode (build cross compiler for host CPU/OS) -->
+    <PropertyGroup Condition="'$(RealTargetOS)' == '' and '$(RealTargetArchitecture)' == ''">
+      <RealTargetOS>$(HostOS)</RealTargetOS>
+      <RealTargetArchitecture>$(BuildArchitecture)</RealTargetArchitecture>
+    </PropertyGroup>
+
     <!-- iOS/tvOS specific options -->
     <PropertyGroup Condition="'$(TargetstvOS)' == 'true' or '$(TargetsiOS)' == 'true'">
       <!-- FIXME: Disable for simulator -->

But it won't work. Therefore we have three choices:

1. Don't use Mariner for this lane, revert to whatever we were using before
2. Create a new multi-targeting Mariner docker image, and use that (passing an overridden crossrootfs property where needed to the appropriate targets, since we assume we use the same property throughout)
3. Split this job in two - an offset generation job using an arm64 crossrootfs, and the AOT compiler job using an x64 crossrootfs (this is basically the model for official builds, where we generate offsets in eng/pipelines/mono/templates/generate-offsets.yml and consume them in eng/pipelines/common/global-build-job.yml using /p:MonoCrossAOTTargetOS=OSNAMEFROMMONOAOTCROSSDOTPROJ /p:SkipMonoCrossJitConfigure=true /p:BuildMonoAOTCrossCompilerOnly=true

[kotlarmilos]

Thanks for the explanation. It sounds to me that the 3rd option is aligned with the official builds and may be a path we want to consider. If that makes sense, I can try to split this job into offset generation and the AOT compilation.

[SamMonoRT]

@directhex @kotlarmilos @steveisok - Milos reminded me of this while checking on another PR We need to move forward with this, given a lack of desktop full aot llvm lane. Two things we should do asap - 1. Disable this lane (even in extra-platforms) prior to RC1 cutoff. 2. Work on option 3 suggested above for 9.0 (essentially at the earliest once RC1 is out of the way next week). For this we will need some help understanding the cross-compiler/official build.

Moving this to 9.0, but we need to disable the lane prior to rc1 cutoff for cleaner CI to remove the blocking green ci tag

[steveisok]

Moving this to 9.0, but we need to disable the lane prior to rc1 cutoff for cleaner CI to remove the blocking green ci tag

@directhex can you please do this before Monday's cutoff?