Move to zlib-ng for better CPU intrinsics support (ARM64 performance)

[waf]

This was originally researched for inclusion in .NET 7, but then autoclosed due to inactivity. I'm making a new open issue to track this. There is also a closed PR #61883 with a great amount of context and discussion.

The .NET Runtime repo currently maintains two external native libraries for zlib:

• zlib-intel - a version of zlib with optimizations for x86 and x64.
• zlib - the original project used as fallback for other architectures

This means that ARM64 is not optimized, despite its increasing importance. The zlib-ng project has the optimizations of zlib-intel, as well as ARM64 optimizations.

From zlib-ng's README:

• Zlib compatible API with support for dual-linking
• Modernized native API based on zlib API for ease of porting
• Modern C11 syntax and a clean code layout
• Deflate medium and quick algorithms based on Intel’s zlib fork
• Support for CPU intrinsics when available

I believe we could remove zlib and zlib-intel and replace them with the newer zlib-ng, which includes a superset of the optimizations in zlib and zlib-intel.

Most of the work comes down to making dotnet's CMake configuration call zlib-ng's CMake configuration (perhaps via add_subdirectory). We cannot ignore zlib-ng's CMake configuration and build the source files directly, as zlib-ng's CMake configuration is responsible for enabling the optimizations during build. There is discussion around this in the PR linked above (#61883 (comment)).

[ghost]

Tagging subscribers to this area: @dotnet/area-system-io-compression
See info in area-owners.md if you want to be subscribed.

Issue Details

---

This was originally researched for inclusion in .NET 7, but then autoclosed due to inactivity. I'm making a new open issue to track this. There is also a closed PR #61883 with a great amount of context and discussion.

The .NET Runtime repo currently maintains two external native libraries for zlib:

• zlib-intel - a version of zlib with optimizations for x86 and x64.
• zlib - the original project used as fallback for other architectures

This means that ARM64 is not optimized, despite its increasing importance. The zlib-ng project has the optimizations of zlib-intel, as well as ARM64 optimizations.

From zlib-ng's README:

• Zlib compatible API with support for dual-linking
• Modernized native API based on zlib API for ease of porting
• Modern C11 syntax and a clean code layout
• Deflate medium and quick algorithms based on Intel’s zlib fork
• Support for CPU intrinsics when available

I believe we could remove zlib and zlib-intel and replace them with the newer zlib-ng, which includes a superset of the optimizations in zlib and zlib-intel.

Most of the work comes down to making dotnet's CMake configuration call zlib-ng's CMake configuration (perhaps via add_subdirectory). We cannot ignore zlib-ng's CMake configuration and build the source files directly, as zlib-ng's CMake configuration is responsible enabling the optimizations during build. There is discussion around this in the PR linked above (#61883).

Author:	waf
Assignees:	-
Labels:	area-System.IO.Compression, untriaged
Milestone:	-

[MichalPetryka]

Does the runtime static link zlib or does it use the system one on Linux platforms?

[prayaas-a]

@dotnet/area-system-io-compression any updates on this? At the moment, compression on Arm64 is limited to single core and takes 10 times longer in some cases than on otherwise similarly performing x86 hardware... This is really painful for PowerShell users as well.

[waf]

Closing this; it's now available in .NET 9 Preview 7 (release notes)! Work was tracked in #101465.