Managing native dependencies for .NET components

[mthalman]

Managing native dependencies for .NET components

Problem statement

The .NET product relies on the native libraries within the operating system for critical functionality. Each of the three supported operating systems (Linux, macOS, Windows) has its own set of dependencies required by .NET, as well as optional dependencies for other scenarios such as System.Drawing.Common's dependency on libgdiplus in Linux. Managing .NET's native dependencies is currently done in an ad hoc fashion with little to no formal process. This has lead to a variety of issues outlined below.

Community usage

While the primary set of native dependencies are documented, it doesn't provide a full accounting of all dependencies. Without that information, .NET developers are currently left to discover what dependencies are required by running a scenario and seeing what breaks. A few examples of this:

• System.DirectoryServices.Protocols - Linux support #36888 (comment)
• Using System.DirectoryServices on a Windows container with ASP.NET Core returns an error dotnet-docker#1767

These developers would benefit by being able to determine up front which native dependencies an application requires without actually running it.

Product consistency

Maintaining product consistency has become a challenge due to the breadth of product assets that need to keep track of native dependencies. As the product evolves, new dependencies are added, old dependencies may be removed, and existing dependencies may be modified to target a different version. This requires updates to the various product assets to reflect these changes. There isn't a defined process for this and no central location for tracking these changes.

If a dependency change does occur, it requires product contributors to make a series of changes in disparate dotnet repos to update the affected assets. These are often overlooked, leading to out-of date documentation (see dotnet/docs#18989) or other missing dependencies in downstream assets (see #68365). In addition, not all dependency changes necessitate the same asset updates. There may be a native dependency added for functionality not commonly used in which case it wouldn't be considered required or even recommended. But even in those cases, it's important for contributors to at least be aware of such changes so they can make the proper evaluation of its need.

Accounting

Without a holistic view of the .NET product's native dependency profile, there's not an efficient process to actually get that information. If a security incident were to occur for some Linux package, how easy would it be to know whether .NET assets were affected or that products built on .NET might be affected due to their usage of a specific .NET library? By having a full accounting of these dependencies, it can help eliminate guesswork and scrounging through code for such situations.

Affected product assets

These are the product assets — packages, Docker images, documentation — that are potentially affected by any dependency change:

• deb packages
• RPM packages
• Snap packages
• Docker images
• dotnet/core Linux pre-reqs doc
• .NET Installation guide docs

It is important that these assets be kept up-to-date as product changes are made. You'll notice that these are all downstream assets from the core product components. This requires a well-defined process in order to communicate dependency updates.

Scope

Operating system

Due to the pervasiveness of package managers in Linux distros, this need is most relevant for Linux packages but also applies more broadly to other operating systems like Windows. Windows has several SKUs — Windows Server, Windows Server Core, Nano Server, Windows IoT — each having varying levels of functionality, not necessarily encompassing all the native dependencies .NET may require. Here are several examples where Nano Server doesn't provide the set of native dependencies required by .NET Core that is provided by Windows Server Core:

• Using System.DirectoryServices on a Windows container with ASP.NET Core returns an error dotnet-docker#1767
• Unable to load DLL 'gdiplus.dll' when using Windows based docker images dotnet-docker#1098
• Unable to use HttpSys with ASP.NET Core 2.0 and Windows Server Nano aspnet/aspnet-docker#398

Ideally, a solution that describes .NET's dependencies would support descriptions across all supported operating systems: Linux, macOS, and Windows.

Servicing

.NET 5.0 is not the only version undergoing change with respect to dependencies. As long as 2.1 and 3.1 are supported, the package dependencies they have are an ever-changing landscape as new distro versions are released. This is specifically the case for packages like the ICU packages where an older version gets replaced by a newer one with a different package name (e.g. dotnet/dotnet-docker#614).

Potential solutions

What follows are a set of proposals that describe, at a high-level, approaches for solving the problems stated above. A more detailed design would need to be developed.

The central idea is to define a machine-readable metadata file that describes the set of native dependencies contained within each .NET component. How this content gets defined is open for debate. It could be manually managed, requiring that product contributors be diligent in maintaining it. Or perhaps it could be automatically generated through static analysis tools.

As part of the .NET engineering processes, transform logic could then be applied to this file to generate updated content for the various downstream assets (packages, Docker images, documentation). At the very least, having a single source of truth for dependencies within source control allows for a mechanism for affected parties to be notified of changes. Potentially, this transformation logic could even be automated to ensure all assets are automatically maintained.

From a customer standpoint, the minimum bar is having up-to-date and comprehensive documentation that describes the native dependencies. Having this generated directly from an authoritative source rather through ad hoc means would be beneficial in achieving that goal.

[MichaelSimons]

Another consumer that would benefit from this list is the end user documentation. Specifically the docs that list the prereqs for the various distros. As you can see this documentation is currently out of date.

The way I've been thinking about this is there would be a code artifact (e.g. json file) that defines the native dependencies. By defining them as a code artifact, it would hopefully be kept up to date as the code that defines the dependencies evolves. This code artifact would be captured and published during a build so that consumers can make use of it just like other built binaries.

Is this something we could work on for 5.0? I'd be willing to help out on this as I see this as something that will improve the .NET Core acquisition experience.

cc @dagood, @leecow, @dleeapho

[MichaelSimons]

Assigning to @mthalman to add more details on the problem space.

[mthalman]

I've updated the original issue posting to include a more comprehensive description of the problems being faced.

[mthalman]

cc @leecow, @richlander, @NikolaMilosavljevic, @dagood

[mthalman]

A new epic has been created in dotnet/core to replace this proposal: dotnet/core#5651.