Manage .NET's platform dependencies

[mthalman]

Situation

There are a set of .NET-related customer workflows that require the use of platform libraries contained within the operating system. These are divided into two categories:

• Execution of a .NET process: Runtime components defer some of their logic to platform libraries. Examples include ICU libraries for globalization support, GNU C Library, and others.
• Building .NET from source code: The toolchain required to build the .NET source consists of platform dependency prerequisites on the build machine. Examples include CMake, curl, Python, and others.

Clearly, these platform libraries need to be installed in order to execute these workflows. It would be a reasonable expectation for developers to manage this set of dependencies on their own if the set was small, static, and well-documented. The problem is that these dependencies vary based on usage, they change as the product and the dependent ecosystem evolve, and the documentation is often out-of-date with the product [1]. This leaves the community with few options other than an iterative and error-prone approach of attempting to execute their workflow and finding what breaks due to missing dependencies [2]. Even worse is when there are cases where the community has discovered incompatible dependencies in environments that are supported [3].

Objective

We need to provide a better user experience for those developers that continually encounter issues of missing platform dependencies in their systems. By providing .NET developers access to self-service tools that consume an up-to-date and comprehensive database of platform dependency information, we can help these developers to accurately configure their systems.

We also need systems in place to guard against changes to platform dependencies that either break workflows or cause inconsistencies in product assets.

While these experiences are distinct, there is enough commonality in the requirements that allows us to achieve a single comprehensive solution that addresses both of them. This can be done by producing a layered set of multi-purpose deliverables.

Scope

In this context, platform dependency refers to a pre-existing artifact in the operating environment that satisfies the usage of a .NET component. For example, in order to run a .NET application in a Linux environment, the GNU C Library must exist. Thus, dependencies like NuGet packages and any assets contained in those packages (managed, native, or otherwise), are outside the scope of this work. The concern described here is solely focused on what the operating environment must be pre-configured to contain in order to operate on .NET scenarios.

Execution

Phase 1

The first priority is to get an accurate accounting of what platform dependencies the runtime and build toolchain workflows have [4]. This information will be encoded in a machine-readable format to provide an accurate picture of the platform dependencies across all supported platforms and versions. This provides the basis of data upon which all other phases rely. This is a useful but rudimentary start to the implementation and allows manual consumption of this data.

Phase 2

This phase begins the process of automated consumption of the platform dependency data in order to keep other product assets up-to-date [5]. This will consist of developing a client library for reading the data. This will be followed by tooling that will consume the data in the following product areas:

• dotnet/core - Auto-generate the dependency info in the release notes (We need a tool to generate markdown for dependency information #7453).
• dotnet/dotnet-docker - Ensure that the Linux packages installed by Dockerfiles are up-to-date (Add validation of packages installed by the Dockerfiles so that they align with the product dotnet-docker#246).
• dotnet/docs - Ensure that the documentation for dependencies accurately reflects the modeled dependencies for each platform (Keep Linux dependencies up-to-date in docs docs#22967).

Phase 3

Next, we will assist .NET contributors by alerting them when changes have been made to the platform dependencies. This can be implemented in a crawl-walk-run fashion as the system evolves with the following examples:

• Crawl: Define a written policy for how platform dependencies are to be managed for the product. Contributors are made aware of the policy.
• Walk: Bot automatically adds a "Platform Dependency Change" label to a PR when it detects a change. Contributor must manually update platform dependency model to reflect the change.
• Run: Updating of the platform dependency model is generated automatically by merged PRs. Stakeholders are automatically notified when dependency changes are proposed.

Phase 4

This is a continuation of the work begun in phase 2, expanding the scope of the target product areas to include the management of dependencies through tooling in the following product areas:

• snaps - Ensure that the stage packages referenced by the snap files are up-to-date.
• Linux packages - Ensure that the package dependencies referenced by the config files are up-to-date.
• .NET install scripts - Provide a way for install scripts to install missing dependencies.

Phase 5

Finally, we will assist .NET developers by providing them the ability to determine their application's platform dependencies through a website and tooling. The website would allow users to search and browse the database of dependencies with relevant links to the various package repository websites. A dotnet global tool will be made available that can analyze .NET projects to determine their platform dependencies.

Footnotes

1. While the primary set of runtime platform dependencies are documented, it doesn't provide a full accounting of all dependencies. Even worse, the documentation can become out-of-date compared to what the product actually requires (see dotnet/docs#18989).

2. Developers are often left to discover what dependencies are required by running their app and seeing what breaks (examples: dotnet/runtime#36888 (comment) and dotnet/dotnet-docker#1767). This is a tedious and error-prone process. This experience could be improved if the developer could know exactly what platform dependencies are required without even needing to run their app.

3. Source build toolchain dependencies are another set of dependencies that require careful management. There were recent changes that are made which caused the required version of cmake to be updated such that .NET could not be source built in many Linux distros (dotnet/runtime#38755). This required significant effort to revert this dependency. Capturing these kinds of dependency changes early can help to reduce unnecessary work and hopefully not release such changes in the wild.

4. A recent breakdown in understanding what platform dependencies we have is dotnet/aspnetcore#27950 where it wasn't known we even had a dependency on the procps package.

5. In addition to documentation, there are a variety of product assets that need an accurate reflection of the required dependencies:

• deb packages (https://github.com/dotnet/installer/tree/master/src/redist/targets/packaging/deb)
• RPM packages (https://github.com/dotnet/installer/tree/master/src/redist/targets/packaging/rpm)
• CBL-Mariner RPM package
• Snap packages (https://github.com/dotnet/runtime/tree/master/src/installer/snaps)
• Docker images (https://github.com/dotnet/dotnet-docker)

Keeping these up-to-date is key to maintaining a functioning product. When product dependencies are changed or changes are made externally by the operating system or ecosystem (such as Linux package versions), these changes need to be coordinated across all product assets.

User Stories

User Stories under this Epic:

• Describe platform dependencies
• Client library for reading platform dependency data
• Validate installed packages against actual product dependencies
• Keep Linux dependencies up-to-date in docs
• Platform dependency change detection
• Add install script option to install dependencies
• Website to browse platform dependency data
• Global tool for discovering an app's platform dependencies

[dleeapho]

/cc @shawnro

[danmoseley]

@mthalman @dleeapho this was missing a parent -- I put it under #5437. Feel free to correct if I'm wrong.

[mthalman]

@danmosemsft - There's not yet a theme for this and may not be. It's part of the Engineering Services work. @shawnro can provide more info.

[danmoseley]

Ah, OK. If it doesn't connect to a theme you may want to label it "bottom up work". Please do detach from #5437 if its not part of it

[omajid]

Thanks for creating this! I had a couple of thoughts in no particular order:

• For things like the C library, there are choices: users/developers can either use glibc or musl. But those are exclusive and shouldn't be mixed for one build. We will probably need some way to express these mutually exclusive dependencies.

• There's a note about updating stakeholders when dependencies are changed. I wonder if it would be better to automate even more: define the dependencies available in platforms and warn if updating dependencies breaks compat with those defined platforms.

• For platforms that are not supported - such as a Linux distribution not in the "supported list" - maybe the tooling can offer generic advice? Like OpenSSL 1.1 instead of a distro-specific package name?

• After skimming System.DirectoryServices.Protocols - Linux support runtime#36888 (comment), I am wondering how we might go about "full" vs partial dependencies. Which dependencies are listed as required and which are lightup or optional-depending-on-use-case?

[mthalman]

After skimming dotnet/runtime#36888 (comment), I am wondering how we might go about "full" vs partial dependencies. Which dependencies are listed as required and which are lightup or optional-depending-on-use-case?

@omajid - Yes, that's been a goal of mine as well. I think this is particularly relevant for scenarios like diagnostics where a developer wants to make use of diagnostic tools in which case LTTng would be listed as an optional library, for example. My intention was that requirements would be described for the core runtime as a whole and then other NuGet packages would each have their own set of distinct requirements. So while it's optional to make use of the System.DirectoryServices.Protocols NuGet package, libldap would be listed as a requirement of that package.

[mthalman]

I've found another source of .NET's native dependencies described in documentation for PowerShell (which is also out-of-date): https://docs.microsoft.com/powershell/scripting/install/installing-powershell-core-on-linux?view=powershell-7.1#dependencies

[hamarb123]

Hi, I'm trying to understand why libobjc.dylib is included in https://github.com/dotnet/core/blob/main/release-notes/6.0/runtime-deps.json since according to dotnet/runtime#49555 (comment), inherent components of the OS aren't listed in that file. Please correct me if I'm wrong in my assumption or interpretation, thanks.

[mthalman]

Hi, I'm trying to understand why libobjc.dylib is included in https://github.com/dotnet/core/blob/main/release-notes/6.0/runtime-deps.json since according to dotnet/runtime#49555 (comment), inherent components of the OS aren't listed in that file. Please correct me if I'm wrong in my assumption or interpretation, thanks.

Yes, that could be removed.

[mthalman]

Here's an example of an external system requiring knowledge of .NET dependencies across a wide range of platforms: https://github.com/microsoft/azure-pipelines-agent/blob/76d798f063189671a88f9b9f79d15ec5e9639867/src/Misc/layoutbin/installdependencies.sh

[mthalman]

In addition to runtime, SDK scenarios have their own dependencies with Native AOT compilation.

[mairaw]

Is this a .NET 6 epic @mthalman? Is this and its user stories still valid?

[mthalman]

Is this a .NET 6 epic @mthalman? Is this and its user stories still valid?

@mairaw - Yes, it was. But it kept being pushed down in priority. The fundamental thing to be solved here is still valid and would add value.