Linux packages JSON format

[richlander]

We have a packages file in markdown format currently. The JSON file is intended to be the source of truth and to be used to generate the markdown file (proposal in place for both formats).

I made a few design choices:

• Separate logical from physical packages.
• Document all the interesting notes (including the scenarios) on the logical packages, keeping the physical package info as lean (and duplication-free) as possible.
• Order the packages such that it is likely/possible to be able to foreach+install (like curl + jq + bash) required packages for both NAOT or CoreCLR w/o installing anything extra. This fits a bit into the "clever" category.

The intent is that any distro can be added here. NuGet packages are missing. We can add those, too.

I started with .NET 9 to prove out the idea. Adding the other .NET versions is trivial since the packages are all the same. The only difference is that the distro versions may fully, partially or not at all overlap. It's another example of a historical document.

I named this linux-packages.json to match linux-packages.md. It would be good to add platforms like FreeBSD here, too. I'm wondering if now is a good time to rename these files as required-packages.json or runtime-deps.json or similar.

Note: Here is a very similar past effort: https://github.com/dotnet/core/blob/main/release-notes/6.0/runtime-deps.json

@mthalman @ashnaga @Falco20019 @omajid @leecow

[richlander]

Here's a demo of one way the JSON format can be used.

$ docker run --rm -it ubuntu:24.04
root@243a281eaab7:/# curl -s https://raw.githubusercontent.com/dotnet/core/linux-packages/release-notes/9.0/linux-packages.json | jq -r .distributions[2].releases[1].packages[0:7][].name | xargs echo                
libc6 libgcc-s1 ca-certificates libssl3t64 libstdc++6 libicu74 tzdata
root@243a281eaab7:/# apt install $(curl -s https://raw.githubusercontent.com/dotnet/core/linux-packages/release-notes/9.0/linux-packages.json | jq -r .distributions[2].releases[1].packages[0:7][].name | xargs echo)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
libc6 is already the newest version (2.39-0ubuntu8.2).
libgcc-s1 is already the newest version (14-20240412-0ubuntu1).
ca-certificates is already the newest version (20240203).
ca-certificates set to manually installed.
libssl3t64 is already the newest version (3.0.13-0ubuntu3.1).
libstdc++6 is already the newest version (14-20240412-0ubuntu1).
The following NEW packages will be installed:
  libicu74 tzdata
0 upgraded, 2 newly installed, 0 to remove and 2 not upgraded.
Need to get 11.1 MB of archives.
After this operation, 38.5 MB of additional disk space will be used.

[mthalman]

My expectation is that this file would eventually be used as the source for generating more than just the release markdown file.

For other reviewers: what is being proposed here is essentially an implementation of #5646 which is part of a larger epic: #5651. There are a multitude of areas that need to derive this dependency data (read about them in the #5651 epic).

I'm fine with constraining the scope here for now to just target the release markdown files.

I'm wondering if now is a good time to rename these files as required-packages.json or runtime-deps.json or similar.

I don't have too much of an opinion here other than to say the naming should be specific to Linux and not Windows since that's the scope of this file. My schema in #5646 had attempted to combine Linux and Windows dependencies with a combined description but, looking back, it seemed unnecessary and made things ugly. If we ever want to describe Windows dependencies, that should be done in a separate file with its own schema.

[richlander]

I really just meant that it would be good to have the option to include FreeBSD (and the like) in this list. I agree that including Windows is a step too far. Also, the product doesn't have dependencies on macOS or Windows.

Yes, there are other opportunities to use dependency lists. One of the things I learned from my other efforts is that generating markdown is sufficient for validating the schema and ensuring that the data is maintained.

[richlander]

I added a formats description based on the request from @mthalman for more information on the new format. I'm sure that more can be added to the descriptions. I'll add more as I go. Feel free to do same.

489d5cd

[richlander]

Good question for @leecow.

#9412 (comment)

[richlander]

I added a new min-version concept for packages. @bartonjs and @vcsjones are considering increasing the min-version of OpenSSL to 1.1.1 (which is likely a good idea). I wanted to try that out. I updated the schema, included this particular example in the JSON, and included this data in the "Notes" section of the markdown table.

core/release-notes/9.0/os-packages.json

Line 40 in 3a8b35f

"min-version": "1.1.1",

FYI: There still needs to be an issue posts on this OpenSSL min-version change. This PR isn't the disclosure / public conversation for that. I intend to merge this PR as-is and see where there process goes.

[richlander]

I plan to merge this PR on Monday.

[mthalman]

The removal of Debian 11 made me realize that this content should be in sync with https://github.com/dotnet/core/blob/main/release-notes/9.0/supported-os.json. In other words, one could imagine a validation script that checks that all the distros listed here can be cross-referenced to supported-os.json. I see that supported-os.json has both an id and name for each distro. What's the need for both? Should this file reference by id instead?

Similarly for the release, you list a friendly name like Debian 12 (Bookworm). Why is that detailed information presented here (for a file that is package-centric) when that information isn't even contained in supported-os.json? Really, I see that information as being superfluous. You just need to know the distro id/name and version. In fact, I would argue that release should be renamed to version to more clearly indicate the relationship between these two files.

[richlander]

The friendly name is so that it is easy to generate markdown. Certainly, we could use the following implementation: $"{distro} {version}". I like having the codenames in the markdown since we use both the version numbers and codenames in different parts of our infra. It also enables a long distro name like "Red Hat Enterprise Linux", which doesn't fit very nicely in an "id" field.

https://github.com/dotnet/core/pull/9412/files#diff-c3805f9c7f5bc85d9d552816365e70fad4a2243ba220a607e2a1c1360e44fc17R33

In the other file, the id field can be thought of as endoflifedate-id. Those values match what endoflife.date uses: https://endoflife.date/. The tools we wrote query that database extensively. The name field is what is used in the generated markdown.

Excellent point on keeping in sync. I'm going to be writing some more tools related to staying in sync. That's a good scenario to include.

[richlander]

I haven't published a schema for supported-os.json yet. I'll make sure the connection with endoflife.date is documented.

[arrowd]

On FreeBSD we have libc as part of the base system and it doesn't need to be installed, so we need a way to somehow hide dependencies based on OS. The same goes for libgcc as we have LLVMs libcompilerrt as a replacement.

[richlander]

@arrowd -- I don't think that will be a problem. The nominal packages list is intended to be a superset view of packages. Each distro then is able to subscribe to that list, with a partial or full set. When the markdown is generated, it uses the distro packages only. So, I think the scheme already works as you need. Feel free to contribute to FreeBSD packages and I think it will just work.

On libcompilerrt, we can liberalize the package description in nominal package. That's what I did with libc to accomdate both musl and glibc. There are multiple alternative ecosystems.

[arrowd]

Sure, I'll try to find time for a follow-up after this PR lands.