Native tools bootstrapping via Arcade is useful to enable our toolset policy

[MattGal]

• Guidance is updated and enables https://github.com/dotnet/arcade/blob/master/Documentation/Policy/ToolsetPolicy.md
• Native tools bootstrapping is conditional by OS
• Works well in a Docker container as well
• Won't re-install if already available

======================================

Originally from @dougbu ; basically the ASP.NET Team needs tar for Windows, which we lost in a Windows Client SKU version snafu.

ASP.NET has made a workaround for now, but there remains no way to have a tool dependency in the global.json that is only for builds on a specific OS, so for any team that has a native dependency they'd like to bootstrap that only applies to one of three of the main OS choices, we don't really have a story for these folks.

Simplest thing to do is likely to have some kind of property in said JSON with an OS check, but there could be nicer ways to do it.

Recently Triaged Issues

All issues in this section should be triaged by the v-team into one of their business objectives or features.

• Enable building our repo's without requiring Visual Studio installed #64
• Explore moving some of the installer script logic into the common library #833
• Native Bootstrapping stops and aborts further processing when it encounters a tool without an executable #2248
• Native toolset installation scripts between Windows / Linux are not at parity #2674
• Support "ignore errors" type mode in native toolset installs #4482
• Don't install tools that are already available #4780
• Creating native toolset bundles is sensitive to compression tool #4786
• Fragile native tool bundle format #4787
• Native tool asset access is not easily available #4788
• Native bootstrapping needs to support xcopy deployable tools with no installer 'exe' in the package #4790
• Improve perf of native toolset boostrapping #4791
• Native toolset bootstrapper should support private cache #4792
• Native toolset bootstrapping should have better support for shared tools #4793
• [Discussion] Blockers for native toolset bootstrapping adoption in .NET Core repos #4795
• Refactor common utilities functions into common/util.ps1 #4822
• native-tools in global.json should be added to the path #5434
• Investigate Native bootstrapping errors in macos #7327

[dougbu]

Thanks @MattGal❕

tar is a solid Windows-only example. We could have handled that by ensuring init-tools-native.cmd or init-tools-native.ps1 is invoked and only on Windows.

However the opposite case happens with tools such as the nginx reverse proxy. We need that everywhere except on Windows.

[dougbu]

Ping. Any update on this issue?

[markwilkie]

@dougbu - do you think this warrants an increase in urgency? We know it's a priority and needs to be done - but perhaps timing is important?

[dougbu]

Timing is important mainly w.r.t. https://github.com/dotnet/arcade/blob/master/Documentation/Policy/ToolsetPolicy.md. That doesn't have a deadline I'm aware of but let's avoid a mad rush once it does. The earlier we unblock relying on Arcade for more tools, the smoother things will go.

[markwilkie]

A fair point for sure, and I for one am in agreement. I've converted this issue to an epic to make sure this is prioritized sooner than later.

[chcosta]

Rough costing

Major issues:

• Permit defining native tools installation / OS
  o Cost: 2 - 8 weeks
  o The SWAG here is pretty large because the implementation will require a rewrite of native toolset bootstrapping.
   2 - 4 weeks – define a rid model like https://github.com/dotnet/runtime/tree/master/src/libraries/pkg/Microsoft.NETCore.Platforms. The pain point here is figuring out how to parse the model using our current design which is Powershell (Windows) or Bash (Linux) and supporting it in both places. XML parsing is notoriously difficult in Bash. The simplest solution (2 weeks) is to just bake the rid graph for all OS / distros into the bash rather than trying to parse a common graph model file.
   8 weeks - It doesn’t really make sense for us to write our own rid graph fallback model. The original implementation went the direction of using powershell / bash because the languages are simple and readily available on all OS/distros including those not yet supported by .NET Core. I think unsupported platforms are less of a concern now than they were in the past and we should optimize for platforms supported by .NET Core. We should rewrite native toolset bootstrapping to acquire the dotnet CLI, and use the CLI for things like OS versions (https://github.com/dotnet/runtime/tree/master/src/libraries/pkg/Microsoft.NETCore.Platforms). We would still need some shim scripts to acquire the CLI, but otherwise we can use a single code base (this would also address Native toolset installation scripts between Windows / Linux are not at parity #2674).
• Don't install tools that are already available
  o Cost: 2 – 4 weeks
  o I see a couple obvious solutions with different costings
   2 week solution - Simplest solution is to implement some more rigor around a tool artifact so that we can associate version discovery for a tool, its default install location, etc… so that we can discover tools in default locations or in the path and then do version matching. Then we can do non-exact matching for dev mode builds and exact version matching for ci builds.
   2 week solution – I hear there is some version matching we are already doing for artifacts w.r.t. os onboarding, we could investigate re-using that logic
   4 week solution – One proposal is to require native tools are run in contains. It would take some time to figure out what this would look like, generate containers, wire it in to the builds, figure out how this would be handled (or ignored) on Mac

[MattGal]

Assigned to @jonfortescue to close out; his current epic should encompass most of this.

[jonfortescue]

Kicked a few things to triage but most of these could just be closed. Closing the epic now.