Update to Rust/Cargo v1.75 and simplify Windows Conda setup

[oskirby]

Description

It took me three tries to get this done, but this should upgrade our Cargo/Rust toolchain to version v1.75 up from v1.69. The work was fairly straightforward for every platform except Windows, where all we really needed to do was bump a version number and work around some minor build quirks.

Windows however, turned out to be a beast, and I found it unable to build after touching the rust crates. This forced me to finally reckon with my personal development machine and figure out why it never worked with Conda. A quick tour of the changes on Windows that made this all work for me:

• The rust-std-apple-xxx packages now have a dependency on the pseudo-package __unix which makes them uninstallable on Windows. For this reason, I felt it was best to split out conda env.yml file into a UNIX and Windows variant. This enabled some other cleanup in the Windows env too and to omit the extras.ps1 script.
• A bump to Clang/LLVM 16.x.x brings with it significant improvement in the ability to autodetect the Visual Studio installation and any associated SDKs. This allows the Conda toolchain to work with an existing VS2019 installation.
• We write a toolchain file scripts/windows/conda-toolchain.cmake that can be used as a one-stop-shop to configure CMake with Clang-CL and the conda environment. This effectively replaces all of the environment hacking in the old conda_setup_win_sdk.ps1 script.
• We auto-generate a config.toml for Windows that passes the CMake toolchain to cargo this eliminates much of the environment hacking needed to get cargo to build.
• I wrote a new conda-setup-xwin-sdk.ps1 script using the latest version Jake-Shadle/xwin project using the new --use-winsysroot-style option which allows us to massively simplify the arguments to Clang and LLVM in order to connect the two.
• The MSM/MergeModules aren't installed by the Jake-Shadle/xwin project, and word on the internet is that they are only required to support Windows 8 and earlier. Given that we explicitly require Windows 10 (and even block installation on older version) we can just remove this and it ought to work.
• The MSM/MergeModules aren't installed by the Jake-Shadle/xwin project, so in order to have them available for the Wix installer, it was necessary to write a new tool scripts/windows/fetch-vsix-package.py that can parse the Visual Studio manifests and download VSIX extensions. This tool is then used to fetch the vcredit MergeModules into the xwin SDK.

I think this was especially painful for me because my Windows machine has MS Visual Studio 2019 installed for other projects too and I am hesitant to uninstall it to get conda working. Hopefully with these changes we should get a working build environment both with and without VS2019 installed.

Reference

First Attempt: #9812
Second Attempt: #9914

Checklist

• My code follows the style guidelines for this project
• I have not added any packages that contain high risk or unknown licenses (GPL, LGPL, MPL, etc. consult with DevOps if in question)
• I have performed a self review of my own code
• I have commented my code PARTICULARLY in hard to understand areas
• I have added thorough tests where needed

[strseb]

We don't support compiling android on windows, so no need to have those :)

[oskirby]

Huzzah!

[strseb]

I am waaaay to far away rn from a real windows PC to test this but afaik we require a VC_redist to be installed for the client to run. So if none are installed, (which is unlikey tbf) things might break, no ?

[oskirby]

My understanding is that from Windows 10 onwards microsoft has switched to a universal CRT that no longer requires a vcredist, and since the WIX installer goes out of its way to deny installation on earlier versions of Windows then this shouldn't be needed anymore.

But yes, much testing is required.

[oskirby]

Alas, I finally went and tested it and unfortunately it seems we really do need to install the MSVC redistributable somehow for the VPN client to run on Windows 10. Ugh. Back to the drawing board with this one.

[oskirby]

Okay, so, I really don't want to have to download the entire VS environment just to extract the MSM/VCRedist package, but I can't really find a good link to it on its own. The official word from Microsoft is that vcredists as MergeModules are deprecated and that we should be using the vcredist_x64.exe intstead, which should be version-agnostic.

However, this isn't a great solution either because of how the vcredist is generated... see this thread for some complaints from the VS developer community, and some notes about how it prevents automated installation using MSIs.

Running the vcredist_[arch].exe from the Wix installer will fail with error 1618: Another installation is currently in progress.

[oskirby]

Oh no... that sounds like a familiar mess: https://developercommunity.visualstudio.com/t/MSMs-for-VCRT-from-VS2019-makes-MSI-pack/1143857

[strseb]

This line brings me so much joy.