PSA: New Manifest Schema

[denelon]

New Manifest Schema

We're about to implement major breaking changes to the YAML manifest syntax.

Discussion
JSON schemas

The intent is to enable a more robust way of describing packages with multiple architectures, languages, and installers. We will also be releasing a new CLI (ETA: May 1st) to make it easier to generate and submit pull requests for manifests (it will be open-source). Some of the new keys are in anticipation of new capabilities that are still in development (like dependencies).

There is a singleton schema for a "single file" YAML manifest for packages with only one locale and one installer. We will be converting all of the current manifests to this new format.

For all other packages there is a "multiple file" YAML structure. There is a "version" file that acts essentially as a reference or pointer for the other files. The "default locale" file provides the descriptive meta-data, additional "locale" files provide descriptive meta-data for other locales. The "installer" file provides the meta-data for one or more installers.

In addition, we will be partitioning the manifest directory by the (lower case )first letter of the publisher name. This will help with GitHub limitations around how many files/folders are displayed in the web UI. For example /m/Microsoft/PowerToys/.

We will be taking a snapshot of the repository and working through the conversion process which will incur downtime. All of the validation and publishing pipelines will be halted for this upgrade. Our first attempt will be Monday, March 22nd March 29th. Any open pull requests along with any new pull requests during this conversion will not be validated.

The Windows Package Manager client will continue to function normally during this transition.

If you are aware of any third party tools or utilities like the sandbox, PowerShell scripts or the list below, please let them know about the changes coming.

• https://github.com/ptorr-msft/WinGetYamlGenerator/ by @ptorr-msft
• https://winstall.app/ by @MehediH
• https://github.com/goranalkovic/WinGetGui by @goranalkovic
• https://winget.run/ by @Dragon1320 and @MattheousDT

Note: I will make edits to update the list of projects and authors/contributors as I am informed.

[DrewNaylor]

Having a version file will help a lot with trying to figure out where each manifest is. Sounds like it'll be a fun challenge to switch over support in my own code once this is complete.

[jedieaston]

Hooray! Now I don't have to worry about my indentation going wacky.

Is the current version of winget-cli able to consume the new manifest format or should we look for an update?

[denelon]

Our intent is to have the current client able to install packages that were submitted with the new schema. There are some new YAML keys that are present for future functionality that is not yet supported. So for those new behaviors, one would need to upgrade to a newer version of the client, but this is not a case where we anticipate an immediate upgrade of the Windows Package Manager (winget.exe) client.

[denelon]

I've got a draft PR with some examples microsoft/winget-cli#816.

We're working on something that can parse the JSON schemas and generate documentation so hopefully we will stay more current and accurate for documentation to be published in the next few days.

[denelon]

We ran into some internal issues this morning. If all goes well, we will try again tomorrow.

[denelon]

We've successfully tested in our pre-production environment. We're looking to start the migration this afternoon in production. I'll update this Issue when we start the migration and disable the validation pipelines. I'll also send a tweet out to try and make sure as many people can be informed as possible.

Any open PRs will remain open, but they will not validate as the schema will have been changed. Authors will need to update their PRs.

[jedieaston]

I was attempting to test a 1.0 manifest in preparation for this, and upon doing winget install -m...

PS C:\Users\WDAGUtilityAccount\Desktop\altmanifest> winget install -m .\Microsoft.VisualStudio.Community.yaml                                         An unexpected error occurred while executing the command:
Unsupported ManifestVersion: 1.0.0
0x8a150007 : Cannot process manifest. The manifest version is higher than supported. Please update the client.
PS C:\Users\WDAGUtilityAccount\Desktop\altmanifest>     

winget validate also fails with a similar message.

Should I open an issue for this in winget-cli?

[denelon]

@jedieaston, I'm pushing the current version of the package manager that knows how to validate 1.0 manifests out tomorrow. No need to open an issue. It includes several bug fixes as well as the winget import command.

[denelon]

We've got a bit more work to do on the back end to have enough time to feel good about the migration. We're pushing out until Monday March 29th. We have quite a few different services that all needed to be upgraded, and the one that got us was part of our manual validation. We're testing in our Pre-Production Environment (PPE) and we'll throw a few more scenarios at it tomorrow and Thursday. Updates will continue here as we make progress.

[denelon]

https://github.com/microsoft/winget-cli/releases/tag/v-0.2.10771-preview has just been published via GitHub. I am publishing this release to the Microsoft Store now. I'll update once it's available.

[denelon]

3/29 update:

All of the fully automated scenarios for validation and publishing are working in PPE.
We're locking down on a gap with the automation to help out with manual validation steps.
Once that's complete, I'll update this Issue with the schedule.

[denelon]

We are targeting the production migration to start tomorrow morning at approximately 8:30 AM PST. We expect this will take six to eight hours to complete. @KevinLaMS and I will both be presenting at the Microsoft MVP Summit starting at 8:00 AM so it's likely I won't be able to update until a bit later in the morning after the first presentation is complete. Hopefully, the migration will be finished before our second presentation at 5:00 PM PST.

[denelon]

I just finished the first MVP Summit presentation with @KevinLaMS and saw the first notification in my e-mail for the migration in progress. We're on our way to the new 1.0 schema.

[jedieaston]

:O

[denelon]

Yuck! I haven't seen that message before.