Error Recovery Example

This page describes an actual incident in which releasing to production failed and recovery steps were necessary. Each incident will be different and require analysis to determine the exact steps necessary, but this case may be useful as an example.

The Incident

In publishing the what was to be NUnit-Console release 3.16.1, the step PublishToNuget failed on AppVeyor with the following message (repeated for each package):

Pushing NUnit.Console.3.16.1.nupkg to 'https://www.nuget.org/api/v2/package'...
  PUT https://www.nuget.org/api/v2/package/
  Forbidden https://www.nuget.org/api/v2/package/ 969ms
Response status code does not indicate success: 403 (The specified API key is invalid, has expired, or does not have permission to access the specified package.).

Because each publishing step is allowed to run independently of the others, the PublishToChocolatey step executed without error. However, the final CreateProductionRelease step, which completes the GitHub release, was not executed.

Recovery Plan

In order to complete the release, the following tasks must be run locally:

• Package
• PublishToNuGet
• CreateProductionRelease

It should be noted that this implies that the binary files uploaded to NuGet.org and to GitHub must be created in such a way that they match exactly the file, which was already successfully uploaded to Choco.org. This is a potential weakness of the approach used but it is mitigated by using extreme care that the binaries are created and packaged in exactly the same way in the local environment as was done on AppVeyor. So far, this has not been a problem but, should the issue arise,other approaches are possible.

Detailed Steps

Packaging

1. In my local environment I switched to the version3 branch and did a git pull. This updated my local repository with the changes already merged into the branch when I had clicked Publish in the GitHub web environment.

2. I ran build -t BuildTestAndPackage to build the console and engine, run all the tests and create and test the packages. Just Package would have been enough, but I feel more confident also running the tests. These steps are exactly what is run on AppVeyor, followed there by the publishing steps where the failure occurred. So I'm confident that I now have the same packages that AppVeyor tried to publish.

Publishing to NuGet

3. I ran build -t PublishToNuGet to see if the key set in my local environment would work. It didn't: I got the same errors.

4. Looking at a list of Api Keys I maintain locally I selected one to try. I used set NUGET_API_KEY=xxxx (substituting the key) to place it in my environment and reran build -t PublishToNuGet. This time it worked.

Releasing on GitHub

5. I next ran build -t CreateProductionRelease, which uploaded all the assets to GitHub and added comments to each closed issue.

Finishing Up

6. I was now able to send out notices about the release.

7. I updated my local environment so that the correct API key will be set on startup.

8. I created a secure environment variable for the API key and updated appveyor.yml to use it.