Build scripts for Packaging and Publishing GitVersion

[arturcic]

This is a PR for the issue #1445.

Some details about the scripts - inspired by the work done in the cake-build project.
There is a powershell core script - bootstrap that runs on all platforms - run.ps1
Added support for building on Azure Pipelines for all 3 platforms
The run.cake script is the one covering the build/testing/packaging and publishing.

The unit tests run on each of the platforms both dotnet test as well as using NUnit.

The Build/ Test and Packaging run on each build on each platform so that we could actually publish from any build server.

The Publish is currently enabled from AppVeyor.

There are 2 type of releases - from GitTools/GitVersion repository master branch. If it is tagged - the release is a stable release. If it's not tagged it's a pre-release.

The pre-release artifacts are published only to AppVeyor/Azure Pipelines. For the Stable release it's also published from AppVeyor to DockerHub, Nuget.org, Chocolatey.org, rubygems.org, and from Travis the Docker images. I decided to enable the Docker build and publish only if it's a stable or pre-release as it takes a lot of time to build them on windows - ~ 30 mins

I have tested pushing for all packages (using self hosted proget server) except for TFS.

There are 4 Docker images -
2 Windows images - one based on nano with dotnet core runtime image, one based on windowsservercore with full framework 4.7.2
2 Linux images - one based on dotnet core runtime image, one based on mono.

There are several required environment variables that we need to update with corresponding values:

GITHUB_TOKEN
DOCKER_USERNAME
DOCKER_PASSWORD
NUGET_API_KEY
NUGET_API_URL
CHOCOLATEY_API_KEY
CHOCOLATEY_API_URL
TFX_TOKEN
RUBY_GEM_API_KEY

There are also some optional env variables that act as flags to disable some parts of the build script:
DISABLE_UNIT_TESTS
DISABLE_PUBLISH_GEM
DISABLE_PUBLISH_TFS
DISABLE_PUBLISH_NUGET
DISABLE_PUBLISH_CHOCOLATEY
DISABLE_PUBLISH_DOCKER

Currently the README.md file has a badge for Azure pipelines build status that needs to be updated with the GitTools team's Azure pipelines url when it's set up.

I copied the nuspec files to the root of the repo as well as I have moved the code for ruby gems to a separate folder.

I kept the old build scripts so that you can compare them side by side with the new one.

I wasn't able to test the push of TFS package so I have included the code but it's not reachable at the moment. I'll need some assistance on that.

Please have a look and review it. Let me know if it covers all the scenarios and all the comments are welcomed.

@JakeGinnivan @dazinator @asbjornu @gep13

[dazinator]

Great stuff.

[arturcic]

I'll have to check why the Gem packaging is failing when it's a pull request - it's somehow related to the version generated when it's a pull request

[asbjornu]

This looks like a huge improvement over the current build. Would it be possible to have prereleases auto-published to NuGet? Otherwise I think this looks great. Thanks for the work you put into this, @arturcic! Is there anything I can do to help?

[arturcic]

Actually you can help - please review which publish tasks should be included when pre-release and when stable release, so that the appropriate criteria is applied. And would be great if you or who has access to AppVeyor account set the correct values for the required environment variables so that we can start testing/fixing the publish tasks.

Would be great if you can also setup a GitTools team account on Azure Pipelines so that the build happens on that CI as well.

[asbjornu]

@arturcic: Excellent. I'd like every package repository that supports unstable/pre-releases to be included when we do pre-releases. I don't know much about other package repositories than NuGet and Homebrew, so I can't say authoritatively which repositories should and shouldn't be included other than NuGet, since it supports it. Homebrew does not, so it should be excluded.

I have access to AppVeyor, but I don't have access to Docker, NuGet, Chocolatey, TFX (what's that?) or RubyGems, so I can't generate new credentials. I'm also not entirely sure how to generate a GitHub auth token properly. What access does AppVeyor need and who should own the token?

I've created an Azure DevOps account for GitTools and started setting up a pipeline, but stopped at the template step as I'm unsure whether the suggested ".NET Desktop" template would be a good fit. Perhaps we could hash this out on Gitter to speed up the interaction?

[JakeGinnivan]

If you DM me on twitter I can probably grant access to anything. This is awesome, thanks @arturcic

We should push to docker as a tagged image, and only update latest on stable

[arturcic]

I have access to AppVeyor, but I don't have access to Docker, NuGet, Chocolatey, TFX (what's that?) or RubyGems, so I can't generate new credentials. I'm also not entirely sure how to generate a GitHub auth token properly. What access does AppVeyor need and who should own the token?

tfx-cli is the tool used for packaging/publishing to Visual Studio marketplace, https://github.com/Microsoft/tfs-cli, so basically it's a task that can be used in TFS or Azure DevOps

@arturcic: Excellent. I'd like every package repository that supports unstable/pre-releases to be included when we do pre-releases.

As far as I know, Nuget, Chocolatey, RubyGems support pre-release, not sure about VS marketplace.
Docker has tags that we can use for pre-releases as well. Regarding Homebrew, we don't support that yet, I was planning to add that after this is merged

[arturcic]

We should push to docker as a tagged image, and only update latest on stable

Agree. I was also thinking to add the multi-arch manifest creation for Docker, but unfortunatelly it is not supported on any of the build servers.

[kaylangan]

I'm a Program Manager on Azure Pipelines. Let me know if you have any questions or suggestions. I'm happy to help!

[arturcic]

I'm a Program Manager on Azure Pipelines. Let me know if you have any questions or suggestions. I'm happy to help!

Good to know, thanks

[gep13]

@arturcic this is an AWESOME pull request. I have only had a very quick look through the changes, but everything seems legit to me.

@JakeGinnivan were you able to setup access to everything that is required? If not, let me know what is still outstanding, and I can take a look.

[gep13]

@arturcic thanks for making those changes!

[gep13]

So, all in all, this looks good to me, an EPIC PR from @arturcic.

@asbjornu @dazinator @JakeGinnivan you guys ok with this getting pulled in?

[arturcic]

Quick update on this

• fixed the versioning schema for artifacts, it remains to fix the tfs versioning as it does not acccept pre-releases
• tested the publish to Docker (you can check the dockerhub of gittools). After the merge of this PR I'll update the descriptions of the docker images on dockerhub to point to dockerfiles from github
• tested the publish to my own nuget/choco server
• tested publish to rubygem (https://rubygems.org/gems/gitversion there is already published)

Right now I'm finilizing the TFS publishing so could you wait with the merge till I finish with the changes @gep13? Hopefully today I'll have everything done.

[gep13]

@arturcic said...
Right now I'm finilizing the TFS publishing so could you wait with the merge till I finish with the changes @gep13? Hopefully today I'll have everything done.

Sounds good to me!

[asbjornu]

Big 👍 from me!

[arturcic]

I have tested the changes, the only thing that is not working - Publish-TFS task - is because of the personal access token expired when trying to publish to visual studion marketplace. There is a flag on appveyor that allows disabling the publishing for now for TFS till we get the new personal access token in place, so that we can merge the PR and see how the new scripts work. I have tested the Publish-TFS with a different name of the extension for testing purposes and it was working.

So basically the PR is ready for merge.

[dazinator]

@arturcic does this change sit ok with you: 2cf0fb1

[arturcic]

Yeah I guess it is ok for the time we merge that branch. We'll just add the version of the sdk directly in the bootstrap script.

[kll]

I went and actually read the docs now and I think we can just specify the major.minor version to pin us to the 2.1 series which is LTS and probably a reasonable thing to do without getting tied to a specific release that quickly gets out of date. If it works like I think is that an agreeable middle ground?

[dazinator]

@kll that sounds good.

[kll]

Never mind. It definitely did not work like I thought it did. I also didn't realize this PR was reading in the contents of the file as part of the new build process so I've just added it back with a matching version number. Even with a specific version number it looks like it rolls forward on the patch version automatically which prevents needing to go find an old exact version number to make it build. Hopefully it's kept up to date because I rarely have old versions of the SDK installed. :)

[jmezach]

Is this still waiting for something in particular, or is this ready to merge?

[asbjornu]

@jmezach: This is such a giant leap in the right direction that regardless of what's missing, I'm going ahead and merging this. Thanks for the poke!

[arturcic]

Thanks for merging the PR, I'll check if the build for the merged code is publishing correctly and fix the issues. Still the TFS publishing is not working because we need a valid personal access token (PAT) for publishing to visual studio marketplace. Hope @JakeGinnivan will have time to have a look and provide a valid PAT

[asbjornu]

@arturcic: Please poke @JakeGinnivan on Gitter as he doesn’t have capacity to stay on top of GitHub issue notifications. 🙏 😄

[kll]

I know there was a good reason to keep the old build scripts in place for reference but it is a bit confusing. I vote to clean them up because the old ones can always be viewed in the git history.

[arturcic]

I think I'll remove or probably rename back to build.cake and build.ps1 after the #1445 is closed