Merge pull request #573 from SMI/release/v1.14.0

Release v1.14.0

.azure-pipelines/README.md

@@ -0,0 +1,54 @@
+# Azure Pipelines Build Notes
+
+This directory contains the Azure Pipelines definitions and all other files used to test SmiServices.
+
+## Pipelines
+
+We currently have the following pipelines:
+-   [Linux test & package](https://dev.azure.com/smiops/Public/_build?definitionId=3). Defined [here](/.azure-pipelines/linux.yml). Runs the C# and Java tests on Linux. If the build is for a tag, also packages both sets of services and uploads them to a GitHub release
+-   [Windows test & package](https://dev.azure.com/smiops/Public/_build?definitionId=4). Defined [here](/.azure-pipelines/windows.yml). Same as above, but on Windows. Runs a reduced set of tests, since some services are not available (see below).
+-   [Create GitHub Release](https://dev.azure.com/smiops/Public/_build?definitionId=5). Defined [here](/.azure-pipelines/create-gh-release.yml). Runs automatically for tags on master to create a GitHub release which the other pipelines will publish packages to
+
+## Directory Contents
+
+-   `docker-compose/` - docker-compose files and lockfiles (see below)
+-   `jobs/` - job definitions
+-   `scripts/` - scripts used by these pipelines
+-   `steps/` - individual task step definitions
+-   `*.yml` - The top-level pipeline definitions
+-   `vars.json` - Variables for the pipelines
+
+Note that yaml files which are used as templates have the extension `.tmpl.yml`.
+
+## Variables
+
+Variables for the pipelines are loaded from the `vars.json` file at the start of each run. These are often combined with [pre-defined variables](https://docs.microsoft.com/en-us/azure/devops/pipelines/build/variables).
+
+## Services
+
+Service | Linux Provider (`ubuntu-18.04`) | Windows Provider (`windows-2019`)
+ ------ | -------------- | ----------------
+RabbitMQ | Docker | -
+MongoDB | Docker | pre-installed
+MsSQL | Docker | [SqlLocalDB](https://docs.microsoft.com/en-us/sql/database-engine/configure-windows/sql-server-express-localdb?view=sql-server-ver15)
+MariaDB | Docker | -
+Redis | Docker | Unavailable
+
+## Docker
+
+We use Docker containers for running the external services wherever possible. This currently means "when on Linux", since the Azure Windows OS currently doesn't support WSL2 / running Linux containers. Other options could be investigated for Windows, e.g. see a similar discussion [here](https://github.com/opensafely/job-runner/issues/76).
+
+The docker-compose files reference the `latest` tag for each image. During the pipeline run however, these are replaced with specific image digest versions using the [docker-lock](https://github.com/safe-waters/docker-lock) tool. This is so the docker-compose files can be used as cache keys to enable repeatable builds.
+
+## Caches
+
+The [Cache](https://docs.microsoft.com/en-us/azure/devops/pipelines/release/caching?view=azure-devops) task is used in multiple cases to speed-up the build by re-using previously built/downloaded assets. These are restored based on the `key` value, which can contain references to files which are hashed to generate the final key.
+
+## Notes
+
+-   `set -x` in bash tasks may interfere with the `##vso` syntax, as AP will interpret any stdout containing that string as a command. This can cause problems with variables being set twice and having the wrong quote escaping. See:
+    -   https://github.com/Microsoft/azure-pipelines-tasks/issues/10165
+    -   https://github.com/microsoft/azure-pipelines-tasks/issues/10331
+    -   https://developercommunity.visualstudio.com/content/problem/375679/pipeline-variable-incorrectly-inserts-single-quote.html
+
+-   Dealing with variables containing path separators across multiple platforms (i.e. Linux and Windows) can be tricky. Variables pre-defined by AP will have the correct separators for the current OS. When used in `Bash` tasks, it's safe enough for either OS to just use Linux (`/`) separators when combining paths, so long as they are always quoted. However, if the variable is also used as part of a cache `path`, it may need to be manually re-created with Windows-style (`\`) separators before use.

.azure-pipelines/create-gh-release.yml

@@ -0,0 +1,44 @@
+---
+
+# NOTE(rkm 2021-02-02) Only trigger for tags on master
+# See https://stackoverflow.com/a/60160047
+trigger:
+  branches:
+    include:
+      - master
+  tags:
+    include:
+      - "*"
+  paths:
+    exclude:
+      - /*
+
+pr: none
+
+pool:
+  vmImage: 'ubuntu-latest'
+
+steps:
+  - template: steps/set-git-tag-variable.tmpl.yml
+  - task: UsePythonVersion@0
+    inputs:
+      versionSpec: '3.x'
+      addToPath: true
+      architecture: 'x64'
+  - task: PythonScript@0
+    inputs:
+      scriptSource: 'filePath'
+      scriptPath: .azure-pipelines/scripts/write_release_changelog.py
+      arguments: $(tag)
+      failOnStderr: true
+  - task: GitHubRelease@1
+    inputs:
+      gitHubConnection: 'GitHub - SMI-Bot'
+      repositoryName: '$(Build.Repository.Name)'
+      action: 'create'
+      target: '$(Build.SourceVersion)'
+      tagSource: 'gitTag'
+      releaseNotesFilePath: 'release_changelog.md'
+      assets:
+      addChangeLog: false
+

.azure-pipelines/docker-compose/linux-dotnet.yml

@@ -0,0 +1,35 @@
+---
+version: "3.7"
+
+services:
+    rabbitmq:
+        container_name: rabbitmq
+        image: rabbitmq:latest
+        ports:
+            - 5672:5672
+    mariadb:
+        container_name: mariadb
+        image: mariadb:latest
+        environment:
+            - MYSQL_ROOT_PASSWORD=${DB_PASSWORD:-'YourStrongPassw0rd'}
+        ports:
+            - 3306:3306
+    redis:
+        container_name: redis
+        image: redis:latest
+        ports:
+            - 6379:6379
+    mssql:
+        container_name: mssql
+        image: mcr.microsoft.com/mssql/server:2017-latest
+        environment:
+            - ACCEPT_EULA=Y
+            - SA_PASSWORD=${DB_PASSWORD:-'YourStrongPassw0rd'}
+        ports:
+            - 1433:1433
+    mongodb:
+        container_name: mongodb
+        image: mongo:latest
+        command: mongod --replSet rs0
+        ports:
+            - 27017:27017

.azure-pipelines/docker-compose/linux-dotnet.yml.lock

@@ -0,0 +1,36 @@
+{
+	"composefiles": {
+		"linux-dotnet.yml": [
+			{
+				"name": "mariadb",
+				"tag": "latest",
+				"digest": "cdc553f0515a8d41264f0855120874e86761f7c69407b5cfbe49283dc195bea8",
+				"service": "mariadb"
+			},
+			{
+				"name": "mongo",
+				"tag": "latest",
+				"digest": "7722bd2778a299b6f4a62b93a0d2741c734ba7332a090131030ca28261a9a198",
+				"service": "mongodb"
+			},
+			{
+				"name": "mcr.microsoft.com/mssql/server",
+				"tag": "2017-latest",
+				"digest": "c9832ca564c8410bf13a7d6276a0bf6f5d83d0f6a15c6a5004ba07a04bf93920",
+				"service": "mssql"
+			},
+			{
+				"name": "rabbitmq",
+				"tag": "latest",
+				"digest": "1fc999fbdd8054b0c34c285901474f136792ad4a42d39186f3793ac40de04dba",
+				"service": "rabbitmq"
+			},
+			{
+				"name": "redis",
+				"tag": "latest",
+				"digest": "0f724af268d0d3f5fb1d6b33fc22127ba5cbca2d58523b286ed3122db0dc5381",
+				"service": "redis"
+			}
+		]
+	}
+}

.azure-pipelines/docker-compose/linux-java.yml

@@ -0,0 +1,9 @@
+---
+version: "3.7"
+
+services:
+    rabbitmq:
+        container_name: rabbitmq
+        image: rabbitmq:latest
+        ports:
+            - 5672:5672

.azure-pipelines/docker-compose/linux-java.yml.lock

@@ -0,0 +1,12 @@
+{
+	"composefiles": {
+		"linux-java.yml": [
+			{
+				"name": "rabbitmq",
+				"tag": "latest",
+				"digest": "1fc999fbdd8054b0c34c285901474f136792ad4a42d39186f3793ac40de04dba",
+				"service": "rabbitmq"
+			}
+		]
+	}
+}

.azure-pipelines/jobs/package.tmpl.yml

@@ -0,0 +1,42 @@
+---
+parameters:
+    vmImage:
+    os:
+
+jobs:
+    - job: build_artefacts
+      displayName: Build Artefacts
+      pool:
+          vmImage: ${{ parameters.vmImage }}
+      condition: startsWith(variables['Build.SourceBranch'], 'refs/tags/')
+      dependsOn: [dotnet_tests, java_tests]
+      steps:
+          - template: ../steps/vars.tmpl.yml
+          - template: ../steps/use-dotnet.tmpl.yml
+          - template: ../steps/set-git-tag-variable.tmpl.yml
+          - task: Bash@3
+            displayName: Build artefacts
+            inputs:
+              targetType: inline
+              script: |
+                  set -euxo pipefail
+                  os="${{ parameters.os }}"
+                  python .azure-pipelines/scripts/buildArtefacts.py $os $(tag)
+          - task: Bash@3
+            displayName: List files
+            inputs:
+              targetType: inline
+              script: |
+                set -euxo pipefail
+                find dist -type f
+                cat dist/**/MD5SUMS*
+          - task: GitHubRelease@1
+            inputs:
+              gitHubConnection: 'GitHub - SMI-Bot'
+              repositoryName: '$(Build.Repository.Name)'
+              action: 'edit'
+              target: '$(Build.SourceVersion)'
+              tag: '$(tag)'
+              assets: dist/$(tag)/*
+              assetUploadMode: 'replace'
+              addChangeLog: false

.azure-pipelines/jobs/tests.tmpl.yml

@@ -0,0 +1,118 @@
+---
+parameters:
+    vmImage:
+    # NOTE(rkm 2020-12-27) OS is required for the template conditions which are evaluated at compile-time before the Agent.OS is known
+    os:
+
+jobs:
+    - job: dotnet_tests
+      displayName: Dotnet Tests
+      pool:
+          vmImage: ${{ parameters.vmImage }}
+      steps:
+          - template: ../steps/vars.tmpl.yml
+          - template: ../steps/use-dotnet.tmpl.yml
+          - task: Cache@2
+            displayName: Cache NuGet packages
+            inputs:
+                key: 'nuget | "$(Agent.OS)" | **/*.csproj'
+                path: $(NUGET_PACKAGES)
+          - task: Cache@2
+            displayName: Cache tesseract data
+            inputs:
+                key: 'tessdata | "$(Agent.OS)"'
+                restoreKeys: |
+                    tessdata
+                path: $(TESS_CACHE_DIR)
+                cacheHitVar: TESS_CACHE_RESTORED
+          - task: Bash@3
+            displayName: Download tessdata if cache not restored
+            condition: eq(variables.TESS_CACHE_RESTORED, 'false')
+            inputs:
+                targetType: "inline"
+                script: |
+                    set -euxo pipefail
+                    mkdir -p "$(TESS_CACHE_DIR)"
+                    curl -L \
+                        https://github.com/tesseract-ocr/tessdata/raw/master/eng.traineddata \
+                        > "$(TESS_CACHE_DIR)/eng.traineddata"
+                    ls -l "$(TESS_CACHE_DIR)"
+          - task: Bash@3
+            displayName: Symlink tessdata into the repo
+            inputs:
+                targetType: "inline"
+                script: |
+                  ln -s \
+                    "$(TESS_CACHE_DIR)/eng.traineddata" \
+                    "$(Build.SourcesDirectory)/data/tessdata/eng.traineddata"
+          - ${{ if eq( parameters.os, 'linux' ) }}:
+                - template: ../steps/docker-cache.tmpl.yml
+                  parameters:
+                      os: ${{ parameters.os }}
+                      language: dotnet
+                - template: ../steps/linux-dotnet-docker.yml
+          - ${{ if eq( parameters.os, 'win' ) }}:
+                - template: ../steps/win-dotnet-services.yml
+          - task: UseDotNet@2
+            displayName: Use .NET Core Runtime 2.2 for RDMP
+            inputs:
+                packageType: runtime
+                version: 2.2.x
+                installationPath: $(Agent.ToolsDirectory)/dotnet
+          - template: ../steps/rdmp-setup.tmpl.yml
+            parameters:
+                os: ${{ parameters.os }}
+          # NOTE(rkm 2021-02-02) See:
+          # https://github.com/actions/setup-dotnet/issues/155
+          # https://github.com/dotnet/core/issues/5881
+          - ${{ if eq( parameters.os, 'win' ) }}:
+                - task: Bash@3
+                  displayName: Attempt to clean Nuget cache in Windows builds
+                  inputs:
+                      targetType: "inline"
+                      script: dotnet nuget locals all --clear
+          - task: Bash@3
+            displayName: Run Dotnet tests
+            inputs:
+                filePath: .azure-pipelines/scripts/run-dotnet-tests.bash
+                failOnStderr: true
+
+    - job: java_tests
+      displayName: Java Tests
+      pool:
+          vmImage: ${{ parameters.vmImage }}
+      steps:
+          - template: ../steps/vars.tmpl.yml
+          - task: JavaToolInstaller@0
+            displayName: Use Java 11
+            inputs:
+                versionSpec: "11"
+                jdkArchitectureOption: "x64"
+                jdkSourceOption: "PreInstalled"
+          - task: Cache@2
+            displayName: Cache Maven local repo
+            inputs:
+                key: 'maven | "$(Agent.OS)" | **/pom.xml'
+                restoreKeys: |
+                    maven | "$(Agent.OS)"
+                    maven
+                path: $(MAVEN_CACHE_FOLDER)
+          - ${{ if eq( parameters.os, 'linux' ) }}:
+                - template: ../steps/docker-cache.tmpl.yml
+                  parameters:
+                      os: ${{ parameters.os }}
+                      language: java
+                - template: ../steps/linux-java-docker.yml
+          - ${{ if eq( parameters.os, 'win' ) }}:
+                - task: Bash@3
+                  displayName: Skip Java integration tests on Windows
+                  inputs:
+                      targetType: "inline"
+                      script: echo "##vso[task.setvariable variable=MVN_PROFILE]-PunitTests"
+          - task: Bash@3
+            displayName: Run Java tests
+            inputs:
+                filePath: .azure-pipelines/scripts/run-java-tests.bash
+                # TODO(rkm 2020-12-11) See if we can silence the current bogus stderr output (log4j warning & SMI_LOGS_ROOT)
+                failOnStderr: false
+                arguments: $(MVN_PROFILE)

.azure-pipelines/linux.yml

@@ -0,0 +1,18 @@
+---
+trigger:
+    branches:
+        include:
+            - "*"
+    tags:
+        include:
+            - "*"
+
+jobs:
+    - template: jobs/tests.tmpl.yml
+      parameters:
+          vmImage: ubuntu-18.04
+          os: linux
+    - template: jobs/package.tmpl.yml
+      parameters:
+          vmImage: ubuntu-18.04
+          os: linux

.azure-pipelines/scripts/Databases.yaml

@@ -0,0 +1 @@
+SqlServer: 'Server=(localdb)\MSSQLLocalDB;'