Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add job matrix generation scripts #1429

Merged
6 commits merged into from
Feb 19, 2021
Merged

Add job matrix generation scripts #1429

6 commits merged into from
Feb 19, 2021

Conversation

benbp
Copy link
Member

@benbp benbp commented Feb 19, 2021

This PR is a port of functionality that is currently duplicated across the net/java/python repositories. The intent was to settle on an implementation before moving it to the /eng/common/scripts directory. After merge to this location, I'll update the net/java/python and js repos to point to the eng/common/scripts location and remove the scripts from eng/scripts.

Here is the PR text used against the other repos for reference:

This adds scripts, docs and samples supporting dynamic, cross-product matrix generation for azure pipeline jobs.
It aims to replicate the cross-product matrix functionality in github actions,
but also adds some additional features like sparse matrix generation, cross-product includes and excludes, parameter grouping and matrix filters.

This functionality is made possible by the ability for the azure pipelines yaml to take a dynamic variable as an input
for a job matrix definition
(see the code sample at the bottom of the linked section).

See the README.md file for more details on the config file syntax and usage, as well as implementation details.

The tests (test-matrix-functions.tests.ps1) contain a lot of detail on expected data structures at various processing stages. The -test-matrix-functions.ps1` file could perhaps be split up or use some more organization, so let me know if it's hard to navigate.

Example:

{
  "displayNames": {
    "true": "TestFromSource"
  },
  "matrix": {
    "Agent": {
      "ubuntu-18.04": { "OSVmImage": "MMSUbuntu18.04", "Pool": "azsdk-pool-mms-ubuntu-1804-general" },
      "windows-2019": { "OSVmImage": "MMS2019", "Pool": "azsdk-pool-mms-win-2019-general" },
      "macOS-10.15": { "OSVmImage": "macOS-10.15", "Pool": "Azure Pipelines" }
    },
    "JavaTestVersion": [ "1.8", "1.11" ],
    "AZURE_TEST_HTTP_CLIENTS": [ "okhttp", "netty" ]
  },
  "include": [
    {
      "Agent": {
          "ubuntu-18.04": { "OSVmImage": "MMSUbuntu18.04", "Pool": "azsdk-pool-mms-ubuntu-1804-general" }
      },
      "JavaTestVersion": "1.11",
      "AZURE_TEST_HTTP_CLIENTS": "netty",
      "TestFromSource": true
    }
  ]
}

Sparse matrix job generation in a pipeline: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=705622&view=results

image

Related discussion: microsoft/azure-pipelines-yaml#20

@benbp benbp requested a review from a team as a code owner February 19, 2021 16:32
@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

@benbp benbp requested a review from weshaggard February 19, 2021 17:55
@benbp
Copy link
Member Author

benbp commented Feb 19, 2021

@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

@benbp benbp requested a review from weshaggard February 19, 2021 20:01
@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

@benbp benbp self-assigned this Feb 19, 2021
@benbp benbp added the Central-EngSys This issue is owned by the Engineering System team. label Feb 19, 2021
@ghost
Copy link

ghost commented Feb 19, 2021

Hello @azure-sdk!

Because this pull request has the auto-merge label, I will be glad to assist with helping to merge this pull request once all check-in policies pass.

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (@msftbot) and give me an instruction to get started! Learn more here.

@benbp
Copy link
Member Author

benbp commented Feb 19, 2021

/check-enforcer evaluate

@ghost ghost merged commit c770daa into Azure:master Feb 19, 2021
@benbp benbp deleted the benbp/job-matrix-scripts branch February 19, 2021 23:01
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Central-EngSys This issue is owned by the Engineering System team.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants