Sync main to feature/storage-stg87 (#4407)

* update comment (#4364) * update comment * jghjg * update broken link * Update sdk/keyvault/tools/cleanup/src/cleanup.cpp Co-authored-by: Anton Kolesnyk <[email protected]> --------- Co-authored-by: Anton Kolesnyk <[email protected]> * Docker comment (#4375) * update comment * add comment about vcpkg * dsfs * Trigger`keyvault` on proxy changes (#4343) * autotrigger the keyvault and storage CI when a testproxy file is changed * Generate API review for C++ using new parser (#4302) * Add vmImage back to common perf.yml - Fixes #5466 - Partially reverts #5456 (#4376) Co-authored-by: Mike Harder <[email protected]> * Fix pipelines path (#4358) * test path * qwq * dsda * asas * dsada * sdsds * Update sdk/keyvault/azure-security-keyvault-certificates/perf-tests.yml Co-authored-by: Mike Harder <[email protected]> * Update sdk/keyvault/azure-security-keyvault-certificates/perf-tests.yml Co-authored-by: Mike Harder <[email protected]> * remove warmup --------- Co-authored-by: Mike Harder <[email protected]> * Test proxy & storage tests improvements (#4241) * Temporarily pin Node 18 to 18.13.0 - Fixes #5536 (#4378) Co-authored-by: Mike Harder <[email protected]> * Storage tests improvement (#4382) * Update CODEOWNERS (#4380) * show headers and query parameters in storage CI pipeline (#4379) * Sync eng/common directory with azure-sdk-tools for PR 5431 #2501 Co-authored-by: Konrad Jamrozik <[email protected]> * Sync eng/common directory with azure-sdk-tools for PR 5562 (#4384) * Add todos to update packages to pick up the newest CODEOWNERS interpreter * update --------- Co-authored-by: Konrad Jamrozik <[email protected]> * Fix Share Client failure #4377 (#4381) * Fix Share Client failure #4377 * decrease request count to avoid throttling errors for storage tests (#4385) * Sync eng/common directory with azure-sdk-tools for PR 5568 (#4387) * we encourage folks to place their assets.jsons at the package level * update generate-assets-json.ps1 to only include src/**/session-records so as to avoid picking up the duplicated 'target' sessionrecords --------- Co-authored-by: scbedd <[email protected]> * Add support to ignore invalid cert common name (#4361) * update to version 7.4 for admin. update tests (#4388) * update proxy version to include Info/Active against individual sessions + allow delayed response (#4391) Co-authored-by: scbedd <[email protected]> * Sync eng/common directory with azure-sdk-tools for PR 5540 (#4396) * add parameter to set cadl emitter options * remove emitter name in the additional parameter --------- Co-authored-by: chunyu3 <[email protected]> * Follow-up to update changelog to reflect community contribution (#4393) * Follow-up to update changelog to reflect community contribution * Upgrade cspell version from 0.1 to 0.2 * logging api post request body (#4404) Co-authored-by: Albert Cheng <[email protected]> * Sync eng/common directory with azure-sdk-tools for PR 5595 (#4400) * Use "npm ci" to install cspell and respect package-lock.json * Review feedback * Pipe npm ci output to Write-Host --------- Co-authored-by: Daniel Jurek <[email protected]> --------- Co-authored-by: George Arama <[email protected]> Co-authored-by: Anton Kolesnyk <[email protected]> Co-authored-by: Scott Beddall <[email protected]> Co-authored-by: Praven Kuttappan <[email protected]> Co-authored-by: Azure SDK Bot <[email protected]> Co-authored-by: Mike Harder <[email protected]> Co-authored-by: JinmingHu <[email protected]> Co-authored-by: Konrad Jamrozik <[email protected]> Co-authored-by: chunyu3 <[email protected]> Co-authored-by: Ahson Khan <[email protected]> Co-authored-by: Albert Cheng <[email protected]> Co-authored-by: Daniel Jurek <[email protected]>
Azure · Mar 6, 2023 · ad6eb88 · ad6eb88
1 parent 2df533e
commit ad6eb88
Show file tree

Hide file tree

Showing 88 changed files with 3,738 additions and 4,759 deletions.
diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
@@ -29,7 +29,7 @@
 /sdk/keyvault/           @vhvb1989 @gearama @antkmsft @rickwinter
 
 # PRLabel: %Storage
-/sdk/storage/            @vinjiang @katmsft @Jinming-Hu @EmmaZhu @antkmsft @vhvb1989 @gearama @LarryOsterman @microzchang
+/sdk/storage/            @vinjiang @Jinming-Hu @EmmaZhu @antkmsft @vhvb1989 @gearama @LarryOsterman @microzchang
 
 # PRLabel: %EngSys
 /sdk/template/           @danieljurek @weshaggard @LarryOsterman @RickWinter 

diff --git a/.vscode/cspell.json b/.vscode/cspell.json
@@ -1,5 +1,5 @@
 {
-  "version": "0.1",
+  "version": "0.2",
   "language": "en",
   "languageId": "cpp",
   "dictionaries": [

diff --git a/eng/common/pipelines/templates/jobs/perf.yml b/eng/common/pipelines/templates/jobs/perf.yml
@@ -8,9 +8,15 @@ parameters:
 - name: LinuxPool
   type: string
   default: 'azsdk-pool-mms-ubuntu-2204-perf'
+- name: LinuxVmImage
+  type: string
+  default: 'ubuntu-22.04'
 - name: WindowsPool
   type: string
   default: 'azsdk-pool-mms-win-2022-perf'
+- name: WindowsVmImage
+  type: string
+  default: 'windows-2022'
 - name: Language
   type: string
   default: ''
@@ -68,10 +74,12 @@ jobs:
       ${{ if contains(parameters.OperatingSystems, 'Linux') }}:
         Linux:
           Pool: ${{ parameters.LinuxPool }}
+          OsVmImage: ${{ parameters.LinuxVmImage }}
           MatrixName: 'Linux'
       ${{ if contains(parameters.OperatingSystems, 'Windows') }}:
         Windows:
           Pool: ${{ parameters.WindowsPool }}
+          OsVmImage: ${{ parameters.WindowsVmImage }}
           MatrixName: 'Windows'
   variables:
   - ${{ parameters.Variables }}
@@ -82,6 +90,7 @@ jobs:
       value: ''
   pool:
     name: $(Pool)
+    vmImage: $(OSVmImage)
   steps:
 
   - template: /eng/common/pipelines/templates/steps/sparse-checkout.yml
@@ -100,6 +109,10 @@ jobs:
         Commitish: ${{ parameters.ToolsRepoCommitish }}
         WorkingDirectory: $(System.DefaultWorkingDirectory)/azure-sdk-tools
 
+  - template: /eng/common/pipelines/templates/steps/verify-agent-os.yml
+    parameters:
+      AgentImage: $(OSVmImage)
+
   - ${{ parameters.InstallLanguageSteps }}
 
   - template: /eng/common/TestResources/deploy-test-resources.yml

diff --git a/eng/common/pipelines/templates/steps/check-spelling.yml b/eng/common/pipelines/templates/steps/check-spelling.yml
@@ -18,8 +18,8 @@ steps:
     - task: NodeTool@0
       condition: and(succeededOrFailed(), ne(variables['Skip.SpellCheck'],'true'))
       inputs:
-        versionSpec: 18.x
-      displayName: Use Node.js 18.x
+        versionSpec: 18.13.0
+      displayName: Use Node.js 18.13.0
 
     - task: PowerShell@2
       displayName: Check spelling (cspell)

diff --git a/eng/common/scripts/Cadl-Project-Generate.ps1 b/eng/common/scripts/Cadl-Project-Generate.ps1
@@ -4,7 +4,9 @@
 param (
     [Parameter(Position=0)]
     [ValidateNotNullOrEmpty()]
-    [string] $ProjectDirectory
+    [string] $ProjectDirectory,
+    [Parameter(Position=1)]
+    [string] $CadlAdditionalOptions ## addtional cadl emitter options, separated by semicolon if more than one, e.g. option1=value1;option2=value2
 )
 
 $ErrorActionPreference = "Stop"
@@ -78,6 +80,12 @@ try {
         }
     }
     $cadlCompileCommand = "npx cadl compile $mainCadlFile --emit $emitterName$emitterAdditionalOptions"
+    if ($CadlAdditionalOptions) {
+        $options = $CadlAdditionalOptions.Split(";");
+        foreach ($option in $options) {
+            $cadlCompileCommand += " --option $emitterName.$option"
+        }
+    }
     Write-Host($cadlCompileCommand)
     Invoke-Expression $cadlCompileCommand
 

diff --git a/eng/common/scripts/artifact-metadata-parsing.ps1 b/eng/common/scripts/artifact-metadata-parsing.ps1
@@ -34,6 +34,9 @@ function CreateReleases($pkgList, $releaseApiUrl, $releaseSha) {
       body             = $releaseNotes
     }
 
+    Write-Host "Post Request Body:"
+    Write-Host $body
+
     $headers = @{
       "Content-Type"  = "application/json"
       "Authorization" = "token $($env:GH_TOKEN)"

diff --git a/eng/common/scripts/get-codeowners.ps1 b/eng/common/scripts/get-codeowners.ps1
@@ -62,7 +62,7 @@ param (
   # but not this one:
   #    Remove the obsolete, prefix-based CODEOWNERS matcher & related tests
   #    https://github.com/Azure/azure-sdk-tools/pull/5431
-  [string]$ToolVersion = "1.0.0-dev.20230214.3",
+  [string]$ToolVersion = "1.0.0-dev.20230223.4",
   [string]$ToolPath = (Join-Path ([System.IO.Path]::GetTempPath()) "codeowners-tool-path"),
   [string]$DevOpsFeed = "https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-net/nuget/v3/index.json",
   [string]$VsoVariable = "",
@@ -168,7 +168,7 @@ if ($Test) {
   $azSdkToolsCodeowners = (Resolve-Path "$PSScriptRoot/../../../.github/CODEOWNERS")
   TestGetCodeowners -targetPath "eng/common/scripts/get-codeowners.ps1" -codeownersFileLocation $azSdkToolsCodeowners -includeNonUserAliases $true -expectReturn @("konrad-jamrozik", "weshaggard", "benbp")
 
-  $testCodeowners = (Resolve-Path "$PSScriptRoot/../../../tools/code-owners-parser/Azure.Sdk.Tools.RetrieveCodeOwners.Tests/TestData/glob_path_CODEOWNERS")
+  $testCodeowners = (Resolve-Path "$PSScriptRoot/../../../tools/code-owners-parser/Azure.Sdk.Tools.RetrieveCodeOwners.Tests/TestData/test_CODEOWNERS")
   TestGetCodeowners -targetPath "tools/code-owners-parser/Azure.Sdk.Tools.RetrieveCodeOwners.Tests/TestData/InputDir/a.txt" -codeownersFileLocation $testCodeowners -includeNonUserAliases $true -expectReturn @("2star")
   exit 0
 }

diff --git a/eng/common/spelling/Invoke-Cspell.ps1 b/eng/common/spelling/Invoke-Cspell.ps1
@@ -25,7 +25,7 @@ created in the temp folder, package*.json files will be placed in that folder.
 .PARAMETER LeavePackageInstallCache
 If set the PackageInstallCache will not be deleted. Use if there are multiple
 calls to Invoke-Cspell.ps1 to prevent creating multiple working directories and
-redundant calls `npm install`.
+redundant calls `npm ci`.
 
 .PARAMETER Test
 Run test functions against the script logic
@@ -167,9 +167,7 @@ $originalLocation = Get-Location
 
 try {
   Set-Location $PackageInstallCache
-  npm install npx | Out-Null
-  npm install cspell | Out-Null
-  npm install | Out-Null
+  npm ci | Write-Host
 
   # Use the mutated configuration file when calling cspell
   $command = "npx cspell $JobType --config $CSpellConfigPath --no-must-find-files --root $SpellCheckRoot --relative"

diff --git a/eng/common/testproxy/target_version.txt b/eng/common/testproxy/target_version.txt
@@ -1 +1 @@
-1.0.0-dev.20230201.10
+1.0.0-dev.20230224.5
diff --git a/eng/common/testproxy/transition-scripts/README.md b/eng/common/testproxy/transition-scripts/README.md
@@ -60,12 +60,12 @@ You will not be able to clean them up however. There exists [planned work](https
 - `language` repo - An individual language repository eg. azure-sdk-for-python or azure-sdk-for-net etc.
 - `assets` repo - The repository where assets are being moved to. <https://github.com/Azure/azure-sdk-assets>
 
-The `test-proxy` tool is integrated with the ability to automatically restore these assets. This process is kick-started by the presence of an `assets.json` alongside a dev's actual code. This means that while assets will be cloned down externally, the _map_ to those assets will be stored alongside the tests. Normally, it is recommended to create an `assets.json` under the path `sdk/<ServiceDirectory>`. However, more granular storage is also possible.
+The `test-proxy` tool is integrated with the ability to automatically restore these assets. This process is kick-started by the presence of an `assets.json` alongside a dev's actual code. This means that while assets will be cloned down externally, the _map_ to those assets will be stored alongside the tests. Normally, it is recommended to create an `assets.json` under the path `sdk/<ServiceDirectory>/<package>`. More granular storage than on an individual package level is possible, but each language's test framework would need to support that on a case-by-case basis.
 
-Service/Package-Level examples:
+Examples of current assets.json locations:
 
-- `sdk/storage/assets.json`
-- `sdk/storage/azure-storage-file-datalake/assets.json`
+- [`sdk/data/aztables/assets.json`](https://github.com/Azure/azure-sdk-for-go/blob/main/sdk/data/aztables/assets.json)
+- [`sdk/keyvault/azure-keyvault-keys/assets.json`](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/keyvault/azure-keyvault-keys/assets.json)
 
 The location of the actual test code is referred to as the `language repo`.
 

diff --git a/eng/common/testproxy/transition-scripts/generate-assets-json.ps1 b/eng/common/testproxy/transition-scripts/generate-assets-json.ps1
@@ -72,7 +72,7 @@ if ($UseTestRepo) {
 # 3. ios
 $LangRecordingDirs = @{"cpp" = "recordings";
   "go"                       = "recordings";
-  "java"                     = "session-records";
+  "java"                     = "src.*?session-records";
   "js"                       = "recordings";
   "net"                      = "SessionRecords";
   "python"                   = "recordings";
@@ -321,8 +321,8 @@ Function Move-AssetsFromLangRepo {
   )
   $filter = $LangRecordingDirs[$language]
   Write-Host "Language recording directory name=$filter"
-  Write-Host "Get-ChildItem -Recurse -Filter ""*.json"" | Where-Object { `$_.DirectoryName.Split([IO.Path]::DirectorySeparatorChar) -contains ""$filter"" }"
-  $filesToMove = Get-ChildItem -Recurse -Filter "*.json" | Where-Object { $_.DirectoryName.Split([IO.Path]::DirectorySeparatorChar) -contains "$filter" }
+  Write-Host "Get-ChildItem -Recurse -Filter ""*.json"" | Where-Object { if ($filter.Contains(""*"")) { $_.DirectoryName -match $filter } else { $_.DirectoryName.Split([IO.Path]::DirectorySeparatorChar) -contains ""$filter"" }"
+  $filesToMove = Get-ChildItem -Recurse -Filter "*.json" | Where-Object { if ($filter.Contains("*")) { $_.DirectoryName -match $filter } else { $_.DirectoryName.Split([IO.Path]::DirectorySeparatorChar) -contains "$filter" } }
   [string] $currentDir = Get-Location
 
   foreach ($fromFile in $filesToMove) {

diff --git a/eng/pipelines/templates/jobs/archetype-sdk-client.yml b/eng/pipelines/templates/jobs/archetype-sdk-client.yml
@@ -1,3 +1,6 @@
+# cSpell:ignore vsts
+# cSpell:ignore parseazuresdkcpp
+
 parameters:
   - name: Artifacts
     type: object
@@ -84,6 +87,20 @@ jobs:
 
           - template: /eng/common/pipelines/templates/steps/set-default-branch.yml
 
+          - task:  UniversalPackages@0
+            displayName: Download ApiView Parser.
+            inputs:
+              command: download
+              vstsFeed: 'internal/AzureSDKForCpp'
+              vstsFeedPackage: 'parseazuresdkcpp'
+              downloadDirectory: '$(System.DefaultWorkingDirectory)/parser'
+              vstsPackageVersion: 0.6.0
+            condition: >-
+                and(
+                  succeeded(),
+                  ne(variables['Skip.CreateApiReview'], 'true'),eq(variables['System.TeamProject'], 'internal')
+                )
+
           - task: Powershell@2
             inputs:
               filePath: $(System.DefaultWorkingDirectory)/eng/scripts/Create-APIReview.ps1
@@ -96,6 +113,8 @@ jobs:
                 -SourceBranch $(Build.SourceBranchName)
                 -DefaultBranch $(DefaultBranch)
                 -ConfigFileDir $(Build.ArtifactStagingDirectory)/PackageInfo
+                -ParserPath $(System.DefaultWorkingDirectory)/parser/RelWithDebInfo/ParseAzureSdkCpp.exe
+                -SourcePath $(Build.SourcesDirectory)/sdk/${{ parameters.ServiceDirectory }}/${{ artifact.name }}/inc
               pwsh: true
               workingDirectory: $(Pipeline.Workspace)
             displayName: Create API Review for ${{ artifact.name }}

diff --git a/eng/scripts/Create-APIReview.ps1 b/eng/scripts/Create-APIReview.ps1
@@ -14,30 +14,31 @@ Param(
     [Parameter(Mandatory=$True)]
     [string] $DefaultBranch,
     [Parameter(Mandatory=$True)]
-    [string] $ConfigFileDir
+    [string] $ConfigFileDir,
+    [Parameter(Mandatory=$True)]
+    [string] $ParserPath,
+    [Parameter(Mandatory=$True)]
+    [string] $SourcePath
 )
 
 Write-Host "$PSScriptRoot"
 . (Join-Path $PSScriptRoot .. common scripts common.ps1)
 $createReviewScript = (Join-Path $PSScriptRoot .. common scripts Create-APIReview.ps1)
-Set-Location $PSScriptRoot
 
-Write-Host "Creating API review artifact for $ArtifactName"
-New-Item -ItemType Directory -Path $OutPath/$ArtifactName -force
+$apiviewSettings = Join-Path $SourcePath "ApiViewSettings.json"
+if (!(Test-Path $apiviewSettings))
+{
+    Write-Host "ApiViewSettings.json file is not found in $($SourcePath). APIView settings file is required to generate API review file."
+    exit 1
+}
 
-$gitroot = Join-Path $PSScriptRoot .. ..
-Write-Host "Get-ApiViewCommandLine.ps1 $gitroot $ArtifactName"
-$cmdLine = & $PSScriptRoot/Get-ApiViewCommandLine.ps1 $gitroot $ArtifactName
-Write-Host "Executing clang++ command:"
-Write-Host $cmdLine
-$cmd, $cmdArgs = $cmdLine -split ' '
-# Get-ApiViewCommandLine.ps1 returns a string representing a clang++ command that needs to be run, e.g.
-# clang++ <space separated list of header files> -Xclang -ast-dump -I <space separated list of header files>
-# ApiView expects a zip of this ast as the format for a C++ language artifact.
-& $cmd $cmdArgs > clangAstOutput
+Write-Host "Creating API review artifact for $($ArtifactName)"
+New-Item -ItemType Directory -Path $OutPath/$ArtifactName -force
+$parentPath = Split-Path $ParserPath  -Parent
+Write-Host "Contents in $($parentPath)"
+Get-ChildItem -Path $parentPath -Recurse
 
-Compress-Archive -Path clangAstOutput -DestinationPath $OutPath/$ArtifactName/$ArtifactName
-Rename-Item $OutPath/$ArtifactName/$ArtifactName.zip -NewName "$ArtifactName.cppast"
+& $ParserPath -o $OutPath/$ArtifactName/$ArtifactName.json $SourcePath
 
 Write-Host "Send request to APIView to create review for $ArtifactName"
 &($createReviewScript) -ArtifactPath $OutPath -APIViewUri $ApiviewUri -APIKey $ApiKey -APILabel $ApiLabel -PackageName $ArtifactName -SourceBranch $SourceBranch -DefaultBranch $DefaultBranch -ConfigFileDir $ConfigFileDir
diff --git a/eng/scripts/Language-Settings.ps1 b/eng/scripts/Language-Settings.ps1
@@ -101,7 +101,7 @@ function SetPackageVersion ($PackageName, $Version, $ServiceDirectory, $ReleaseD
 
 function Find-cpp-Artifacts-For-Apireview($ArtifactPath, $PackageName)
 {
-  $artifact = Get-ChildItem -Path (Join-Path $ArtifactPath $PackageName) -Filter "*.cppast"
+  $artifact = Get-ChildItem -Path (Join-Path $ArtifactPath $PackageName) -Filter "*.json"
   if ($artifact)
   {
     $packages = @{

diff --git a/sdk/attestation/assets.json b/sdk/attestation/assets.json
@@ -2,5 +2,5 @@
   "AssetsRepo": "Azure/azure-sdk-assets",
   "AssetsRepoPrefixPath": "cpp",
   "TagPrefix": "cpp/attestation",
-  "Tag": "cpp/attestation_10abfdc3e0"
+  "Tag": "cpp/attestation_6398169251"
 }
diff --git a/sdk/attestation/azure-security-attestation/test/ut/policycertmgmt_test.cpp b/sdk/attestation/azure-security-attestation/test/ut/policycertmgmt_test.cpp
@@ -171,8 +171,6 @@ namespace Azure { namespace Security { namespace Attestation { namespace Test {
 
   TEST_F(CertificateTests, AddPolicyManagementCertificate_LIVEONLY_)
   {
-    CHECK_SKIP_TEST()
-
     auto adminClient(CreateClient(ServiceInstanceType::Isolated));
 
     auto isolatedCertificateBase64(GetEnv("ISOLATED_SIGNING_CERTIFICATE"));
@@ -233,8 +231,6 @@ namespace Azure { namespace Security { namespace Attestation { namespace Test {
 
   TEST_F(CertificateTests, RemovePolicyManagementCertificate_LIVEONLY_)
   {
-    CHECK_SKIP_TEST()
-
     auto adminClient(CreateClient(ServiceInstanceType::Isolated));
 
     auto isolatedCertificateBase64(GetEnv("ISOLATED_SIGNING_CERTIFICATE"));