Merge branch 'master' of https://github.com/Azure/azure-sdk-for-python …

…into regenerate_keys

* 'master' of https://github.com/Azure/azure-sdk-for-python:
  Set http_logging_policy in Configuration (Azure#12218)
  Sync eng/common directory with azure-sdk-tools repository (Azure#11990)
  AzureCliCredential instructs CLI not to color output (Azure#11362)
  Sdk automation/track2 azure mgmt storage (Azure#12238)
  Fix changelog of CustomVision (Azure#12225)
  Doc update for conf file name (Azure#12224)
  update doc for content_type (Azure#12220)
  update API version to use 2020-06-30 (Azure#12208)
  Use MSAL's custom transport API (Azure#11892)
  add breadcumbs for training filter (Azure#12196)
  [formrecognizer] arch board feedback renames (Azure#12207)
  Dataplane autogeneration (Azure#12210)
  update version number and API_version support (Azure#12154)
  Update SECURITY.md (Azure#12209)
  adding a pip freeze to ensure we fully understand what our environment has (Azure#12173)
  FaceAPI 0.4.1 (Azure#12199)
  [ServiceBus] Enable Async Unittest (Azure#12001)
  add search owver (Azure#12190)
  update training docstring (Azure#12168)

.github/CODEOWNERS

@@ -33,6 +33,7 @@
 /sdk/hdinsight/                                                      @idear1203
 /sdk/machinelearningcompute/                                         @shutchings
 /sdk/recoveryservices/                                               @DheerendraRathor
+/sdk/search/                                                         @xiangyan99
 /sdk/servicefabric/                                                  @QingChenmsft @samedder
 /sql/sql/                                                            @jaredmoo
 /sdk/servicebus/                                                     @annatisch @yunhaoling @YijunXieMS @KieranBrantnerMagee

SECURITY.md

@@ -1,8 +1,41 @@
-# Security Policy
+<!-- BEGIN MICROSOFT SECURITY.MD V0.0.5 BLOCK -->
 
-## Reporting a Vulnerability
+## Security
 
-Security issues and bugs should be reported privately, via email, to the Microsoft Security Response Center (MSRC) <[email protected]>. You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Further information, including the MSRC PGP key, can be found in the [Security TechCenter](https://www.microsoft.com/msrc/faqs-report-an-issue).
+Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/).
 
-Please do not open issues for anything you think might have a security implication.
+If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://docs.microsoft.com/en-us/previous-versions/tn-archive/cc751383(v=technet.10)), please report it to us as described below.
 
+## Reporting Security Issues
+
+**Please do not report security vulnerabilities through public GitHub issues.**
+
+Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://msrc.microsoft.com/create-report).
+
+If you prefer to submit without logging in, send email to [[email protected]](mailto:[email protected]).  If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://www.microsoft.com/en-us/msrc/pgp-key-msrc).
+
+You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://www.microsoft.com/msrc). 
+
+Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue:
+
+  * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.)
+  * Full paths of source file(s) related to the manifestation of the issue
+  * The location of the affected source code (tag/branch/commit or direct URL)
+  * Any special configuration required to reproduce the issue
+  * Step-by-step instructions to reproduce the issue
+  * Proof-of-concept or exploit code (if possible)
+  * Impact of the issue, including how an attacker might exploit the issue
+
+This information will help us triage your report more quickly.
+
+If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://microsoft.com/msrc/bounty) page for more details about our active programs.
+
+## Preferred Languages
+
+We prefer all communications to be in English.
+
+## Policy
+
+Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://www.microsoft.com/en-us/msrc/cvd).
+
+<!-- END MICROSOFT SECURITY.MD BLOCK -->

doc/dev/mgmt/mgmt_release.md

@@ -18,6 +18,22 @@ Once you have a PR that contains accurate with correct tests (or no tests at all
 
 IMPORTANT NOTE: All the commands in this page assumes you have loaded the [dev_setup](../dev_setup.md) in your currently loaded virtual environment.
 
+## Manual generation
+
+If the automation is not doing its job to create an auto PR, Python has a SwaggerToSdk CLI that can be used to generate a specific Readme. You need
+a virtual environment loaded with at least `tools/azure-sdk-tools` installed.
+
+```shell
+# Using default configuration (this can be a Github raw link)
+generate_sdk -v -m ..\azure-rest-api-specs\specification\compute\resource-manager\readme.md
+
+# Forcing Track1 generation
+generate_sdk -v -c eng\swagger_to_sdk_config_v4.json -m ..\azure-rest-api-specs\specification\cognitiveservices\data-plane\Face\readme.md
+
+# For more details about the available options
+generate_sdk --help
+```
+
 ## Building the packaging information
 
 If the automation is doing its job correctly, there is a pipeline called "update PR" that is supposed to update the package on the branch.

eng/common/TestResources/New-TestResources.ps1.md

@@ -412,6 +412,23 @@ Accept pipeline input: False
 Accept wildcard characters: False
 ```
 
+### -OutFile
+save test environment settings into a test-resources.json.env file next to test-resources.json. 
+The file is protected via DPAPI. The environment file would be scoped to the current repository directory.
+Note: Supported only on Windows.
+
+```yaml
+Type: SwitchParameter
+Parameter Sets: (All)
+Aliases:
+
+Required: False
+Position: Named
+Default value: None
+Accept pipeline input: False
+Accept wildcard characters: False
+```
+
 ### CommonParameters
 This cmdlet supports the common parameters: -Debug, -ErrorAction, -ErrorVariable, -InformationAction, -InformationVariable, -OutVariable, -OutBuffer, -PipelineVariable, -Verbose, -WarningAction, and -WarningVariable. For more information, see [about_CommonParameters](http://go.microsoft.com/fwlink/?LinkID=113216).
 

eng/common/TestResources/README.md

@@ -35,6 +35,12 @@ eng\common\TestResources\New-TestResources.ps1 `
   -TestApplicationSecret (ConvertFrom-SecureString $sp.Secret -AsPlainText)
 ```
 
+If you are running this for a .NET project on Windows, the recommended method is to 
+add the `-OutFile` switch to the above command. This will save test environment settings 
+into a test-resources.json.env file next to test-resources.json. The file is protected via DPAPI. 
+The environment file would be scoped to the current repository directory and avoids the need to 
+set environment variables or restart your IDE to recognize them.
+
 Along with some log messages, this will output environment variables based on
 your current shell like in the following example:
 

eng/common/pipelines/templates/steps/create-pull-request.yml

@@ -35,27 +35,31 @@ steps:
   workingDirectory: ${{ parameters.WorkingDirectory }}
   ignoreLASTEXITCODE: true
 
-- pwsh: |
-    ${{ parameters.ScriptDirectory }}/git-branch-push.ps1 `
-      -PRBranchName "${{ parameters.PRBranchName }}" `
-      -CommitMsg "${{ parameters.CommitMsg }}" `
-      -GitUrl "https://$(azuresdk-github-pat)@github.com/${{ parameters.PROwner }}/${{ parameters.RepoName }}.git" `
-      -PushArgs "${{ parameters.PushArgs }}"
-
+- task: PowerShell@2
   displayName: Push changes
-  workingDirectory: ${{ parameters.WorkingDirectory }}
   condition: and(succeeded(), eq(variables['HasChanges'], 'true'))
+  inputs:
+    pwsh: true
+    workingDirectory: ${{ parameters.WorkingDirectory }}
+    filePath: ${{ parameters.ScriptDirectory }}/git-branch-push.ps1
+    arguments: >
+      -PRBranchName "${{ parameters.PRBranchName }}"
+      -CommitMsg "${{ parameters.CommitMsg }}"
+      -GitUrl "https://$(azuresdk-github-pat)@github.com/${{ parameters.PROwner }}/${{ parameters.RepoName }}.git"
+      -PushArgs "${{ parameters.PushArgs }}"
 
-- pwsh: |
-    ${{ parameters.ScriptDirectory }}/Submit-PullRequest.ps1 `
-      -RepoOwner "${{ parameters.RepoOwner }}" `
-      -RepoName "${{ parameters.RepoName }}" `
-      -BaseBranch "${{ parameters.BaseBranchName }}" `
-      -PROwner "${{ parameters.PROwner }}" `
-      -PRBranch "${{ parameters.PRBranchName }}" `
-      -AuthToken "$(azuresdk-github-pat)" `
-      -PRTitle "${{ parameters.PRTitle }}"
-
+- task: PowerShell@2
   displayName: Create pull request
-  workingDirectory: ${{ parameters.WorkingDirectory }}
-  condition: and(succeeded(), eq(variables['HasChanges'], 'true'))
+  condition: and(succeeded(), eq(variables['HasChanges'], 'true'))
+  inputs:
+    pwsh: true
+    workingDirectory: ${{ parameters.WorkingDirectory }}
+    filePath: ${{ parameters.ScriptDirectory }}/Submit-PullRequest.ps1
+    arguments: >
+      -RepoOwner "${{ parameters.RepoOwner }}"
+      -RepoName "${{ parameters.RepoName }}"
+      -BaseBranch "${{ parameters.BaseBranchName }}"
+      -PROwner "${{ parameters.PROwner }}"
+      -PRBranch "${{ parameters.PRBranchName }}"
+      -AuthToken "$(azuresdk-github-pat)"
+      -PRTitle "${{ parameters.PRTitle }}"

eng/common/pipelines/templates/steps/create-tags-and-git-release.yml

@@ -10,9 +10,13 @@ steps:
 - task: PowerShell@2
   displayName: 'Verify Package Tags and Create Git Releases'
   inputs:
-    targetType: filePath
     filePath: ${{ parameters.ScriptDirectory }}/create-tags-and-git-release.ps1
-    arguments: -artifactLocation ${{parameters.ArtifactLocation}} -packageRepository ${{parameters.PackageRepository}} -releaseSha ${{parameters.ReleaseSha}} -repoId ${{parameters.RepoId}} -workingDirectory '${{parameters.WorkingDirectory}}'
+    arguments: >
+      -artifactLocation ${{ parameters.ArtifactLocation }}
+      -packageRepository ${{ parameters.PackageRepository }}
+      -releaseSha ${{ parameters.ReleaseSha }}
+      -repoId ${{ parameters.RepoId }}
+      -workingDirectory '${{ parameters.WorkingDirectory }}'
     pwsh: true
   timeoutInMinutes: 5
   env:

eng/common/pipelines/templates/steps/docs-metadata-release.yml

@@ -16,10 +16,10 @@ parameters:
 steps:
 - pwsh: |
     git clone https://github.com/${{ parameters.TargetDocRepoOwner }}/${{ parameters.TargetDocRepoName }} ${{ parameters.WorkingDirectory }}/repo
-    
+
     try {
       Push-Location ${{ parameters.WorkingDirectory }}/repo
-    
+
       Write-Host "git checkout smoke-test"
       git checkout smoke-test
     } finally {
@@ -33,14 +33,14 @@ steps:
   inputs:
     targetType: filePath
     filePath: ${{ parameters.ScriptDirectory }}/update-docs-metadata.ps1
-    arguments: > 
-      -ArtifactLocation ${{parameters.ArtifactLocation}} 
-      -Repository ${{parameters.PackageRepository}} 
-      -ReleaseSHA ${{parameters.ReleaseSha}} 
-      -RepoId ${{parameters.RepoId}} 
-      -WorkDirectory '${{parameters.WorkingDirectory}}' 
-      -DocRepoLocation "${{parameters.WorkingDirectory}}/repo" 
-      -Language "${{parameters.Language}}" 
+    arguments: >
+      -ArtifactLocation ${{ parameters.ArtifactLocation }}
+      -Repository ${{ parameters.PackageRepository }}
+      -ReleaseSHA ${{ parameters.ReleaseSha }}
+      -RepoId ${{ parameters.RepoId }}
+      -WorkDirectory "${{ parameters.WorkingDirectory }}"
+      -DocRepoLocation "${{ parameters.WorkingDirectory }}/repo"
+      -Language "${{parameters.Language}}"
       -DocRepoContentLocation ${{ parameters.DocRepoDestinationPath }}
     pwsh: true
   env:
@@ -54,5 +54,5 @@ steps:
     CommitMsg: "Update readme content for ${{ parameters.ArtifactName }}"
     PRTitle: "Docs.MS Readme Update."
     BaseBranchName: smoke-test
-    WorkingDirectory: ${{parameters.WorkingDirectory}}/repo
-    ScriptDirectory: ${{parameters.WorkingDirectory}}/${{parameters.ScriptDirectory}}
+    WorkingDirectory: ${{ parameters.WorkingDirectory }}/repo
+    ScriptDirectory: ${{ parameters.WorkingDirectory }}/${{ parameters.ScriptDirectory }}

eng/common/pipelines/templates/steps/publish-blobs.yml

@@ -7,16 +7,20 @@ parameters:
 
 steps:
 - pwsh: |
-    Invoke-WebRequest -MaximumRetryCount 10 -Uri "https://aka.ms/downloadazcopy-v10-windows" `
-    -OutFile "azcopy.zip" | Wait-Process; Expand-Archive -Path "azcopy.zip" -DestinationPath "$(Build.BinariesDirectory)/azcopy/"
+    Invoke-WebRequest -MaximumRetryCount 10 -Uri "https://aka.ms/downloadazcopy-v10-windows" -OutFile "azcopy.zip" | Wait-Process;
+    Expand-Archive -Path "azcopy.zip" -DestinationPath "$(Build.BinariesDirectory)/azcopy/"
   workingDirectory: $(Build.BinariesDirectory)
   displayName: Download and Extract azcopy Zip
 
 - task: Powershell@2
   inputs:
-    targetType: 'filePath'
     filePath: ${{ parameters.ScriptPath }}
-    arguments: -AzCopy $(Resolve-Path "$(Build.BinariesDirectory)/azcopy/azcopy_windows_amd64_*/azcopy.exe")[0] -DocLocation "${{ parameters.FolderForUpload }}" -SASKey "${{ parameters.BlobSASKey }}" -Language "${{ parameters.TargetLanguage }}" -BlobName "${{ parameters.BlobName }}"
+    arguments: >
+      -AzCopy $(Resolve-Path "$(Build.BinariesDirectory)/azcopy/azcopy_windows_amd64_*/azcopy.exe")[0]
+      -DocLocation "${{ parameters.FolderForUpload }}"
+      -SASKey "${{ parameters.BlobSASKey }}"
+      -Language "${{ parameters.TargetLanguage }}"
+      -BlobName "${{ parameters.BlobName }}"
     pwsh: true
     workingDirectory: $(Pipeline.Workspace)
   displayName: Copy Docs to Blob

eng/common/pipelines/templates/steps/verify-agent-os.yml

@@ -1,4 +1,3 @@
-# Template for all Python Scripts in this repository
 parameters: 
   OSVmImage: $(OSVmImage)
 

eng/common/pipelines/templates/steps/verify-changelog.yml

@@ -0,0 +1,25 @@
+parameters:
+- name: PackageName
+  type: string
+  default: 'not-specified'
+- name: ServiceName
+  type: string
+  default: 'not-specified'
+- name: ForRelease
+  type: boolean
+  default: false
+
+steps:
+  - task: Powershell@2
+    inputs:
+      filePath: /eng/common/scripts/Verify-ChangeLog.ps1
+      arguments: >
+        -PackageName ${{ parameters.PackageName }}
+        -ServiceName ${{ parameters.ServiceName }}
+        -RepoRoot $(Build.SourcesDirectory)
+        -RepoName $(Build.Repository.Name)
+        -ForRelease ${{ parameters.ForRelease }}
+      pwsh: true
+      workingDirectory: $(Pipeline.Workspace)
+    displayName: Verify ChangeLog / Release Notes
+    continueOnError: false

eng/common/pipelines/templates/steps/verify-links.yml

@@ -0,0 +1,12 @@
+parameters:
+  Directory: 'not-specified'
+
+steps:
+  - task: PowerShell@2
+    displayName: Link verification check
+    inputs:
+      pwsh: true
+      workingDirectory: $(Build.SourcesDirectory)/${{ parameters.Directory }}
+      filePath: eng/common/scripts/Verify-Links.ps1 
+      arguments: >
+        -urls $(dir -r -i *.md) -rootUrl "file://$(Build.SourcesDirectory)/${{ parameters.Directory }}"

eng/common/scripts/Submit-PullRequest.ps1

@@ -41,21 +41,25 @@ param(
   $PRBody = $PRTitle
 )
 
-Write-Host "> $PSCommandPath $args"
+$headers = @{
+  Authorization = "bearer $AuthToken"
+}
 
 $query = "state=open&head=${PROwner}:${PRBranch}&base=${BaseBranch}"
 
-$resp = Invoke-RestMethod "https://api.github.com/repos/$RepoOwner/$RepoName/pulls?$query"
+try {
+  $resp = Invoke-RestMethod -Headers $headers "https://api.github.com/repos/$RepoOwner/$RepoName/pulls?$query"
+}
+catch { 
+  Write-Error "Invoke-RestMethod [https://api.github.com/repos/$RepoOwner/$RepoName/pulls?$query] failed with exception:`n$_"
+  exit 1
+}
 $resp | Write-Verbose
 
 if ($resp.Count -gt 0) {
     Write-Host -f green "Pull request already exists $($resp[0].html_url)"
 }
 else {
-  $headers = @{
-    Authorization = "bearer $AuthToken"
-  }
-
   $data = @{
     title                 = $PRTitle
     head                  = "${PROwner}:${PRBranch}"
@@ -64,9 +68,16 @@ else {
     maintainer_can_modify = $true
   }
 
-  $resp = Invoke-RestMethod -Method POST -Headers $headers `
-                            https://api.github.com/repos/$RepoOwner/$RepoName/pulls `
-                            -Body ($data | ConvertTo-Json)
+  try {
+    $resp = Invoke-RestMethod -Method POST -Headers $headers `
+                              "https://api.github.com/repos/$RepoOwner/$RepoName/pulls" `
+                              -Body ($data | ConvertTo-Json)
+  }
+  catch {
+    Write-Error "Invoke-RestMethod [https://api.github.com/repos/$RepoOwner/$RepoName/pulls] failed with exception:`n$_"
+    exit 1
+  }
+
   $resp | Write-Verbose
   Write-Host -f green "Pull request created https://github.com/$RepoOwner/$RepoName/pull/$($resp.number)"
 }

eng/common/scripts/Verify-ChangeLog.ps1

@@ -0,0 +1,22 @@
+# Wrapper Script for ChangeLog Verification
+param (
+    [Parameter(Mandatory=$true)]
+    [string]$PackageName,
+    [Parameter(Mandatory=$true)]
+    [string]$ServiceName,
+    [string]$RepoRoot,
+    [ValidateSet("net","java","js","python")]
+    [string]$Language,
+    [string]$RepoName,
+    [boolean]$ForRelease=$False
+)
+
+Import-Module "${PSScriptRoot}/modules/common-manifest.psd1"
+
+if ([System.String]::IsNullOrEmpty($Language))
+{
+    $Language = $RepoName.Substring($RepoName.LastIndexOf('-') + 1)
+}
+
+$PackageProp = Get-PkgProperties -PackageName $PackageName -ServiceName $ServiceName -Language $Language -RepoRoot $RepoRoot
+Confirm-ChangeLog -ChangeLogLocation $PackageProp.pkgChangeLogPath -VersionString $PackageProp.pkgReadMePath -ForRelease $ForRelease