Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
…into add_webpubsub_tests

* 'main' of https://github.com/Azure/azure-sdk-for-python:
  [Key Vault] Add support for multi-tenant authentication (Azure#21290)
  [webpubsub] regen with hub as a client parameter (Azure#21688)
  update automatic close mechanism (Azure#21580)
  [Test Proxy] Add fixture to automatically start/stop Docker container (Azure#21538)
  Update Monitor Query API ref link (Azure#21683)
  Migration Guide from Azure-loganalytics (Azure#21674)
  Update docs for Web PubSub GA (Azure#21659)
  Update CHANGELOG.md (Azure#21681)
  Increment version for formrecognizer releases (Azure#21678)
  Increment version for videoanalyzer releases (Azure#21455)
  Increment version for cognitivelanguage releases (Azure#21566)
  Increment version for storage releases (Azure#21652)
  Increment version for communication releases (Azure#21667)
  raise decode error instead of ContentDecodingError (Azure#19433)
  Update CHANGELOG.md (Azure#21679)
  resolve mac agent failure (Azure#21677)
  Re-add get-codeowners.ps1 (Azure#21676)
  [SchemaRegistry] remove schema prefix in params (Azure#21675)
  Validate python docs packages using docker (Azure#21657)
  update git helper (Azure#21670)
  • Loading branch information
iscai-msft committed Nov 10, 2021
2 parents d547bd7 + 0a0cc97 commit fbb680e
Show file tree
Hide file tree
Showing 107 changed files with 2,642 additions and 1,066 deletions.
49 changes: 49 additions & 0 deletions eng/common/scripts/get-codeowners.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
param (
$TargetDirectory, # should be in relative form from root of repo. EG: sdk/servicebus
$RootDirectory, # ideally $(Build.SourcesDirectory)
$VsoVariable = "" # target devops output variable
)
$target = $TargetDirectory.ToLower().Trim("/")
$codeOwnersLocation = Join-Path $RootDirectory -ChildPath ".github/CODEOWNERS"
$ownedFolders = @{}

if (!(Test-Path $codeOwnersLocation)) {
Write-Host "Unable to find CODEOWNERS file in target directory $RootDirectory"
exit 1
}

$codeOwnersContent = Get-Content $codeOwnersLocation

foreach ($contentLine in $codeOwnersContent) {
if (-not $contentLine.StartsWith("#") -and $contentLine){
$splitLine = $contentLine -split "\s+"

# CODEOWNERS file can also have labels present after the owner aliases
# gh aliases start with @ in codeowners. don't pass on to API calls
$ownedFolders[$splitLine[0].ToLower().Trim("/")] = ($splitLine[1..$($splitLine.Length)] `
| ? { $_.StartsWith("@") } `
| % { return $_.substring(1) }) -join ","
}
}

$results = $ownedFolders[$target]

if ($results) {
Write-Host "Found a folder $results to match $target"

if ($VsoVariable) {
$alreadyPresent = [System.Environment]::GetEnvironmentVariable($VsoVariable)

if ($alreadyPresent) {
$results += ",$alreadyPresent"
}
Write-Host "##vso[task.setvariable variable=$VsoVariable;]$results"
}

return $results
}
else {
Write-Host "Unable to match path $target in CODEOWNERS file located at $codeOwnersLocation."
Write-Host ($ownedFolders | ConvertTo-Json)
return ""
}
14 changes: 2 additions & 12 deletions eng/common/testproxy/test-proxy-tool.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,17 +15,11 @@ steps:
Write-Host "##vso[task.setvariable variable=ASPNETCORE_Kestrel__Certificates__Default__Password]password"
displayName: 'Configure Kestrel Environment Variables'
- task: UseDotNet@2
displayName: "Use .NET Core SDK"
inputs:
packageType: sdk
version: 5.0.205

- pwsh: |
dotnet tool install azure.sdk.tools.testproxy `
--tool-path $(Build.BinariesDirectory)/test-proxy `
--add-source https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-net/nuget/v3/index.json `
--version 1.0.0-dev.20210811.2
--version 1.0.0-dev.20211104.2
displayName: "Install test-proxy"
- pwsh: |
Expand All @@ -37,11 +31,7 @@ steps:
# nohup does NOT continue beyond the current session if you use it within powershell
- bash: |
sudo nohup $(Build.BinariesDirectory)/test-proxy/test-proxy &
nohup $(Build.BinariesDirectory)/test-proxy/test-proxy &
displayName: "Run the testproxy - linux/mac"
condition: and(succeeded(), ne(variables['Agent.OS'],'Windows_NT'))
workingDirectory: "${{ parameters.rootFolder }}"
- pwsh: |
Write-Host "##vso[task.setvariable variable=PATH]$(OriginalPath)"
displayName: 'Restore .NET version by resetting path'
11 changes: 9 additions & 2 deletions eng/pipelines/docindex.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ jobs:
DocRepoLocation: $(Pipeline.Workspace)/docs
DocRepoOwner: MicrosoftDocs
DocRepoName: azure-docs-sdk-python
DocValidationImageId: azuresdkimages.azurecr.io/pyrefautocr:latest
steps:
# Docs CI uses Python 3.6.8 but that is not available on this image
- task: UsePythonVersion@0
Expand All @@ -21,6 +22,12 @@ jobs:
- pwsh: pip install --upgrade pip wheel setuptools
displayName: Update python tools for package verification

# Pull and build the docker image.
- template: /eng/common/pipelines/templates/steps/docker-pull-image.yml
parameters:
ContainerRegistryClientId: $(azuresdkimages-cr-clientid)
ContainerRegistryClientSecret: $(azuresdkimages-cr-clientsecret)
ImageId: "$(DocValidationImageId)"
# Sync docs repo onboarding files/folders
- template: /eng/common/pipelines/templates/steps/sparse-checkout.yml
parameters:
Expand All @@ -38,7 +45,7 @@ jobs:
inputs:
pwsh: true
filePath: eng/common/scripts/Update-DocsMsPackages.ps1
arguments: -DocRepoLocation $(DocRepoLocation)
arguments: -DocRepoLocation $(DocRepoLocation) -ImageId '$(DocValidationImageId)'
displayName: Update Docs Onboarding
condition: and(succeeded(), or(eq(variables['Build.Reason'], 'Schedule'), eq(variables['Force.MainUpdate'], 'true')))
# Push changes to docs repo
Expand Down Expand Up @@ -70,7 +77,7 @@ jobs:
inputs:
pwsh: true
filePath: eng/common/scripts/Update-DocsMsPackages.ps1
arguments: -DocRepoLocation $(DocRepoLocation) -PackageSourceOverride "https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-python/pypi/simple/"
arguments: -DocRepoLocation $(DocRepoLocation) -PackageSourceOverride "https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-python/pypi/simple/" -ImageId '$(DocValidationImageId)'
displayName: Update Docs Onboarding for Daily branch
- template: /eng/common/pipelines/templates/steps/git-push-changes.yml
parameters:
Expand Down
39 changes: 32 additions & 7 deletions eng/scripts/Language-Settings.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -162,18 +162,43 @@ function Get-python-GithubIoDocIndex()
}

function ValidatePackage($packageName, $packageVersion, $workingDirectory) {
$packageExpression = "$packageName$packageVersion"
Write-Host "Validating $packageExpression"

$installTargetFolder = Join-Path $workingDirectory $packageName
New-Item -ItemType Directory -Force -Path $installTargetFolder | Out-Null

# Add more validation by replicating as much of the docs CI process as
# possible
# https://github.com/Azure/azure-sdk-for-python/issues/20109
if (!$ImageId) {
Write-Host "Validating using pip command directly on $packageName."
FallbackValidation -packageName "$packageName" -packageVersion "$packageVersion" -workingDirectory $workingDirectory
}
else {
Write-Host "Validating using $ImageId on $packageName."
DockerValidation -packageName "$packageName" -packageVersion "$packageVersion"
}
}
function DockerValidation($packageName, $packageVersion) {
$packageExpression = "$packageName==$packageVersion"
docker run -e TARGET_PACKAGE=$packageExpression -t $ImageId
# The docker exit codes: https://docs.docker.com/engine/reference/run/#exit-status
# If the docker failed because of docker itself instead of the application,
# we should skip the validation and keep the packages.
if ($LASTEXITCODE -eq 125 -Or $LASTEXITCODE -eq 126 -Or $LASTEXITCODE -eq 127) {
Write-Host $commandLine
LogWarning "The `docker` command does not work with exit code $LASTEXITCODE. Fall back to npm install $packageName directly."
FallbackValidation -packageName "$packageName" -packageVersion "$packageVersion"
}
elseif ($LASTEXITCODE -ne 0) {
Write-Host $commandLine
LogWarning "Package $($Package.name) ref docs validation failed."
return $false
}
return $true
}

function FallbackValidation($packageName, $packageVersion, $workingDirectory) {
$installTargetFolder = Join-Path $workingDirectory $packageName
New-Item -ItemType Directory -Force -Path $installTargetFolder | Out-Null
$packageExpression = "$packageName$packageVersion"
try {
$pipInstallOutput = ""
$extraIndexUrl = " --extra-index-url=$PackageSourceOverride"
if ($PackageSourceOverride) {
Write-Host "pip install $packageExpression --no-cache-dir --target $installTargetFolder --extra-index-url=$PackageSourceOverride"
$pipInstallOutput = pip `
Expand Down
2 changes: 1 addition & 1 deletion scripts/devops_tasks/git_helper.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
'azure-cosmos': '3.2.0',
'azure-servicebus': '0.50.3',
'azure-eventgrid': '1.3.0',
'azure-schemaregistry-avroserializer': '1.0.0b2',
'azure-schemaregistry-avroserializer': '1.0.0b3',
'azure-storage-blob-changefeed' : '12.0.0b2',
'azure-storage-file-datalake': '12.2.0b1',
'azure-communication-identity': '1.0.0',
Expand Down
70 changes: 36 additions & 34 deletions scripts/release_issue_status/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,7 @@

import reply_generator as rg
from utils import update_issue_body, get_readme_and_output_folder, \
get_python_pipelines, get_pipeline_url, auto_close_issue

get_python_pipelines, get_pipeline_url, auto_close_issue

_NULL = ' '
_FILE_OUT = 'release_issue_status.csv'
Expand All @@ -23,6 +22,7 @@
logging.basicConfig(level=logging.INFO,
format='[auto-reply log] - %(funcName)s[line:%(lineno)d] - %(levelname)s: %(message)s')


def my_print(cmd):
print('==' + cmd + ' ==\n')

Expand All @@ -39,12 +39,13 @@ def output_python_md(issue_status_python):
file_out.write('| ------ | ------ | ------ | ------ | ------ | ------ | ------ | :-----: |\n')
file_out.writelines([item.output_python() for item in sorted(issue_status_python, key=_key_select)])


def output_csv(issue_status):
with open(_FILE_OUT, 'w') as file_out:
file_out.write('language,issue,author,package,created date,delay from created date,latest update time,'
'delay from latest update,status,bot advice\n')
file_out.writelines([item.output() for item in sorted(issue_status, key=_key_select)])


class IssueStatus:
link = _NULL
Expand Down Expand Up @@ -75,7 +76,6 @@ def output(self):
self.delay_from_latest_update,
self.status, self.bot_advice)


def output_python(self):
package = self.package.split('-')[-1]
create_date = str(date.fromtimestamp(self.create_date).strftime('%m-%d'))
Expand All @@ -85,12 +85,13 @@ def output_python(self):
else:
days_from_target = ' '

return '| [#{}]({}) | {} | {} | {} | {} | {} | {} | {} |\n'.format(self.link.split('/')[-1], self.link, self.author,
package, self.assignee, self.bot_advice,
create_date,
target_date,
days_from_target
)
return '| [#{}]({}) | {} | {} | {} | {} | {} | {} | {} |\n'.format(self.link.split('/')[-1], self.link,
self.author,
package, self.assignee, self.bot_advice,
create_date,
target_date,
days_from_target
)


def _extract(str_list, key_word):
Expand Down Expand Up @@ -148,7 +149,8 @@ def auto_reply(item, request_repo, rest_repo, sdk_repo, duplicated_issue, python
if 'auto-link' not in item.labels:
item.issue_object.add_to_labels('auto-link')
try:
package_name, readme_link, output_folder = update_issue_body(request_repo, rest_repo, item.issue_object.number)
package_name, readme_link, output_folder = update_issue_body(request_repo, rest_repo,
item.issue_object.number)
logging.info("pkname, readme", package_name, readme_link)
item.package = package_name
key = ('Python', item.package)
Expand All @@ -163,7 +165,7 @@ def auto_reply(item, request_repo, rest_repo, sdk_repo, duplicated_issue, python
readme_link, output_folder = get_readme_and_output_folder(request_repo, rest_repo, item.issue_object.number)
except Exception as e:
logging.info('Issue: {} get pkname and output folder failed'.format(item.issue_object.number))
item.bot_advice = 'failed to find Readme link and output folder. Please check !!'
item.bot_advice = 'failed to find Readme link and output folder! <br>'
item.issue_object.add_to_labels('attention')
logging.info(e)
raise
Expand All @@ -176,7 +178,7 @@ def auto_reply(item, request_repo, rest_repo, sdk_repo, duplicated_issue, python
if 'Configured' in item.labels:
item.issue_object.remove_from_labels('Configured')
except Exception as e:
item.bot_advice = 'auto reply failed, Please intervene manually !!'
item.bot_advice = 'auto reply failed! <br>'
logging.info('Error from auto reply')
logging.info('Issue:{}'.format(item.issue_object.number))
logging.info(traceback.format_exc())
Expand Down Expand Up @@ -208,7 +210,8 @@ def main():
issue.author = item.user.login
issue.package = _extract(item.body.split('\n'), 'azure-.*')
issue.target_date = [x.split(':')[-1].strip() for x in item.body.split('\n') if 'Target release date' in x][0]
issue.days_from_target = int((time.mktime(time.strptime(issue.target_date, '%Y-%m-%d')) - time.time())/3600/24)
issue.days_from_target = int(
(time.mktime(time.strptime(issue.target_date, '%Y-%m-%d')) - time.time()) / 3600 / 24)
issue.create_date = item.created_at.timestamp()
issue.delay_from_create_date = int((time.time() - item.created_at.timestamp()) / 3600 / 24)
issue.latest_update = item.updated_at.timestamp()
Expand All @@ -220,10 +223,9 @@ def main():
issue.whether_author_comment = _whether_author_comment(item.get_comments())
issue.issue_object = item
issue.labels = [label.name for label in item.labels]
issue.days_from_latest_commit = _latest_comment_time(item.get_comments(), issue.delay_from_create_date)
issue.days_from_latest_comment = _latest_comment_time(item.get_comments(), issue.delay_from_create_date)
if item.assignee:
issue.assignee = item.assignee.login

issue_status.append(issue)
key = (issue.language, issue.package)
duplicated_issue[key] = duplicated_issue.get(key, 0) + 1
Expand All @@ -239,51 +241,52 @@ def main():
# rule7: if delay from created date is over 15 days and owner never reply, remind owner to handle it.
for item in issue_status:
if item.language == 'Python':
issue_status_python.append(item)
assigner_repo = assigner_repoes[item.assignee]
item.issue_object = assigner_repo.get_issue(number=item.issue_object.number)
issue_status_python.append(item)
if item.status == 'release':
item.bot_advice = 'better to release asap.'
elif (item.comment_num == 0 or 'Configured' in item.labels) and 'Python' in item.labels:
item.bot_advice = 'new issue and better to confirm quickly.'
item.bot_advice = 'new issue ! <br>'
if 'assigned' not in item.labels:
time.sleep(0.1)
assign_count = int(str(time.time())[-1]) % len(_PYTHON_SDK_ASSIGNEES)
item.issue_object.remove_from_assignees(item.assignee)
item.issue_object.add_to_assignees(_PYTHON_SDK_ASSIGNEES[assign_count])
item.assignee=item.issue_object.assignee.login
item.assignee = item.issue_object.assignee.login
item.issue_object.add_to_labels('assigned')
try:
auto_reply(item, request_repo, rest_repo, sdk_repo, duplicated_issue, python_piplines, assigner_repoes)
except Exception as e:
continue
elif not item.author_latest_comment in _PYTHON_SDK_ADMINISTRATORS:
item.bot_advice = 'new comment for author.'
elif item.delay_from_latest_update >= 7:
item.bot_advice = 'delay for a long time '
item.bot_advice = 'new comment. <br>'
if item.comment_num > 1 and item.language == 'Python':
try:
auto_close_issue(request_repo, item)
except Exception as e:
item.bot_advice = 'auto-close failed, please check!'
logging.info(f"=====issue: {item.issue_object.number}, {e}")

if 'base-branch-attention' in item.labels:
item.bot_advice = 'new version is 0.0.0, please check base branch! ' + item.bot_advice

if abs(item.days_from_target) < 3:
item.bot_advice += ' release date < 2 ! <br>'
if item.days_from_latest_commit >= 30 and item.language == 'Python' and '30days attention' not in item.labels:
item.issue_object.add_to_labels('30days attention')
item.issue_object.create_comment(f'hi @{item.author}, the issue is closed since there is no reply for a long time. Please reopen it if necessary or create new one.')

if item.days_from_latest_comment >= 15 and item.language == 'Python' and '7days attention' in item.labels and item.days_from_target < 0:
item.issue_object.create_comment(
f'hi @{item.author}, the issue is closed since there is no reply for a long time. Please reopen it if necessary or create new one.')
item.issue_object.edit(state='close')
elif item.days_from_latest_commit >= 15 and item.language == 'Python' and '15days attention' not in item.labels:
item.issue_object.create_comment(f'hi @{item.author}, this release-request has been delayed more than 15 days,'
' please deal with it ASAP. We will close the issue if there is still no response after 15 days!')
item.issue_object.add_to_labels('15days attention')
elif item.days_from_latest_comment >= 7 and item.language == 'Python' and '7days attention' not in item.labels and item.days_from_target < 7:
item.issue_object.create_comment(
f'hi @{item.author}, this release-request has been delayed more than 7 days,'
' please deal with it ASAP. We will close the issue if there is still no response after 7 days!')
item.issue_object.add_to_labels('7days attention')

# judge whether there is duplicated issue for same package
if item.package != _NULL and duplicated_issue.get((item.language, item.package)) > 1:
item.bot_advice = f'duplicated issue for {item.package}. ' + item.bot_advice
item.bot_advice = f'duplicated issue <br>' + item.bot_advice

# output result
output_python_md(issue_status_python)
Expand All @@ -294,12 +297,11 @@ def main():
print_check('git commit -m \"update excel\"')
print_check('git push -f origin HEAD')

# upload to storage account(it is created in advance)
# upload to storage account(it is created in advance)
# blob = BlobClient.from_connection_string(conn_str=os.getenv('CONN_STR'), container_name=os.getenv('FILE'),
# blob_name=_FILE_OUT)
# with open(_FILE_OUT, 'rb') as data:
# blob.upload_blob(data, overwrite=True)


if __name__ == '__main__':
main()
9 changes: 1 addition & 8 deletions sdk/appconfiguration/azure-appconfiguration/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,8 @@
# Release History

## 1.3.0 (Unreleased)

### Features Added

### Breaking Changes
## 1.3.0 (2021-11-09)

### Bugs Fixed

- Fix the issue that data was persisted according to an incorrect schema/in an incorrect format ([#20518](https://github.com/Azure/azure-sdk-for-python/issues/20518))

`SecretReferenceConfigurationSetting` in 1.2.0 used "secret_uri" rather than "uri" as the schema keywords which
Expand All @@ -17,8 +12,6 @@
- Use 1.3.0+ for any `SecretReferenceConfigurationSetting` uses.
- Call a get method for existing `SecretReferenceConfigurationSetting`s and set them back to correct the format.

### Other Changes

## 1.2.0 (2021-07-06)
### Features Added
* Adds `FeatureFlagConfigurationSetting` and `SecretReferenceConfigurationSetting` models
Expand Down
Loading

0 comments on commit fbb680e

Please sign in to comment.