[Service Bus]"Preparer failure when creating resource ServiceBusNamespacePreparer, Resource group could not be found" on UsGov and China cloud

[v-xuto]

We are running live Tests against other clouds like US Gov and Azure China Cloud. The goal is to check whether new azure sdk package work with other clouds or not.

Error Description:
When running the test tests\test_queues.py::ServiceBusQueueTests::test_github_issue_6178 on UsGov and China cloud, it runs failed and the error message is shown as following, for more details please check here:

Error Track:
The code that failed is here：

code link2:

Note: We guessed that the environment variable did not set the value of RESOURCE_REGION, so the default value ‘westus’ was used, so the resource group was not created successfully in UsGov and China Cloud.

Expected Behavior:
The resource group was successfully created and it can be found.

@benbp , @jameszliao-msft , @lmazuel , @lilyjma , @ramya-rao-a and @annatisch for notification.

[benbp]

If you change RESOURCE_REGION to SERVICEBUS_LOCATION it should be able to look up the value correctly based on the context set by the resource deployment scripts.

[v-xuto]

@benbp

1. First, because get_region_override is a method in tools devtools_testutils, so we update location=get_region_override('westus') to location=os.environ.get("SERVICEBUS_LOCATION", None) or 'westus' in class ServiceBusNamespacePreparer(AzureMgmtPreparer). But it still reports the same error, its location is still westus. We suspect that this parameter SERVICEBUS_LOCATION has no value or is 'westus' ?
2. Then, we update location=get_region_override('westus') to location='usgovarizona', it reports the following error, for more details in here:

(test_queues.ServiceBusQueueTests): (LocationNotAvailableForResourceType) The provided location '***' is not available for resource type 'Microsoft.ServiceBus/namespaces'. List of available regions for the resource type is 'australiaeast,australiasoutheast,centralus,eastus,eastus2,westus2,westus,northcentralus,southcentralus,westcentralus,eastasia,southeastasia,brazilsouth,japaneast,japanwest,northeurope,westeurope,centralindia,southindia,westindia,canadacentral,canadaeast,ukwest,uksouth,koreacentral,koreasouth,francecentral,southafricanorth,uaenorth,australiacentral,switzerlandnorth,germanywestcentral,norwayeast,jioindiawest,westus3,centraluseuap,eastus2euap'.

By reporting the error message, we suspect that it is still under the public cloud. This is confusing, where is it going to the corresponding cloud environment.

Do you have any ideas?

[benbp]

@v-xuto sorry this is my mistake! Looks like it's called SERVICE_BUS_LOCATION. You can see the key names of the env outputs that get set in the Deploy test resources logs:

https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1080695&view=logs&j=eb4e8fab-410d-5462-8da4-6074733a2525&t=5877313e-8ad1-5e16-7cf7-7ae7932c113c&l=3267

[v-xuto]

@benbp We updated SERVICEBUS_LOCATION to SERVICE_BUS_LOCATION, there is still the same error, its location is still westus, for more details please check here. Besides, we update location=get_region_override('westus') to location='usgovarizona', by reporting the error message, we suspect that it is still under the public cloud.

[benbp]

@v-xuto I didn't see that it was in the dev tools location. You might try instead setting RESOURCE_REGION as an output value in the test-resources.json corresponding to [resourceGroup().location].

Looking at your branch, you also need to update the azure environment values to reference the dynamic cloud subscription config variables instead of the hardcoded public cloud variable group. That's why it's looking in public:

azure-sdk-for-python/sdk/servicebus/tests.yml

Lines 14 to 17 in a0ae039

	AZURE_SUBSCRIPTION_ID: $(azure-subscription-id)
	AZURE_TENANT_ID: $(aad-azure-sdk-test-tenant-id)
	AZURE_CLIENT_ID: $(aad-azure-sdk-test-client-id)
	AZURE_CLIENT_SECRET: $(aad-azure-sdk-test-client-secret)

https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1093075&view=logs&j=eb4e8fab-410d-5462-8da4-6074733a2525&t=5877313e-8ad1-5e16-7cf7-7ae7932c113c&l=3300

        AZURE_SUBSCRIPTION_ID: $(SERVICEBUS_SUBSCRIPTION_ID)
        AZURE_TENANT_ID: $(SERVICEBUS_TENANT_ID)
        AZURE_CLIENT_ID: $(SERVICEBUS_CLIENT_ID)
        AZURE_CLIENT_SECRET: $(SERVICEBUS_CLIENT_SECRET)

[v-xuto]

@benbp - I have updated the code by your comments. But we encountered the following error, for more details please check here:
Usgov:
azure_devtools.scenario_tests.preparers:preparers.py:82 Preparer failure when creating resource ServiceBusNamespacePreparer for test test_github_issue_6178 (test_queues.ServiceBusQueueTests): , AdalError: Get Token request returned http error: 400 and server response: {"error":"invalid_request","error_description":"AADSTS900382: Confidential Client is not supported in Cross Cloud request.}

China:
azure_devtools.scenario_tests.preparers:preparers.py:82 Preparer failure when creating resource ***erviceBusNamespacePreparer for test test_github_issue_6178 (test_queues.***erviceBusQueueTests): , AdalError: Get Token request returned http error: 400 and server response: {"error":"invalid_request","error_description":"AAD***T***90002: Tenant '***' not found. Check to make sure you have the correct tenant ID and are signing into the correct cloud. Check with your subscription administrator, this may happen if there are no active subscriptions for the tenant.}

Do you have any ideas?

[benbp]

@v-xuto it seems like the tenant id is still not getting set for the sovereign cloud properly, which is generally the issue for errors like not supported in Cross Cloud request. @scbedd I tried to dig a little through the devtools to see whether a custom tenant id isn't getting passed through properly, but I couldn't find anything that indicated a problem. Perhaps you have an idea?

[scbedd]

@benbp If all the surface level stuff appears legit, it's going to be something esoteric. Is this the only python package having issues cross-cloud?

Following up offline to get a bit more context.

[v-xuto]

@scbedd Yeah, currently I have tested the formrecognizer, eventhubs, textanalytics and appconfig, none of them have this problem.
@benbp Is there anything to add?

[scbedd]

Ok if all the others work, it's either A) configuration issue with the servicebus tests specifically (I'm leaning towards this, servicebus hasn't gotten a TON of attention) or B) a service specific issue that we're identifying.

[v-xuto]

@benbp What do you think about these comments? Do you think it is a servicebus configuration problem? How do we solve the issue?

[benbp]

@yunhaoling can you look at this in addition to #20875? Or is there a better sdk service owner?

[yunhaoling]

@benbp , I will first try to verify it locally before deploying it to CI, if it can't work locally, then might be an issue in the SDK

[v-xuto]

@yunhaoling Any progress?

[v-xuto]

@benbp, @yunhaoling By my investigation locally, the errors on usgov and china should be related to the scope https://management.azure.com in mgmt sdk azure-mgmt-resource and azure-mgmt-servicebus. Because I modified the scope (https://management.azure.com) value on usgov and china, the local result of the servicebus test was pass. When the cloud is usgov, its value should be https://management.usgovcloudapi.net; when the cloud is china, its value should be https://management.chinacloudapi.net. In servicebus tests code, it needs to create a resource group and service bus by itself, but the scope in its source code is hard-coded, it is https://management.usgovcloudapi.net. It is similar to the issue of eventhub. I currently have two ideas to solve this problem.

1. File issues about azure-mgmt-resource and azure-mgmt-servicebus. Do you know in which repo I should file mgmt issues? I am not very clear.
   eg:
   azure/mgmt/servicebus/_service_bus_management_client.py
   
   azure/mgmt/servicebus/_configuration.py
   

2. Can we consider changing the test framework and directly use the service created in this part of Deploy test resources for testing. Don't use mgmt sdk to create resources in the test, like key vault keys and app configuration tests.

Regarding these two methods, which one do you prefer? Please give me some advice.

[benbp]

@v-xuto I think we should change places in the test framework that are causing problems, and we should not be deploying via the test code, only the ARM template. I think this is mostly done across the python repository but there are still a few places that use the legacy deployment method. @yunhaoling do you know if/where this work is tracked, or am I wrong?

[v-xuto]

@yunhaoling Do you have any ideas?

[yunhaoling]

hey @benbp @v-xuto , apologize for missing the threads here. I agree that we should update the code within the test framework (to be more specific, it's ServiceBusPreparer) to get tests work on UsGov and China Cloud. However, I think it might take extra time if we want to embrace the ARM template approach completely using PowerShellPreparer (or even not feasible).

Here's my findings:

1. @benbp, as far as I know, we use ARM template + PowerShellPreparer to deploy resources for sample tests (Deploy test resources), but for live tests, each SDK does it differently, some using MgmtClient while some using ARM template. I think it depends on the SDKs and test scenarios.

• for service bus live tests, we need different queues to test various queue/topic/subscription settings.

2. @v-xuto thanks for you investigation, as you pointed out, I think it's because when we're creating the ServiceBusManagementClient in the preparer, we didn't consider other clouds having different base url leading us not to have the option to set the value.

I think there is one easy fix:

1. In servicebus_preparer.py

within each create_resource method, we could read the base_url/credential_scopes information from the environment and pass it to the mgmt client like this:

    def create_resource(self, name, **kwargs):
        if self.is_live:
            base_url = os.envrion.get("BASE_URL","https://management.azure.com")
            credential_scopes = [os.envrion.get("CREDENTIAL_SCOPES","https://management.azure.com/.default")]
            self.client = self.create_mgmt_client(ServiceBusManagementClient, base_url=base_url, credential_scopes=credential_scopes)
            group = self._get_resource_group(**kwargs)
       ...

2. in test.yml, we set the environment variables accordingly:

BASE_URL: https://management.usgovcloudapi.net
REDENTIAL_SCOPE: https://management.usgovcloudapi.net/.default
or
BASE_URL: https://management.chinacloudapi.net
CREDENTIAL_SCOPE: https://management.chinacloudapi.net/.default

However, I am unsure whether those envs could just be put under the CloudConfig section like this or has to be in other format, @benbp would you be able to help tell what is the right format?

        UsGov:
          SubscriptionConfiguration: $(sub-config-gov-test-resources)
          BASE_URL: https://management.usgovcloudapi.net
          REDENTIAL_SCOPE: https://management.usgovcloudapi.net/.default
        China:
          SubscriptionConfiguration: $(sub-config-cn-test-resources)
          Location: 'chinanorth2'
          BASE_URL: https://management.chinacloudapi.net
          CREDENTIAL_SCOPE: https://management.chinacloudapi.net/.default

@v-xuto , I think you could try step 1 with local envs to see if it works, if this approach works, then same change could be applied to your EH PR.

[v-xuto]

@yunhaoling Thanks for your help. I have tried step 1 with local envs and also add same code in Create Resource Group Management Client in file devtools_testutils/resource_testcase.py. And then, the service bus test result is pass locally.

Besides, can I directly modify the code in tools devtools_testutils?

@benbp Please help to set environment variables CREDENTIAL_SCOPES and BASE_URL in China and UsGov clouds, like the parameter Azure_Authority_Host.

[yunhaoling]

@v-xuto I think instead of updating the resource_testcase.py, it's safer to just update the servicebus_preparer.py as we might break other SDKs due to how we construct the envs here.

Note that there're six create_resource in the service_preparer.py which need to be updated.

[v-xuto]

@rakshith91 Do you have any ideas instead of modifying the resource_testcase.py file?

[rakshith91]

@v-xuto im currently working on getting rid of it completely and using the arm template. That will fix this issue. It should be done in a couple of weeks. Sorry for the late update.

[v-xuto]

@rakshith91 How is the current progress of this issue?

[rakshith91]

@v-xuto I have a PR out - i'm hoping to get it merged by next week
#23566

[v-xuto]

@rakshith91 How is your PR going so far? When will this PR be merged?
I'm looking forward to your PR being merged soon.

[rakshith91]

@v-xuto I have a few more tests to fix before merging the PR. Meanwhile, feel free to test it on my branch. You'd be seeing some test failures, but no set up failures.

cc: @kashifkhan

[v-xuto]

@rakshith91 Thank you for your prompt reply. I'm looking forward to your PR being merged soon.

[v-xuto]

@rakshith91 How is your PR going so far to fix this issue?

[v-xuto]

@swathipil What is the current progress on the fix of this issue?

[swathipil]

Hi @v-xuto - I have a PR to fix this and all the tests are passing: #29202

However, CI is failing b/c of a dependency conflict. I have a PR out to fix that dependency conflict. Hoping we can merge all the PRs and close this issue by today. Thanks for your patience!