[Storage] Update documentation for batch APIs to include batch size limit

.github/CODEOWNERS

@@ -141,7 +141,7 @@
 /sdk/containerregistry/                                              @YalinLi0312
 
 # PRLabel: %Digital Twins
-/sdk/digitaltwins/                                                   @johngallardo
+/sdk/digitaltwins/                                                   @johngallardo @Aashish93-stack
 
 # PRLabel: %VideoAnalyzer
 /sdk/videoanalyzer/                                                  @hivyas

doc/dev/test_proxy_migration_guide.md

@@ -6,6 +6,8 @@ the Azure SDK test proxy.
 Documentation of the motivations and goals of the test proxy can be found [here][general_docs] in the azure-sdk-tools
 GitHub repository, and documentation of how to set up and use the proxy can be found [here][detailed_docs].
 
+Please refer to the [troubleshooting guide][troubleshooting] if you have any issues migrating a package.
+
 ## Table of contents
 - [Update existing tests](#update-existing-tests)
   - [Using resource preparers](#using-resource-preparers)
@@ -45,7 +47,7 @@ class TestExample(AzureTestCase):
 ### New test structure
 
 To use the proxy, test classes should inherit from AzureRecordedTestCase and recorded test methods should use a
-`recorded_by_proxy` decorator:
+`recorded_by_proxy` decorator directly on top of the test method:
 
 ```py
 from devtools_testutils import AzureRecordedTestCase, recorded_by_proxy
@@ -69,13 +71,16 @@ way as `recorded_by_proxy`.
 > with "Test" in order to be properly collected by pytest by default. For more information, please refer to
 > [pytest's documentation][pytest_collection].
 
+> **Note:** pure-`pytest` test cases aren't allowed to use an `__init__` constructor. Test classes should instead use
+> other methods of persisting state during a test run; for some `pytest` built-in options, please refer to
+> [pytest's documentation][pytest_setup].
+
 ### Using resource preparers
 
 Test suites that haven't fully migrated to using a `test-resources.json` file for test resource deployment might use
-resource preparers, such as
-[ResourceGroupPreparer](https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/resource_testcase.py).
-Migrating to [PowerShell test resource deployment][test_resources] is recommended (and test proxy migration might be a
-good opportunity to look into this), but the test proxy can work with resource preparers.
+resource preparers, such as [ResourceGroupPreparer][rg_preparer]. Migrating to
+[PowerShell test resource deployment][test_resources] is recommended (and test proxy migration might be a good
+opportunity to look into this), but the test proxy can work with resource preparers.
 
 Resource preparers need a management client to function, so test classes that use them will need to inherit from
 [AzureMgmtRecordedTestCase][mgmt_recorded_test_case] instead of AzureRecordedTestCase.
@@ -84,13 +89,10 @@ Resource preparers need a management client to function, so test classes that us
 
 ### Perform one-time setup
 
-Docker is a requirement for using the test proxy. You can install Docker from
-[docs.docker.com](https://docs.docker.com/get-docker/). After installing, make sure Docker is running and is using
-Linux containers before running tests.
-
-The test proxy is made available for your tests via a Docker container. Some tests require an SSL connection to work, so
-the Docker image used for the container has a certificate imported that you need to trust on your machine. Instructions
-on how to do so can be found [here][proxy_cert_docs] and need to be followed before running tests.
+1. Docker is a requirement for using the test proxy. You can install Docker from [docs.docker.com][docker_install].
+2. After installing, make sure Docker is running and is using Linux containers before running tests.
+3. Follow the instructions [here][proxy_cert_docs] to complete setup. You need to trust a certificate on your machine in
+order to communicate with the test proxy over a secure connection.
 
 ### Start the proxy server
 
@@ -133,7 +135,7 @@ Recordings for a given package will end up in that package's `/tests/recordings`
 do. Recordings that use the test proxy are `.json` files instead of `.yml` files, so migrated test suites no longer
 need old `.yml` recordings.
 
-> **Note:** at this time, support for configuring live or playback tests with a `testsettings_local.cfg` file has been
+> **Note:** support for configuring live or playback tests with a `testsettings_local.cfg` file has been
 > deprecated in favor of using just `AZURE_TEST_RUN_LIVE`.
 
 > **Note:** the recording storage location is determined when the proxy Docker container is created. If there are
@@ -154,28 +156,41 @@ are shared by different tests, using a session fixture declared in a `conftest.p
 [pytest's scoped fixture documentation][pytest_fixtures] for more details.
 
 As a simple example, to emulate the effect registering a name pair with a `vcrpy` scrubber, you can provide the exact
-value you want to sanitize from recordings as the `regex` in the general regex sanitizer. To replace all instances of
-the string "my-key-vault" with "fake-vault" in recordings, you could add something like the following in the package's
+value you want to sanitize from recordings as the `regex` in the general regex sanitizer. With `vcrpy`, you would likely
+do something like the following:
+
+```python
+import os
+from devtools_testutils import AzureTestCase
+
+class TestExample(AzureTestCase):
+    def __init__(self):
+        # scrub the value of AZURE_KEYVAULT_NAME with a fake vault name
+        self.scrubber.register_name_pair(os.getenv("AZURE_KEYVAULT_NAME"), "fake-vault")
+```
+
+To do the same sanitization with the test proxy, you could add something like the following in the package's
 `conftest.py` file:
 
 ```python
+import os
 from devtools_testutils import add_general_regex_sanitizer, test_proxy
 
 # autouse=True will trigger this fixture on each pytest run, even if it's not explicitly used by a test method
 @pytest.fixture(scope="session", autouse=True)
 def add_sanitizers(test_proxy):
-    add_general_regex_sanitizer(regex="my-key-vault", value="fake-vault")
+    add_general_regex_sanitizer(regex=os.getenv("AZURE_KEYVAULT_NAME"), value="fake-vault")
 ```
 
 Note that the sanitizer fixture accepts the `test_proxy` fixture as a parameter to ensure the proxy is started
 beforehand.
 
-For a more advanced scenario, where we want to sanitize the account names of all storage endpoints in recordings, we
+For a more advanced scenario, where we want to sanitize the account names of all Tables endpoints in recordings, we
 could instead call
 
 ```python
 add_general_regex_sanitizer(
-    regex="(?<=\\/\\/)[a-z]+(?=(?:|-secondary)\\.(?:table|blob|queue)\\.core\\.windows\\.net)",
+    regex="(?<=\\/\\/)[a-z]+(?=(?:|-secondary)\\.table\\.core\\.windows\\.net)",
     value="fakeendpoint",
 )
 ```
@@ -213,17 +228,17 @@ live pipeline testing, requests are made directly to the service instead of goin
 ### Fetch environment variables
 
 Fetching environment variables, passing them directly to tests, and sanitizing their real values can be done all at once
-by using the `devtools_testutils`
-[EnvironmentVariableLoader](https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/envvariable_loader.py).
+by using the `devtools_testutils` [EnvironmentVariableLoader][env_var_loader] (formerly known, and sometimes referred
+to, as the PowerShellPreparer).
 
-This loader is nice paired with the PowerShell test resource management commands that are
-documented in [/eng/common/TestResources][test_resources]. It's recommended that all test suites use these scripts for
-live test resource management.
+This loader is meant to be paired with the PowerShell test resource management commands that are documented in
+[/eng/common/TestResources][test_resources]. It's recommended that all test suites use these scripts for live test
+resource management.
 
-For an example of using the EnvironmentVariableLoader with the test proxy, you can refer to the Tables SDK. The CosmosPreparer
-and TablesPreparer defined in this [preparers.py][tables_preparers] file each define an instance of the
-EnvironmentVariableLoader, which are used to fetch environment variables for Cosmos and Tables, respectively. These preparers
-can be used to decorate test methods directly; for example:
+For an example of using the EnvironmentVariableLoader with the test proxy, you can refer to the Tables SDK. The
+CosmosPreparer and TablesPreparer defined in this [preparers.py][tables_preparers] file each define an instance of the
+EnvironmentVariableLoader, which are used to fetch environment variables for Cosmos and Tables, respectively. These
+preparers can be used to decorate test methods directly; for example:
 
 ```python
 from devtools_testutils import AzureRecordedTestCase, recorded_by_proxy
@@ -233,28 +248,37 @@ class TestExample(AzureRecordedTestCase):
 
     @TablesPreparer()
     @recorded_by_proxy
-    def test_example_with_preparer(self, tables_storage_account_name, tables_primary_storage_account_key):
+    def test_example_with_preparer(self, **kwargs):
+        tables_storage_account_name = kwargs.pop("tables_storage_account_name")
+        tables_primary_storage_account_key = kwargs.pop("tables_primary_storage_account_key")
         ...
 ```
 
 Or, they can be used in a custom decorator, as they are in the `cosmos_decorator` and `tables_decorator` defined in
 [preparers.py][tables_preparers]. `@tables_decorator`, for instance, is then used in place of `@TablesPreparer()` for
 the example above (note that the method-style `tables_decorator` is used without parentheses).
 
+Decorated test methods will have the values of environment variables passed to them as keyword arguments, and these
+values will automatically have sanitizers registered with the test proxy.
+
+> **Note:** For tests that are decorated by `@recorded_by_proxy` or `@recorded_by_proxy_async`, the keyword arguments
+> passed by EnvironmentVariableLoader can be listed as positional arguments instead of using `**kwargs`. However, tests
+> without these decorators can only accept arguments through `**kwargs`. It's therefore recommended that you use
+> `**kwargs` in all cases so that tests run successfully with or without `@recorded_by_proxy` decorators.
+
 ### Record test variables
 
 To run recorded tests successfully when there's an element of non-secret randomness to them, the test proxy provides a
-[`variables` API](https://github.com/Azure/azure-sdk-tools/tree/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy#storing-variables).
-This makes it possible for a test to record the values of variables that were used during recording and use the same
-values in playback mode without a sanitizer.
+[`variables` API][variables_api]. This makes it possible for a test to record the values of variables that were used
+during recording and use the same values in playback mode without a sanitizer.
 
 For example, imagine that a test uses a randomized `table_name` variable when creating resources. The same random value
 for `table_name` can be used in playback mode by using this `variables` API.
 
-There are two requirements for a test to use recorded variables. First, the test method should accept `**kwargs` and/or
-a `variables` parameter. Second, the test method should `return` a dictionary with any test variables that it wants to
-record. This dictionary will be stored in the recording when the test is run live, and will be passed to the test as a
-`variables` keyword argument when the test is run in playback.
+There are two requirements for a test to use recorded variables. First, the test method should accept `**kwargs`.
+Second, the test method should `return` a dictionary with any test variables that it wants to record. This dictionary
+will be stored in the recording when the test is run live, and will be passed to the test as a `variables` keyword
+argument when the test is run in playback.
 
 Below is a code example of how a test method could use recorded variables:
 
@@ -264,9 +288,10 @@ from devtools_testutils import AzureRecordedTestCase, recorded_by_proxy
 class TestExample(AzureRecordedTestCase):
 
     @recorded_by_proxy
-    def test_example(self, variables):
+    def test_example(self, **kwargs):
         # in live mode, variables is an empty dictionary
         # in playback mode, the value of variables is {"table_name": "random-value"}
+        variables = kwargs.pop("variables")
         if self.is_live:
             table_name = "random-value"
             variables = {"table_name": table_name}
@@ -356,24 +381,30 @@ of `"start"`.
 
 #### Python
 
-There are two methods in `devtools_testutils`,
-[start_test_proxy](https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/proxy_docker_startup.py#L97)
-and
-[stop_test_proxy](https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/proxy_docker_startup.py#L135),
-that can be used to manually start and stop the test proxy. Like `docker-start-proxy.ps1`, `start_test_proxy` will
-automatically fetch the proxy Docker image for you and start the container if it's not already running.
+There are two methods in `devtools_testutils`, [start_test_proxy][start_test_proxy] and
+[stop_test_proxy][stop_test_proxy], that can be used to manually start and stop the test proxy. Like
+`docker-start-proxy.ps1`, `start_test_proxy` will automatically fetch the proxy Docker image for you and start the
+container if it's not already running.
 
 For more details on proxy startup, please refer to the [proxy documentation][detailed_docs].
 
 [detailed_docs]: https://github.com/Azure/azure-sdk-tools/tree/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md
+[docker_install]: https://docs.docker.com/get-docker/
 [docker_start_proxy]: https://github.com/Azure/azure-sdk-for-python/blob/main/eng/common/testproxy/docker-start-proxy.ps1
+[env_var_loader]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/envvariable_loader.py
 [general_docs]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/README.md
 [mgmt_recorded_test_case]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/mgmt_recorded_testcase.py
 [proxy_cert_docs]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/documentation/trusting-cert-per-language.md
 [py_sanitizers]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/sanitizers.py
 [pytest_collection]: https://docs.pytest.org/latest/goodpractices.html#test-discovery
 [pytest_fixtures]: https://docs.pytest.org/latest/fixture.html#scope-sharing-fixtures-across-classes-modules-packages-or-session
+[pytest_setup]: https://docs.pytest.org/xunit_setup.html
+[rg_preparer]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/resource_testcase.py
 [sanitizers]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md#session-and-test-level-transforms-sanitiziers-and-matchers
+[start_test_proxy]: https://github.com/Azure/azure-sdk-for-python/blob/63a35890a0188dfcac094aa7dc1ec7cc730945cd/tools/azure-sdk-tools/devtools_testutils/proxy_docker_startup.py#L111
+[stop_test_proxy]: https://github.com/Azure/azure-sdk-for-python/blob/63a35890a0188dfcac094aa7dc1ec7cc730945cd/tools/azure-sdk-tools/devtools_testutils/proxy_docker_startup.py#L149
 [tables_preparers]: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/tables/azure-data-tables/tests/preparers.py
 [test_resources]: https://github.com/Azure/azure-sdk-for-python/tree/main/eng/common/TestResources#readme
+[troubleshooting]: https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/test_proxy_troubleshooting.md
+[variables_api]: https://github.com/Azure/azure-sdk-tools/tree/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy#storing-variables
 [vcrpy]: https://vcrpy.readthedocs.io

eng/conda_test_requirements.txt

@@ -1,4 +1,5 @@
 # install from root of repo
+azure-mgmt-resource==20.0.0
 aiohttp>=3.0; python_version >= '3.5'
 tools/azure-devtools
 tools/azure-sdk-tools

eng/test_tools.txt

@@ -15,7 +15,6 @@ pytest-xdist==1.32.0
 coverage==4.5.4
 bandit==1.6.2
 protobuf==3.17.3; python_version == '2.7'
-wrapt<=1.12.1; python_version == '2.7'
 chardet>=2.0,<5.0
 
 # locking packages defined as deps from azure-sdk-tools or azure-devtools

sdk/iothub/azure-mgmt-iothub/CHANGELOG.md

@@ -1,5 +1,12 @@
 # Release History
 
+## 2.2.0 (2022-01-29)
+
+**Features**
+
+  - Model IotHubDescription has a new parameter system_data
+  - Model IotHubProperties has a new parameter enable_data_residency
+
 ## 2.1.0 (2021-08-25)
 
 **Features**

sdk/iothub/azure-mgmt-iothub/_meta.json

@@ -1,11 +1,11 @@
 {
-  "autorest": "3.4.5",
+  "autorest": "3.7.2",
   "use": [
-    "@autorest/python@5.8.4",
-    "@autorest/[email protected].2"
+    "@autorest/python@5.12.0",
+    "@autorest/[email protected].3"
   ],
-  "commit": "1f327d204d0c1835d7ee675d93b423028341a6b7",
+  "commit": "f9a6cb686bcc0f1b23761db19f2491c5c4df95cb",
   "repository_url": "https://github.com/Azure/azure-rest-api-specs",
-  "autorest_command": "autorest specification/iothub/resource-manager/readme.md --multiapi --python --python-mode=update --python-sdks-folder=/home/vsts/work/1/s/azure-sdk-for-python/sdk --track2 --use=@autorest/python@5.8.4 --use=@autorest/[email protected].2 --version=3.4.5",
+  "autorest_command": "autorest specification/iothub/resource-manager/readme.md --multiapi --python --python-mode=update --python-sdks-folder=/home/vsts/work/1/s/azure-sdk-for-python/sdk --python3-only --track2 --use=@autorest/python@5.12.0 --use=@autorest/[email protected].3 --version=3.7.2",
   "readme": "specification/iothub/resource-manager/readme.md"
 }

sdk/iothub/azure-mgmt-iothub/azure/mgmt/iothub/_configuration.py

@@ -12,7 +12,7 @@
 
 from azure.core.configuration import Configuration
 from azure.core.pipeline import policies
-from azure.mgmt.core.policies import ARMHttpLoggingPolicy
+from azure.mgmt.core.policies import ARMChallengeAuthenticationPolicy, ARMHttpLoggingPolicy
 
 from ._version import VERSION
 
@@ -68,4 +68,4 @@ def _configure(
         self.redirect_policy = kwargs.get('redirect_policy') or policies.RedirectPolicy(**kwargs)
         self.authentication_policy = kwargs.get('authentication_policy')
         if self.credential and not self.authentication_policy:
-            self.authentication_policy = policies.BearerTokenCredentialPolicy(self.credential, *self.credential_scopes, **kwargs)
+            self.authentication_policy = ARMChallengeAuthenticationPolicy(self.credential, *self.credential_scopes, **kwargs)