Skip to content

Commit

Permalink
[Test Proxy] Update migration guide (Azure#22729)
Browse files Browse the repository at this point in the history
  • Loading branch information
@mccoyp @rakshith91
mccoyp authored and rakshith91 committed Apr 10, 2022
1 parent 23b947c commit 894a086
Showing 1 changed file with 73 additions and 42 deletions.
115 changes: 73 additions & 42 deletions doc/dev/test_proxy_migration_guide.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ the Azure SDK test proxy.
Documentation of the motivations and goals of the test proxy can be found [here][general_docs] in the azure-sdk-tools
GitHub repository, and documentation of how to set up and use the proxy can be found [here][detailed_docs].

Please refer to the [troubleshooting guide][troubleshooting] if you have any issues migrating a package.

## Table of contents
- [Update existing tests](#update-existing-tests)
- [Using resource preparers](#using-resource-preparers)
Expand Down Expand Up @@ -45,7 +47,7 @@ class TestExample(AzureTestCase):
### New test structure

To use the proxy, test classes should inherit from AzureRecordedTestCase and recorded test methods should use a
`recorded_by_proxy` decorator:
`recorded_by_proxy` decorator directly on top of the test method:

```py
from devtools_testutils import AzureRecordedTestCase, recorded_by_proxy
Expand All @@ -69,13 +71,16 @@ way as `recorded_by_proxy`.
> with "Test" in order to be properly collected by pytest by default. For more information, please refer to
> [pytest's documentation][pytest_collection].
> **Note:** pure-`pytest` test cases aren't allowed to use an `__init__` constructor. Test classes should instead use
> other methods of persisting state during a test run; for some `pytest` built-in options, please refer to
> [pytest's documentation][pytest_setup].
### Using resource preparers

Test suites that haven't fully migrated to using a `test-resources.json` file for test resource deployment might use
resource preparers, such as
[ResourceGroupPreparer](https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/resource_testcase.py).
Migrating to [PowerShell test resource deployment][test_resources] is recommended (and test proxy migration might be a
good opportunity to look into this), but the test proxy can work with resource preparers.
resource preparers, such as [ResourceGroupPreparer][rg_preparer]. Migrating to
[PowerShell test resource deployment][test_resources] is recommended (and test proxy migration might be a good
opportunity to look into this), but the test proxy can work with resource preparers.

Resource preparers need a management client to function, so test classes that use them will need to inherit from
[AzureMgmtRecordedTestCase][mgmt_recorded_test_case] instead of AzureRecordedTestCase.
Expand All @@ -84,13 +89,10 @@ Resource preparers need a management client to function, so test classes that us

### Perform one-time setup

Docker is a requirement for using the test proxy. You can install Docker from
[docs.docker.com](https://docs.docker.com/get-docker/). After installing, make sure Docker is running and is using
Linux containers before running tests.

The test proxy is made available for your tests via a Docker container. Some tests require an SSL connection to work, so
the Docker image used for the container has a certificate imported that you need to trust on your machine. Instructions
on how to do so can be found [here][proxy_cert_docs] and need to be followed before running tests.
1. Docker is a requirement for using the test proxy. You can install Docker from [docs.docker.com][docker_install].
2. After installing, make sure Docker is running and is using Linux containers before running tests.
3. Follow the instructions [here][proxy_cert_docs] to complete setup. You need to trust a certificate on your machine in
order to communicate with the test proxy over a secure connection.

### Start the proxy server

Expand Down Expand Up @@ -133,7 +135,7 @@ Recordings for a given package will end up in that package's `/tests/recordings`
do. Recordings that use the test proxy are `.json` files instead of `.yml` files, so migrated test suites no longer
need old `.yml` recordings.

> **Note:** at this time, support for configuring live or playback tests with a `testsettings_local.cfg` file has been
> **Note:** support for configuring live or playback tests with a `testsettings_local.cfg` file has been
> deprecated in favor of using just `AZURE_TEST_RUN_LIVE`.
> **Note:** the recording storage location is determined when the proxy Docker container is created. If there are
Expand All @@ -154,28 +156,41 @@ are shared by different tests, using a session fixture declared in a `conftest.p
[pytest's scoped fixture documentation][pytest_fixtures] for more details.

As a simple example, to emulate the effect registering a name pair with a `vcrpy` scrubber, you can provide the exact
value you want to sanitize from recordings as the `regex` in the general regex sanitizer. To replace all instances of
the string "my-key-vault" with "fake-vault" in recordings, you could add something like the following in the package's
value you want to sanitize from recordings as the `regex` in the general regex sanitizer. With `vcrpy`, you would likely
do something like the following:

```python
import os
from devtools_testutils import AzureTestCase

class TestExample(AzureTestCase):
def __init__(self):
# scrub the value of AZURE_KEYVAULT_NAME with a fake vault name
self.scrubber.register_name_pair(os.getenv("AZURE_KEYVAULT_NAME"), "fake-vault")
```

To do the same sanitization with the test proxy, you could add something like the following in the package's
`conftest.py` file:

```python
import os
from devtools_testutils import add_general_regex_sanitizer, test_proxy

# autouse=True will trigger this fixture on each pytest run, even if it's not explicitly used by a test method
@pytest.fixture(scope="session", autouse=True)
def add_sanitizers(test_proxy):
add_general_regex_sanitizer(regex="my-key-vault", value="fake-vault")
add_general_regex_sanitizer(regex=os.getenv("AZURE_KEYVAULT_NAME"), value="fake-vault")
```

Note that the sanitizer fixture accepts the `test_proxy` fixture as a parameter to ensure the proxy is started
beforehand.

For a more advanced scenario, where we want to sanitize the account names of all storage endpoints in recordings, we
For a more advanced scenario, where we want to sanitize the account names of all Tables endpoints in recordings, we
could instead call

```python
add_general_regex_sanitizer(
regex="(?<=\\/\\/)[a-z]+(?=(?:|-secondary)\\.(?:table|blob|queue)\\.core\\.windows\\.net)",
regex="(?<=\\/\\/)[a-z]+(?=(?:|-secondary)\\.table\\.core\\.windows\\.net)",
value="fakeendpoint",
)
```
Expand Down Expand Up @@ -213,17 +228,17 @@ live pipeline testing, requests are made directly to the service instead of goin
### Fetch environment variables

Fetching environment variables, passing them directly to tests, and sanitizing their real values can be done all at once
by using the `devtools_testutils`
[EnvironmentVariableLoader](https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/envvariable_loader.py).
by using the `devtools_testutils` [EnvironmentVariableLoader][env_var_loader] (formerly known, and sometimes referred
to, as the PowerShellPreparer).

This loader is nice paired with the PowerShell test resource management commands that are
documented in [/eng/common/TestResources][test_resources]. It's recommended that all test suites use these scripts for
live test resource management.
This loader is meant to be paired with the PowerShell test resource management commands that are documented in
[/eng/common/TestResources][test_resources]. It's recommended that all test suites use these scripts for live test
resource management.

For an example of using the EnvironmentVariableLoader with the test proxy, you can refer to the Tables SDK. The CosmosPreparer
and TablesPreparer defined in this [preparers.py][tables_preparers] file each define an instance of the
EnvironmentVariableLoader, which are used to fetch environment variables for Cosmos and Tables, respectively. These preparers
can be used to decorate test methods directly; for example:
For an example of using the EnvironmentVariableLoader with the test proxy, you can refer to the Tables SDK. The
CosmosPreparer and TablesPreparer defined in this [preparers.py][tables_preparers] file each define an instance of the
EnvironmentVariableLoader, which are used to fetch environment variables for Cosmos and Tables, respectively. These
preparers can be used to decorate test methods directly; for example:

```python
from devtools_testutils import AzureRecordedTestCase, recorded_by_proxy
Expand All @@ -233,28 +248,37 @@ class TestExample(AzureRecordedTestCase):

@TablesPreparer()
@recorded_by_proxy
def test_example_with_preparer(self, tables_storage_account_name, tables_primary_storage_account_key):
def test_example_with_preparer(self, **kwargs):
tables_storage_account_name = kwargs.pop("tables_storage_account_name")
tables_primary_storage_account_key = kwargs.pop("tables_primary_storage_account_key")
...
```

Or, they can be used in a custom decorator, as they are in the `cosmos_decorator` and `tables_decorator` defined in
[preparers.py][tables_preparers]. `@tables_decorator`, for instance, is then used in place of `@TablesPreparer()` for
the example above (note that the method-style `tables_decorator` is used without parentheses).

Decorated test methods will have the values of environment variables passed to them as keyword arguments, and these
values will automatically have sanitizers registered with the test proxy.

> **Note:** For tests that are decorated by `@recorded_by_proxy` or `@recorded_by_proxy_async`, the keyword arguments
> passed by EnvironmentVariableLoader can be listed as positional arguments instead of using `**kwargs`. However, tests
> without these decorators can only accept arguments through `**kwargs`. It's therefore recommended that you use
> `**kwargs` in all cases so that tests run successfully with or without `@recorded_by_proxy` decorators.
### Record test variables

To run recorded tests successfully when there's an element of non-secret randomness to them, the test proxy provides a
[`variables` API](https://github.com/Azure/azure-sdk-tools/tree/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy#storing-variables).
This makes it possible for a test to record the values of variables that were used during recording and use the same
values in playback mode without a sanitizer.
[`variables` API][variables_api]. This makes it possible for a test to record the values of variables that were used
during recording and use the same values in playback mode without a sanitizer.

For example, imagine that a test uses a randomized `table_name` variable when creating resources. The same random value
for `table_name` can be used in playback mode by using this `variables` API.

There are two requirements for a test to use recorded variables. First, the test method should accept `**kwargs` and/or
a `variables` parameter. Second, the test method should `return` a dictionary with any test variables that it wants to
record. This dictionary will be stored in the recording when the test is run live, and will be passed to the test as a
`variables` keyword argument when the test is run in playback.
There are two requirements for a test to use recorded variables. First, the test method should accept `**kwargs`.
Second, the test method should `return` a dictionary with any test variables that it wants to record. This dictionary
will be stored in the recording when the test is run live, and will be passed to the test as a `variables` keyword
argument when the test is run in playback.

Below is a code example of how a test method could use recorded variables:

Expand All @@ -264,9 +288,10 @@ from devtools_testutils import AzureRecordedTestCase, recorded_by_proxy
class TestExample(AzureRecordedTestCase):

@recorded_by_proxy
def test_example(self, variables):
def test_example(self, **kwargs):
# in live mode, variables is an empty dictionary
# in playback mode, the value of variables is {"table_name": "random-value"}
variables = kwargs.pop("variables")
if self.is_live:
table_name = "random-value"
variables = {"table_name": table_name}
Expand Down Expand Up @@ -356,24 +381,30 @@ of `"start"`.

#### Python

There are two methods in `devtools_testutils`,
[start_test_proxy](https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/proxy_docker_startup.py#L97)
and
[stop_test_proxy](https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/proxy_docker_startup.py#L135),
that can be used to manually start and stop the test proxy. Like `docker-start-proxy.ps1`, `start_test_proxy` will
automatically fetch the proxy Docker image for you and start the container if it's not already running.
There are two methods in `devtools_testutils`, [start_test_proxy][start_test_proxy] and
[stop_test_proxy][stop_test_proxy], that can be used to manually start and stop the test proxy. Like
`docker-start-proxy.ps1`, `start_test_proxy` will automatically fetch the proxy Docker image for you and start the
container if it's not already running.

For more details on proxy startup, please refer to the [proxy documentation][detailed_docs].

[detailed_docs]: https://github.com/Azure/azure-sdk-tools/tree/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md
[docker_install]: https://docs.docker.com/get-docker/
[docker_start_proxy]: https://github.com/Azure/azure-sdk-for-python/blob/main/eng/common/testproxy/docker-start-proxy.ps1
[env_var_loader]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/envvariable_loader.py
[general_docs]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/README.md
[mgmt_recorded_test_case]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/mgmt_recorded_testcase.py
[proxy_cert_docs]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/documentation/trusting-cert-per-language.md
[py_sanitizers]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/sanitizers.py
[pytest_collection]: https://docs.pytest.org/latest/goodpractices.html#test-discovery
[pytest_fixtures]: https://docs.pytest.org/latest/fixture.html#scope-sharing-fixtures-across-classes-modules-packages-or-session
[pytest_setup]: https://docs.pytest.org/xunit_setup.html
[rg_preparer]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/resource_testcase.py
[sanitizers]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md#session-and-test-level-transforms-sanitiziers-and-matchers
[start_test_proxy]: https://github.com/Azure/azure-sdk-for-python/blob/63a35890a0188dfcac094aa7dc1ec7cc730945cd/tools/azure-sdk-tools/devtools_testutils/proxy_docker_startup.py#L111
[stop_test_proxy]: https://github.com/Azure/azure-sdk-for-python/blob/63a35890a0188dfcac094aa7dc1ec7cc730945cd/tools/azure-sdk-tools/devtools_testutils/proxy_docker_startup.py#L149
[tables_preparers]: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/tables/azure-data-tables/tests/preparers.py
[test_resources]: https://github.com/Azure/azure-sdk-for-python/tree/main/eng/common/TestResources#readme
[troubleshooting]: https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/test_proxy_troubleshooting.md
[variables_api]: https://github.com/Azure/azure-sdk-tools/tree/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy#storing-variables
[vcrpy]: https://vcrpy.readthedocs.io

0 comments on commit 894a086

Please sign in to comment.