From e9bd025c59c323abf2e8f248b0a1e9b7609544a8 Mon Sep 17 00:00:00 2001 From: Sean Kane <68240067+seankane-msft@users.noreply.github.com> Date: Tue, 1 Sep 2020 14:22:56 -0700 Subject: [PATCH] Remove resources post test (#13379) * pruning tests.yml * potential fix * updated environment variables * another change to tests.yml * change * edits to testcase * setting storage_name and rg_kwargs to None to help identify problem * forcing i_need_to_create_rg to be True * flirting with version change * reverting back to normal for both testcase and version * Packaging update of azure-data-tables * adding azure identity to dev reqs * adding azure-identity to dev_reqs * mirror updates from personal branch * adding continue on error so we can see the env * update shared test case rg_kwargs resource to mapping instead of NoneType * updating dev_reqs * random issues gone? * build-test changes * undoing all the bot changes from AutorestCI * edits thanks to Kristas suggestion, hopefully stops AutorestCI from sabotage * import directly from file, python2 can't find the module * pylint fixes * debugging why python 2.7 can't find azure data tables * undoing last change * reverting setup.py to master branch, adding space in inits just incase * add comprehension of nspkg to azure-data-tables * adding nspkg to shared_reqs * updating shared_req to freeze nspkg at the same version Co-authored-by: Azure SDK Bot Co-authored-by: Scott Beddall Co-authored-by: scbedd <45376673+scbedd@users.noreply.github.com> --- sdk/tables/azure-data-tables/MANIFEST.in | 3 +- sdk/tables/azure-data-tables/README.md | 41 +++++++------------ .../azure-data-tables/dev_requirements.txt | 3 +- .../azure-data-tables/sdk_packaging.toml | 2 + sdk/tables/azure-data-tables/setup.py | 5 ++- .../tests/_shared/testcase.py | 19 +++++---- sdk/tables/tests.yml | 27 +++--------- shared_requirements.txt | 1 + 8 files changed, 40 insertions(+), 61 deletions(-) create mode 100644 sdk/tables/azure-data-tables/sdk_packaging.toml diff --git a/sdk/tables/azure-data-tables/MANIFEST.in b/sdk/tables/azure-data-tables/MANIFEST.in index 428787a39347..c6292d45f925 100644 --- a/sdk/tables/azure-data-tables/MANIFEST.in +++ b/sdk/tables/azure-data-tables/MANIFEST.in @@ -1,6 +1,7 @@ +recursive-include tests *.py *.yaml include *.md include azure/__init__.py include azure/data/__init__.py include LICENSE.txt recursive-include tests *.py -recursive-include samples *.py *.md +recursive-include samples *.py *.md \ No newline at end of file diff --git a/sdk/tables/azure-data-tables/README.md b/sdk/tables/azure-data-tables/README.md index a29e4d753e2c..9d5aaa1546f5 100644 --- a/sdk/tables/azure-data-tables/README.md +++ b/sdk/tables/azure-data-tables/README.md @@ -1,16 +1,17 @@ # Azure Data Tables client library for Python -Azure Data Tables is a NoSQL data storing service that can be accessed from anywhere in the world via authenticated calls using HTTP or HTTPS. -Tables scale as needed to support the amount of data inserted, and allow for the storing of data with non-complex accessing. -Tables scale as needed to support the amount of data inserted, and allow for the storing of data with non-complex accessing. +Azure Data Tables is a NoSQL data storing service that can be accessed from anywhere in the world via authenticated calls using HTTP or HTTPS. +Tables scale as needed to support the amount of data inserted, and allow for the storing of data with non-complex accessing. The Azure Data Tables client can be used to access Azure Storage or Cosmos accounts. -Common uses of Azure Data Tables include: -* Storing structured data in the form of tables +# Usage +* Storing structured data in the form of tables # Usage * Quickly querying data using a clustered index [Source code](https://github.com/Azure/azure-sdk-for-python/tree/master/sdk) | [Package (PyPI)](https://pypi.org) | [API reference documentation](https://aka.ms/azsdk/python/tables/docs) | [Product documentation](https://docs.microsoft.com/azure/storage/) | [Samples](https://github.com/Azure/azure-sdk-for-python/tree/master/sdk) +For code examples, see [MyService Management](https://docs.microsoft.com/python/api/overview/azure/) +on docs.microsoft.com. ## Getting started @@ -38,7 +39,6 @@ or [Azure CLI](https://docs.microsoft.com/azure/storage/common/storage-quickstar # Create a new resource group to hold the storage account - # if using an existing resource group, skip this step az group create --name MyResourceGroup --location westus2 - # Create the storage account az storage account create -n mystorageaccount -g MyResourceGroup ``` @@ -51,12 +51,11 @@ you to access the account: ```python from azure.data.tables import TableServiceClient - service = TableServiceClient(account_url="https://.table.core.windows.net/", credential=credential) ``` #### Looking up the account URL -You can find the account's table service URL using the +You can find the account's table service URL using the [Azure Portal](https://docs.microsoft.com/azure/storage/common/storage-account-overview#storage-account-endpoints), [Azure PowerShell](https://docs.microsoft.com/powershell/module/az.storage/get-azstorageaccount), or [Azure CLI](https://docs.microsoft.com/cli/azure/storage/account?view=azure-cli-latest#az-storage-account-show): @@ -77,7 +76,7 @@ The `credential` parameter may be provided in a number of different forms, depen ```python from datetime import datetime, timedelta from azure.data.tables import TableServiceClient, generate_account_sas, ResourceTypes, AccountSasPermissions - + sas_token = generate_account_sas( account_name="", account_key="", @@ -85,16 +84,15 @@ The `credential` parameter may be provided in a number of different forms, depen permission=AccountSasPermissions(read=True), expiry=datetime.utcnow() + timedelta(hours=1) ) - + table_service_client = TableServiceClient(account_url="https://.table.core.windows.net", credential=sas_token) ``` 2. To use an account [shared key](https://docs.microsoft.com/rest/api/storageservices/authenticate-with-shared-key/) - (aka account key or access key), provide the key as a string. This can be found in the Azure Portal under the "Access Keys" + (aka account key or access key), provide the key as a string. This can be found in the Azure Portal under the "Access Keys" section or by running the following Azure CLI command: ```az storage account keys list -g MyResourceGroup -n mystorageaccount``` - Use the key as the credential parameter to authenticate the client: ```python from azure.data.tables import TableServiceClient @@ -103,12 +101,11 @@ The `credential` parameter may be provided in a number of different forms, depen #### Creating the client from a connection string Depending on your use case and authorization method, you may prefer to initialize a client instance with a -connection string instead of providing the account URL and credential separately. To do this, pass the +connection string instead of providing the account URL and credential separately. To do this, pass the connection string to the client's `from_connection_string` class method: ```python from azure.data.tables import TableServiceClient - connection_string = "DefaultEndpointsProtocol=https;AccountName=xxxx;AccountKey=xxxx;EndpointSuffix=core.windows.net" service = TableServiceClient.from_connection_string(conn_str=connection_string) ``` @@ -139,14 +136,14 @@ Two different clients are provided to to interact with the various components of this client represents interaction with a specific table (which need not exist yet). It provides operations to create, delete, or update a table and includes operations to query, get, and upsert entities within it. - + ### Entities * **Create** - Adds an entity to the table. * **Delete** - Deletes an entity from the table. * **Update** - Updates an entities information by either merging or replacing the existing entity. * **Query** - Queries existing entities in a table based off of the QueryOptions (OData). * **Get** - Gets a specific entity from a table by partition and row key. -* **Upsert** - Merges or replaces an entity in a table, or if the entity does not exist, inserts the entity. +* **Upsert** - Merges or replaces an entity in a table, or if the entity does not exist, inserts the entity. ## Examples @@ -162,7 +159,6 @@ Create a table in your account ```python from azure.data.tables import TableServiceClient - table_service_client = TableServiceClient.from_connection_string(conn_str="") table_service_client.create_table(table_name="myTable") ``` @@ -172,9 +168,7 @@ Create entities in the table ```python from azure.data.tables import TableClient - my_entity = {'PartitionKey':'part','RowKey':'row'} - table_client = TableClient.from_connection_string(conn_str="", table_name="myTable") entity = table_client.create_entity(entity=my_entity) ``` @@ -184,9 +178,7 @@ Querying entities in the table ```python from azure.data.tables import TableClient - my_filter = "text eq Marker" - table_client = TableClient.from_connection_string(conn_str="", table_name="mytable") entity = table_client.query_entities(filter=my_filter) ``` @@ -246,15 +238,12 @@ headers, can be enabled on a client with the `logging_enable` argument: import sys import logging from azure.data.tables import TableServiceClient - # Create a logger for the 'azure.data.tables' SDK logger = logging.getLogger('azure.data.tables') logger.setLevel(logging.DEBUG) - # Configure a console output handler = logging.StreamHandler(stream=sys.stdout) logger.addHandler(handler) - # This client will log detailed information about its HTTP sessions, at DEBUG level service_client = TableServiceClient.from_connection_string("your_connection_string", logging_enable=True) ``` @@ -287,7 +276,7 @@ Several Azure Data Tables Python SDK samples are available to you in the SDK's G * Query entities * Update entities * Upsert entities - + ### Additional documentation For more extensive documentation on Azure Data Tables, see the [Azure Data Tables documentation](https://docs.microsoft.com/azure/storage/tables/) on docs.microsoft.com. @@ -296,4 +285,4 @@ This project welcomes contributions and suggestions. Most contributions require When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA. -This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. +This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. \ No newline at end of file diff --git a/sdk/tables/azure-data-tables/dev_requirements.txt b/sdk/tables/azure-data-tables/dev_requirements.txt index 5547db3c87f7..4a940812c683 100644 --- a/sdk/tables/azure-data-tables/dev_requirements.txt +++ b/sdk/tables/azure-data-tables/dev_requirements.txt @@ -3,4 +3,5 @@ ../../core/azure-core cryptography>=2.1.4 aiohttp>=3.0; python_version >= '3.5' - +azure-identity +../azure-data-nspkg \ No newline at end of file diff --git a/sdk/tables/azure-data-tables/sdk_packaging.toml b/sdk/tables/azure-data-tables/sdk_packaging.toml new file mode 100644 index 000000000000..e7687fdae93b --- /dev/null +++ b/sdk/tables/azure-data-tables/sdk_packaging.toml @@ -0,0 +1,2 @@ +[packaging] +auto_update = false \ No newline at end of file diff --git a/sdk/tables/azure-data-tables/setup.py b/sdk/tables/azure-data-tables/setup.py index 9cf68df37c3d..1651b14ff4cc 100644 --- a/sdk/tables/azure-data-tables/setup.py +++ b/sdk/tables/azure-data-tables/setup.py @@ -70,6 +70,7 @@ # Exclude packages that will be covered by PEP420 or nspkg 'azure', 'tests', + 'azure.data', ]), install_requires=[ "azure-core<2.0.0,>=1.2.2", @@ -77,8 +78,8 @@ # azure-data-tables ], extras_require={ - ":python_version<'3.0'": ['futures'], + ":python_version<'3.0'": ['futures', 'azure-data-nspkg<2.0.0,>=1.0.0'], ":python_version<'3.4'": ['enum34>=1.0.4'], ":python_version<'3.5'": ["typing"] }, -) +) \ No newline at end of file diff --git a/sdk/tables/azure-data-tables/tests/_shared/testcase.py b/sdk/tables/azure-data-tables/tests/_shared/testcase.py index 3dcfde01bf48..d20eb80556db 100644 --- a/sdk/tables/azure-data-tables/tests/_shared/testcase.py +++ b/sdk/tables/azure-data-tables/tests/_shared/testcase.py @@ -13,9 +13,6 @@ import time from datetime import datetime, timedelta -from azure.data.tables import ResourceTypes, AccountSasPermissions -from azure.data.tables._table_shared_access_signature import generate_account_sas - try: import unittest.mock as mock except ImportError: @@ -42,8 +39,8 @@ from io import StringIO from azure.core.credentials import AccessToken -#from azure.data.tabless import generate_account_sas, AccountSasPermissions, ResourceTypes from azure.mgmt.storage.models import StorageAccount, Endpoints +from azure.data.tables import generate_account_sas, AccountSasPermissions, ResourceTypes try: from devtools_testutils import mgmt_settings_real as settings @@ -351,6 +348,9 @@ def storage_account(): i_need_to_create_rg = not (existing_rg_name or existing_storage_name or storage_connection_string) got_storage_info_from_env = existing_storage_name or storage_connection_string + storage_name = None + rg_kwargs = {} + try: if i_need_to_create_rg: rg_name, rg_kwargs = rg_preparer._prepare_create_resource(test_case) @@ -431,11 +431,12 @@ def build_service_endpoint(service): TableTestCase._STORAGE_CONNECTION_STRING = storage_connection_string yield finally: - if not got_storage_info_from_env: - storage_preparer.remove_resource( - storage_name, - resource_group=rg - ) + if storage_name is not None: + if not got_storage_info_from_env: + storage_preparer.remove_resource( + storage_name, + resource_group=rg + ) finally: if i_need_to_create_rg: rg_preparer.remove_resource(rg_name) diff --git a/sdk/tables/tests.yml b/sdk/tables/tests.yml index 86573bce751d..4f5aaf6fec1a 100644 --- a/sdk/tables/tests.yml +++ b/sdk/tables/tests.yml @@ -5,29 +5,12 @@ jobs: parameters: BuildTargetingString: azure-data-tables ServiceDirectory: tables + AllocateResourceGroup: 'false' EnvVars: - STORAGE_ACCOUNT_NAME: $(python-storage-storage-account-name) - STORAGE_ACCOUNT_KEY: $(python-storage-storage-account-key) - STORAGE_DATA_LAKE_ACCOUNT_NAME: $(python-storage-data-lake-account-name) - STORAGE_DATA_LAKE_ACCOUNT_KEY: $(python-storage-data-lake-account-key) - BLOB_STORAGE_ACCOUNT_NAME: $(python-storage-blob-storage-account-name) - BLOB_STORAGE_ACCOUNT_KEY: $(python-storage-blob-storage-account-key) - REMOTE_STORAGE_ACCOUNT_NAME: $(python-storage-remote-storage-account-name) - REMOTE_STORAGE_ACCOUNT_KEY: $(python-storage-remote-storage-account-key) - PREMIUM_STORAGE_ACCOUNT_NAME: $(python-storage-premium-storage-account-name) - PREMIUM_STORAGE_ACCOUNT_KEY: $(python-storage-premium-storage-account-key) - OAUTH_STORAGE_ACCOUNT_NAME: $(python-storage-oauth-storage-account-name) - OAUTH_STORAGE_ACCOUNT_KEY: $(python-storage-oauth-storage-account-key) - ACTIVE_DIRECTORY_APPLICATION_ID: $(aad-azure-sdk-test-client-id) - ACTIVE_DIRECTORY_APPLICATION_SECRET: $(aad-azure-sdk-test-client-secret) - ACTIVE_DIRECTORY_TENANT_ID: $(aad-azure-sdk-test-tenant-id) - CONNECTION_STRING: $(python-storage-blob-connection-string) - BLOB_CONNECTION_STRING: $(python-storage-blob-connection-string) - PREMIUM_CONNECTION_STRING: $(python-storage-premium-connection-string) - TEST_MODE: 'RunLiveNoRecord' - AZURE_SKIP_LIVE_RECORDING: 'True' - AZURE_TEST_RUN_LIVE: 'true' AZURE_TENANT_ID: $(aad-azure-sdk-test-tenant-id) AZURE_SUBSCRIPTION_ID: $(azure-subscription-id) - AZURE_CLIENT_SECRET: $(aad-azure-sdk-test-client-secret) AZURE_CLIENT_ID: $(aad-azure-sdk-test-client-id) + AZURE_CLIENT_SECRET: $(aad-azure-sdk-test-client-secret) + TEST_MODE: 'RunLiveNoRecord' + AZURE_SKIP_LIVE_RECORDING: 'True' + AZURE_TEST_RUN_LIVE: 'true' diff --git a/shared_requirements.txt b/shared_requirements.txt index 9ff3c816582d..e8075c11025b 100644 --- a/shared_requirements.txt +++ b/shared_requirements.txt @@ -10,6 +10,7 @@ azure-common~=1.1 azure-core<2.0.0,>=1.2.2 azure-cosmosdb-table~=1.0 azure-datalake-store~=0.0.18 +azure-data-nspkg<2.0.0,>=1.0.0 azure-eventhub<6.0.0,>=5.0.0 azure-eventgrid~=1.1 azure-graphrbac~=0.40.0