Devtools (Azure#5759)

* Initial commit * Initial commit * Initial commit * wip: initial work - incomplete * Copy code from https://github.com/Azure/azure-cli/tree/master/src/azure-cli-testsdk * Add vcrpy version from azure-cli * Add VSCode ignores * Add Azure imports * Start converting CLI commands to SDK calls * Minor proofreading * Add required common and mock modules * Add deps, update pkgs and url * Package structure * Add more editor ignores * Remove azure deps and nonexistent HISTORY ref * Remove Azure mgmt dependencies * Remove azure.common dependency * Add dependency on six * Remove CLI-specific stuff * Remove CLI exception * Remove most CLI-centric patches * Remove older CLI-centric base class * Add sleep patch back in * First steps toward a unified config * Config and no-record headers Add config object to ScenarioTest Add fake header for deactivating recording Align cmdline options with vcr.py options * Move recording-deactivation header detection to right place * Add way to set default config file from scenario test * Remove 'CLI' from env vars * Reinstate patch_long_run_operation_delay * New record disabling mechanism * Modify config options for better backwards compatibility * Ignore more VS stuff * Remove jmespath dependency, update vcrpy * Remove checkers * Remove unuseds, make patches/processors kwargs * Remove checkers, used only for CLI * Update import and instance var name * Remove unused imports * line organization * Update version number and vcrpy version * Add a little more to README, add setup.cfg * Add unit tests for TestConfig * Add travis.yml and update dependencies * Update author email * Dummy push to try to trigger a Travis build * Another dummy commit for Travis * Switch to README.rst * Update Travis tag pointer * Use io to get encoding arg when reading README * Remove unnecessary import * Expose most everything in top-level namespace * ScenarioTest -> ReplayableTest; update version * Update version * Remove unused os.path import * Fix super call * Fix super call in setUp * Update ReplayableTest import * Use earlier vcrpy, update version num * Don't fix vcrpy version * Drop "cli" from default random name prefix * Add deployment to Travis * Specify universal wheel * Bump version to 0.2.2 for testing CI release process * Let preparer subclasses override create_random_name * De-"privatize" mock_in_unit_test and expose it * Bump version to 0.3.0 * Set disable_recording from kwarg * Don't treat mock_in_unit_test as a unittest * Remove skip for mock_in_unit_test, specify test dir * Update version numbers * Remove extraneous requirements * Remove outdated README.md stuff * Update install for new requirements.txt * Add 3.6-dev to pythons * Update code style and run pylint in Travis * Integrate with codecov.io * Add test cover IntegrationTestBase Also check in IntelliJ IDEA workspace configuration just in case someone else is using PyCharm * Add test case for LiveTest constructor * Add tests cover create_random_name * Add test covering get_sha1_hash * Add test cover RecordingProcessor base class * Add test covering SubscriptionRecordingProcessor * Configure code coverage using configuration file * Use unittest instead of nosetests to drive the automation * Fix a few code style issues * Rename coveragerc file * Add AccessTokenProcessor * Split intro and overview into clauses * Back to paragraphs * Update intro, add vcr link * Update intro paragraph to be more general * Remove reference to command modules * Update test and module refs; clarify VCR.py role * Remove outdated reference to class with builtin preparer * Replace intro text Use intro stolen from defunct recording_vcr_tests.md * s/ScenarioTest/ReplayableTest/ * Add note about semantic linefeeds * Make semantic linefeeds note an HTML comment * Remove CLI examples and add links to consumers * Remove doc for legacy test case class * Finish sentence * Add subclass kwarg information * Fix subscription ID removal and test new cases * Fix no-self-use linter complaint * Downgrade the version of vcrpy dependency from 1.11.1 to 1.11.0 due to a sympton similar to this kevin1024/vcrpy#318 * Bump version to 0.5.0 to prepare for release * Fix import on Py3 not-TravisCI * Fix recordmode * Tight config test * url parsing fix * fixed pylint import order * pylint import fix * added record processor for slashes * changed processor name * Bump version 0.5.0 => 0.5.1 * Fix SubscriptionIdReplacer * Update version 0.5.1 => 0.5.2 * Update __init__.py * Detect leaked LRO poller * PyLint happiness * Improve error message * wip * fix error * use base 64 * simplify * add comments * fix tests * fix lint error * use newer pylint to avoid invalid line errors * add a new test * address review feedback * address review feedback * support large response payload * remove useless single quots from the error message * setup: update version * do not use preparer model for allowing large payload * address review feedback * Support Autorest.Python 3.x LRO leak (Azure#42) * Support Autorest.Python 3.x LRO leak * PyLint happyness * Bump version to 0.5.5 * Update the resource removal sequence Also: 1. Adopt py.test over nosetest 2. Cleanup some test code * remove x-ms-authorization-auxiliary header * fix lint error * update version * Add some CI tools to devtools * Readme update * Some tests configuration * Use Pytest as test launcher * PyLint fixes * CI tools tests are Py3.6 only * PyLint CI tools only if 3.6 * CI tools tests recording * Dont't Pylint the tests * Fix incorrect usage of os.environ * Make tests X-platform valid * Fix test dependent of traceback * Robust preparer testing * PyGithub 1.40 * Release 1.1.0 * Allow clone_to_path to have both PR and SHA1 (Azure#49) * Allow clone_to_path to have both PR and SHA1 * Fix test * Update github_tools.py (Azure#55) * SingleValueReplacer.process_request properly decodes request.body if its type is byte. (Azure#56) * Remove pointless files * Move to tools * Update dev_requirement file * Don't record requests to AAD OAuth2 v2.0 endpoint * Use devtools from repo, not from PyPI * Fix mock dep
rajivnandivada · Jul 3, 2019 · 5f4a12c · 5f4a12c
1 parent 84f23ee
commit 5f4a12c
Show file tree

Hide file tree

Showing 49 changed files with 3,557 additions and 5 deletions.
diff --git a/scripts/dev_setup.py b/scripts/dev_setup.py
@@ -74,15 +74,20 @@ def pip_command(command, additional_dir=".", error_ok=False):
     ]
 )
 
-# Put azure-common in front
-if "azure-common" in content_packages:
-    content_packages.remove("azure-common")
-content_packages.insert(0, "azure-common")
+# Install tests dep first
+if "azure-devtools" in content_packages:
+    content_packages.remove("azure-devtools")
+content_packages.insert(0, "azure-devtools")
 
 if "azure-sdk-tools" in content_packages:
     content_packages.remove("azure-sdk-tools")
 content_packages.insert(1, "azure-sdk-tools")
 
+# Put azure-common in front of content package
+if "azure-common" in content_packages:
+    content_packages.remove("azure-common")
+content_packages.insert(2, "azure-common")
+
 print("Running dev setup...")
 print("Root directory '{}'\n".format(root_dir))
 
@@ -100,8 +105,9 @@ def pip_command(command, additional_dir=".", error_ok=False):
         pip_command("install {}/{}/".format(packages[package_name], package_name))
 
 # install packages
+print("Packages to install: {}".format(content_packages))
 for package_name in content_packages:
-    print("Installing {}".format(package_name))
+    print("\nInstalling {}".format(package_name))
     # if we are running dev_setup with no arguments. going after dev_requirements will be a pointless exercise
     # and waste of cycles as all the dependencies will be installed regardless.
     if os.path.isfile(

diff --git a/tools/azure-devtools/LICENSE b/tools/azure-devtools/LICENSE
@@ -0,0 +1,21 @@
+    MIT License
+
+    Copyright (c) Microsoft Corporation. All rights reserved.
+
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE
diff --git a/tools/azure-devtools/README.rst b/tools/azure-devtools/README.rst
@@ -0,0 +1,32 @@
+.. image:: https://travis-ci.org/Azure/azure-python-devtools.svg?branch=master
+    :target: https://travis-ci.org/Azure/azure-python-devtools
+
+Development tools for Python-based Azure tools
+==============================================
+
+This package contains tools to aid in developing Python-based Azure code.
+
+This includes the following components:
+
+scenario_tests
+--------------
+
+A testing framework to handle much of the busywork
+associated with testing code that interacts with Azure.
+
+ci_tools
+--------
+
+Some tooling to help developing CI tools. This includes some Git helpers,
+Github RestAPI wrapper and a Bot framework for Github issues.
+
+Contributing
+============
+
+This project has adopted the
+`Microsoft Open Source Code of Conduct <https://opensource.microsoft.com/codeofconduct/>`__.
+For more information see the
+`Code of Conduct FAQ <https://opensource.microsoft.com/codeofconduct/faq/>`__
+or contact
+`[email protected] <mailto:[email protected]>`__
+with any additional questions or comments.
diff --git a/tools/azure-devtools/dev_requirements.txt b/tools/azure-devtools/dev_requirements.txt
@@ -0,0 +1,4 @@
+-e .[ci_tools]
+
+mock;python_version<="2.7"
+pytest
diff --git a/tools/azure-devtools/doc/scenario_base_tests.md b/tools/azure-devtools/doc/scenario_base_tests.md
@@ -0,0 +1,101 @@
+# How to write ReplayableTest based VCR tests
+
+The `scenario_tests` package uses the [VCR.py](https://pypi.python.org/pypi/vcrpy) library
+to record the HTTP messages exchanged during a program run
+and play them back at a later time,
+making it useful for creating "scenario tests"
+that interact with Azure (or other) services.
+These tests can be replayed at a later time without any network activity,
+allowing us to detect changes in the Python layers
+between the code being tested and the underlying REST API.
+
+
+## Overview
+
+Tests all derive from the `ReplayableTest` class
+found in `azure_devtools.scenario_tests.base`.
+This class exposes the VCR tests using the standard Python `unittest` framework
+and allows the tests to be discovered by and debugged in Visual Studio.
+
+When you run a test,
+the test driver will automatically detect the test is unrecorded
+and record the HTTP requests and responses in a .yaml file
+(referred to by VCR.py as a "cassette").
+If the test succeeds, the cassette will be preserved
+and future playthroughs of the test will come from the cassette
+rather than using actual network communication.
+
+If the tests are run on TravisCI,
+any tests which cannot be replayed will automatically fail. 
+
+`ReplayableTest` itself derives from `IntegrationTestBase`,
+which provides some helpful methods for use in more general unit tests
+but no functionality pertaining to network communication.
+
+
+## Configuring ReplayableTest
+
+The only configuration of `ReplayableTest` that is "exposed"
+(in the sense of being accessible other than through subclassing)
+is whether tests should be run in "live" or "playback" mode.
+This can be set in the following two ways,
+of which the first takes precedence:
+* Set the environment variable `AZURE_TEST_RUN_LIVE`.
+  Any value will cause the tests to run in live mode;
+  if the variable is unset the default of playback mode will be used.
+* Specify a boolean value for `live-mode` in a configuration file,
+  the path to which must be specified by a `ReplayableTest` subclass as described below
+  (i.e. by default no config file will be read).
+  True values mean "live" mode; false ones mean "playback."
+
+"Live" and "playback" mode are actually just shorthand for recording modes
+in the underlying VCR.py package;
+they correspond to "all" and "once"
+as described in the [VCR.py documentation](http://vcrpy.readthedocs.io/en/latest/usage.html#record-modes).
+
+### Subclassing ReplayableTest and features
+
+Most customization of `ReplayableTest` is accessible only through subclassing.
+
+The two main users of `ReplayableTest` are
+[azure-cli](https://github.com/Azure/azure-cli)
+and [azure-sdk-for-python](https://github.com/Azure/azure-sdk-for-python).
+Each uses a subclass of `ReplayableTest` to add context-specific functionality
+and preserve backward compatibility with test code
+prior to the existence of `azure-devtools`.
+For example, azure-cli's [compatibility layer](https://github.com/Azure/azure-cli/tree/master/src/azure-cli-testsdk) 
+adds methods for running CLI commands and evaluating their output.
+
+Subclasses of `ReplayableTest` can configure its behavior
+by passing the following keyword arguments when they call
+its `__init__` method (probably using `super`):
+
+* `config_file`: Path to a configuration file.
+  It should be in the format described in Python's
+  [ConfigParser](https://docs.python.org/3/library/configparser.html) docs
+  and currently allows only the boolean option `live-mode`.
+* `recording_dir` and `recording_name`:
+  Directory path and file name, respectively,
+  for the recording that should be used for a given test case.
+  By default, the directory will be a `recordings` directory
+  in the same location as the file containing the test case,
+  and the file name will be the same as the test method name.
+  A `.yaml` extension will be appended to whatever is used for `recording_name`.
+* `recording_processors` and `replay_processors`:
+  Lists of `RecordingProcessor` instances for making changes to requests and responses
+  during test recording and test playback, respectively.
+  See [recording_processors.py](src/azure_devtools/scenario_tests/recording_processors.py)
+  for some examples and how to implement them.
+* `recording_patches` and `replay_patches`:
+  Lists of patches to apply to functions, methods, etc.
+  during test recording and playback, respectively.
+  See [patches.py](src/azure_devtools/scenario_tests/patches.py)
+  for some examples. Note the `mock_in_unit_test` function
+  which abstracts out some boilerplate for applying a patch.
+
+
+<!--
+Note: This document's source uses
+[semantic linefeeds](http://rhodesmill.org/brandon/2012/one-sentence-per-line/)
+to make diffs and updates clearer.
+-->
diff --git a/tools/azure-devtools/scripts/ci.sh b/tools/azure-devtools/scripts/ci.sh
@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+
+pylint src/azure_devtools
+pytest src/azure_devtools/scenario_tests/tests --cov=./
diff --git a/tools/azure-devtools/setup.cfg b/tools/azure-devtools/setup.cfg
@@ -0,0 +1,2 @@
+[bdist_wheel]
+universal=1
diff --git a/tools/azure-devtools/setup.py b/tools/azure-devtools/setup.py
@@ -0,0 +1,63 @@
+#!/usr/bin/env python
+
+# --------------------------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License. See License.txt in the project root for license information.
+# --------------------------------------------------------------------------------------------
+
+import io
+from setuptools import setup
+
+
+VERSION = "1.1.1"
+
+
+CLASSIFIERS = [
+    'Development Status :: 3 - Alpha',
+    'Intended Audience :: Developers',
+    'Programming Language :: Python',
+    'Programming Language :: Python :: 2',
+    'Programming Language :: Python :: 2.7',
+    'Programming Language :: Python :: 3',
+    'Programming Language :: Python :: 3.4',
+    'Programming Language :: Python :: 3.5',
+    'Programming Language :: Python :: 3.6',
+    'License :: OSI Approved :: MIT License',
+]
+
+
+DEPENDENCIES = [
+    'ConfigArgParse>=0.12.0',
+    'six>=1.10.0',
+    'vcrpy>=1.11.0',
+]
+
+with io.open('README.rst', 'r', encoding='utf-8') as f:
+    README = f.read()
+
+setup(
+    name='azure-devtools',
+    version=VERSION,
+    description='Microsoft Azure Development Tools for SDK',
+    long_description=README,
+    license='MIT',
+    author='Microsoft Corporation',
+    author_email='[email protected]',
+    url='https://github.com/Azure/azure-python-devtools',
+    zip_safe=False,
+    classifiers=CLASSIFIERS,
+    packages=[
+        'azure_devtools',
+        'azure_devtools.scenario_tests',
+        'azure_devtools.ci_tools',
+    ],
+    extras_require={
+        'ci_tools':[
+            "PyGithub>=1.40", # Can Merge PR after 1.36, "requests" and tests after 1.40
+            "GitPython",
+            "requests>=2.0"
+        ]
+    },
+    package_dir={'': 'src'},
+    install_requires=DEPENDENCIES,
+)
diff --git a/tools/azure-devtools/src/azure_devtools/__init__.py b/tools/azure-devtools/src/azure_devtools/__init__.py
diff --git a/tools/azure-devtools/src/azure_devtools/ci_tools/__init__.py b/tools/azure-devtools/src/azure_devtools/ci_tools/__init__.py
@@ -0,0 +1,4 @@
+# --------------------------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License. See License.txt in the project root for license information.
+# --------------------------------------------------------------------------------------------
diff --git a/tools/azure-devtools/src/azure_devtools/ci_tools/bot_framework.py b/tools/azure-devtools/src/azure_devtools/ci_tools/bot_framework.py
@@ -0,0 +1,121 @@
+from collections import namedtuple
+from functools import lru_cache
+import logging
+import os
+import re
+
+from github import Github, GithubException, UnknownObjectException
+
+from .github_tools import (
+    exception_to_github,
+)
+
+_LOGGER = logging.getLogger(__name__)
+
+
+def order(function):
+    function.bot_order = True
+    return function
+
+WebhookMetadata = namedtuple(
+    'WebhookMetadata',
+    ['repo', 'issue', 'text', 'comment']
+)
+
+def build_from_issue_comment(gh_token, body):
+    """Create a WebhookMetadata from a comment added to an issue.
+    """
+    if body["action"] in ["created", "edited"]:
+        github_con = Github(gh_token)
+        repo = github_con.get_repo(body['repository']['full_name'])
+        issue = repo.get_issue(body['issue']['number'])
+        text = body['comment']['body']
+        try:
+            comment = issue.get_comment(body['comment']['id'])
+        except UnknownObjectException:
+            # If the comment has already disapeared, skip the command
+            return None
+        return WebhookMetadata(repo, issue, text, comment)
+    return None
+
+def build_from_issues(gh_token, body):
+    """Create a WebhookMetadata from an opening issue text.
+    """
+    if body["action"] in ["opened", "edited"]:
+        github_con = Github(gh_token)
+        repo = github_con.get_repo(body['repository']['full_name'])
+        issue = repo.get_issue(body['issue']['number'])
+        text = body['issue']['body']
+        comment = issue  # It's where we update the comment: in the issue itself
+        return WebhookMetadata(repo, issue, text, comment)
+    return None
+
+@lru_cache()
+def robot_name_from_env_variable():
+    github_con = Github(os.environ["GH_TOKEN"])
+    return github_con.get_user().login
+
+
+class BotHandler:
+    def __init__(self, handler, robot_name=None, gh_token=None):
+        self.handler = handler
+        self.gh_token = gh_token or os.environ["GH_TOKEN"]
+        self.robot_name = robot_name or robot_name_from_env_variable()
+
+    def _is_myself(self, body):
+        return body['sender']['login'].lower() == self.robot_name.lower()
+
+    def issue_comment(self, body):
+        if self._is_myself(body):
+            return {'message': 'I don\'t talk to myself, I\'m not schizo'}
+        webhook_data = build_from_issue_comment(self.gh_token, body)
+        return self.manage_comment(webhook_data)
+
+    def issues(self, body):
+        if self._is_myself(body):
+            return {'message': 'I don\'t talk to myself, I\'m not schizo'}
+        webhook_data = build_from_issues(self.gh_token, body)
+        return self.manage_comment(webhook_data)
+
+    def orders(self):
+        """Return method tagged "order" in the handler.
+        """
+        return [order_cmd for order_cmd in dir(self.handler)
+                if getattr(getattr(self.handler, order_cmd), "bot_order", False)]
+
+    def manage_comment(self, webhook_data):
+        if webhook_data is None:
+            return {'message': 'Nothing for me'}
+        # Is someone talking to me:
+        message = re.search("@{} (.*)".format(self.robot_name), webhook_data.text, re.I)
+        response = None
+        if message:
+            command = message.group(1)
+            split_text = command.lower().split()
+            orderstr = split_text.pop(0)
+            if orderstr == "help":
+                response = self.help_order()
+            elif orderstr in self.orders():
+                try:  # Reaction is fun, but it's preview not prod.
+                      # Be careful, don't fail the command if we can't thumbs up...
+                    webhook_data.comment.create_reaction("+1")
+                except GithubException:
+                    pass
+                with exception_to_github(webhook_data.issue):  # Just in case
+                    response = getattr(self.handler, orderstr)(webhook_data.issue, *split_text)
+            else:
+                response = "I didn't understand your command:\n```bash\n{}\n```\nin this context, sorry :(\n".format(
+                    command
+                )
+                response += self.help_order()
+            if response:
+                webhook_data.issue.create_comment(response)
+                return {'message': response}
+        return {'message': 'Nothing for me or exception'}
+
+    def help_order(self):
+        orders = ["This is what I can do:"]
+        for orderstr in self.orders():
+            orders.append("- `{}`".format(orderstr))
+        orders.append("- `help` : this help message")
+        return "\n".join(orders)