Merge from master (Azure#27)

* Remove rdbms public CLI extension. (Azure#179)

* Worm feature and listing usage by region (Azure#181)

* vendored mgmt sdk

* sdk

* updated old recordings for new sdk

* added worm feature

enable listing usage by region

ci fixes

fixes

fixes

flake8 fixes

removed prints

* address feedback

* updated index with new extension path

* version bump in index

* Updating the dev-spaces command from aks group to ads group (Azure#183)

* Updating the dev-spaces command from aks group to ads group
 - refactoring

* Installing cli tooling, if does not exist, on remove command

* Updating min req of azure cli to 2.0.32

* Reverting index related changes
 - will send out a separate PR for this, after updating the generated whl file.

* Updating index: sha and minCliVersion (Azure#185)

* iotext v0.4.5 (Azure#188)

* iotext v0.4.5

* Fix sha256 hash.

* Bumping Dev Spaces version to 0.1.1 and updating the Index (Azure#186)

* VNET extension (Azure#176)

* VNET extension

Adding VNET extension for public repo , Pending changes :  update index.json to add the extension and uploading .whl file to blob repo will happen later .

* fixing indentation

fixing indentation

* fixing more flake8  reported issues

fixing more flake8  reported issues

* Fixing Review comments and formatting issues.

Fixing Review comments and formatting issues.

* Fixing some more issues with source static and bug fixes

Fixing some more issues with source static and bug fixes

* Renaming _client_factory to _client_vnet_factory

Renaming _client_factory to _client_vnet_factory

* Reverting the last change

Reverting the last change

* Renaming to azext_rdbms_vnet

Renaming to azext_rdbms_vnet

* More formatting issue fix

More formatting issue fix

* adding code owners

adding code owners

* Adding my email id

Adding my email id  :)

* Adding extension to index.json

wheel file is already uploaded to prodrdbmsclipackages

* fixing index-verify

fixing index-verify

* Changed the default value for eventdeliveryschema parameter to match the input event schema. (Azure#189)

* Added eventgrid preview extension to index.json.

* Made the hash lowercase.

* Regenerated using Wheel 0.30

* Updating index metadata to match with what's in the package.

* Updated the description to match what's in the package.

* Added minCliCoreVersion in index.json to match with the metadata in the package.

* Changed the default value for eventdeliveryschema parameter to match the input event schema.

* Updated version in index.json and fixed the default in the function argument list.

* Changing SHA256digest to lower case.

* Regenerated extension with 0.30.0 of wheel to resolve CI errors.

* Updating space select functionality (Azure#191)

* Updating space select
 - selecting space on resource create/select both
 - updating azds osx url

* Update

* CR

* Adding comments

* CR

* Management policy and new api-version (Azure#194)

* added new apiversion for mgmt sdk

* updated storage mgmt client for new api

* updated sdk and most recordings except show usage

* finalized commands

* usage to usages and help

* fixed excryption tests

* static check

* updated help

* Management policy recording and new version of wheel (Azure#195)

* updated some recordings

* added new wheel to index

* project url

* updated whl

* update index.json with new botservice package(minor fixes) (Azure#199)

* update index.json with new botservice package(minor fixes)

* update version on index.json to match the new package

* Updating dev-spaces-preview extension to honor azds commands update (Azure#196)

* Updating dev-spaces-preview extension to honor azds commands update

* Adding update option to force install the newer version of azds tool

* Update remove command

* Index Update

* Image-Copy: add support for sources backup by vhd files and snapshots (Azure#201)

* update descriptions and version

* Add checks to verify that the source has a managed disk

* fix procedure to name the temp storage account

* support tags and final image name

* fix lint issue

* remove debug statement

* add version 0.0.6 to index

* support sources backed by vhd blobs (copied images) and snapshots

* fix how we find the source_os_disk_id

* adding image-copy version 0.0.7 to the index

* fix issue with publishing to bot (Azure#202)

* fix missing client error during publish

* update index.json

* remove unused client variable

* Updating some params of use command (Azure#200)

* Updating some params of use command

* Updating index and bumping version

* Removing commands and just keeping the module library to be accessed by azure cli

* Index update

* disable=too-few-public-methods

* extend alotted time for script in travis (Azure#208)

* extend alotted time for script in travis

* longer CI times for surcetests

* Updates to extension documentation (Azure#210)

* doc for vendored_sdks folder

* add README to point at extension docs in azure-cli folder

* forgot to add docs readme

* DMS - Adding PG and MySQL extension  (Azure#207)

* Added functionality for Data Migration Assistant

* Renamed extension

* Updated the default database for PG

* Updated the extension description to remove any CLI references

* Renamed 'mgmt' to 'vendored_sdks'

* Fixed pylint issues

* extend alotted time for script in travis

* Fixed flake8 test results

* Updated code owners

* Added metadata

* Added license headers

* added information to storage-preview extension readme for included features (Azure#212)

* Updating az use call to azds cli (Azure#211)

* Updating az use call to azds cli
 - addressing azds updated cli

* Fix indentation error

* update

* Update

* CR: renaming the prompt variable

* readme changes (Azure#214)

* Updating output messages to be printed in default mode (Azure#215)

* Updating output messages to be printed in default mode

* Code Review Update

* increase travis wait time for source tests

* Add support for oauth connection management in botservice (Azure#213)

* add support for oauth to botservice cli

* minor fixes

* fix style warnings

* more pylint error fixes

* fix flake errors

* addressed comments. bumped up version

* Static web and hierarchical namespace features. (Azure#218)

* updated sdk

* service-properties update

* initial generic update for blob service-properties

* implemented generic update for service props

* added testing and recording update

* fixed bug

* added test

* recording

* updated mgmt sdk for storage

* added hierarchical namespace feature to account create

* version bump

* ci

* pylint and re-recorded all tests

* updated all recordings and tests for new sdks

* IoT extension release 0.5.0 (Azure#219)

* IoT extension release 0.5.0

* fix dep format issue

* change order of index dependencies

* Index storage 0.1.3 (Azure#220)

* index storage

* index edits

* added examples and new features (Azure#221)

* updated api-version of arm to cli (Azure#222)

* updated api-version of arm to cli

* converted all api-versions for resource

* Update batch extension to 2.3.0 (Azure#225)

* changes to dependency check (Azure#226)

* changes to dependency check

* CI

* Fixed: can not connect when using slot option. (Azure#227)

.github/CODEOWNERS

@@ -12,7 +12,7 @@
 
 /src/managementgroups/ @rajshah11
 
-/src/rdbms/ @rohit-joy
+/src/rdbms_vnet/ @rahulgouthamDOTgs
 
 /src/alias/ @chewong @troydai
 
@@ -31,3 +31,5 @@
 /src/dev-spaces-preview/ @saurabsa
 
 /src/mesh/ @linggengmsft
+
+/src/datamigration/ @temandr

.travis.yml

@@ -9,19 +9,19 @@ jobs:
   include:
     - stage: verify
       env: PURPOSE='SourceStatic'
-      script: ./scripts/ci/test_static.sh
+      script: travis_wait ./scripts/ci/test_static.sh
       python: 2.7
     - stage: verify
       env: PURPOSE='SourceStatic'
-      script: ./scripts/ci/test_static.sh
+      script: travis_wait ./scripts/ci/test_static.sh
       python: 3.6
     - stage: verify
       env: PURPOSE='SourceTests'
-      script: ./scripts/ci/test_source.sh
+      script: travis_wait 40 ./scripts/ci/test_source.sh
       python: 2.7
     - stage: verify
       env: PURPOSE='SourceTests'
-      script: ./scripts/ci/test_source.sh
+      script: travis_wait 40 ./scripts/ci/test_source.sh
       python: 3.6
     - stage: verify
       env: PURPOSE='IndexVerify'

README.md

@@ -28,6 +28,7 @@ Add your extension to the index to make it available in these CLI commands:
 - Extension source code goes into the `src` directory.
 - You can place your source code in this repository by creating a PR.
 - Once CI is green and it has been approved, the PR can be merged.
+- SDKs generated from [AutoRest](https://github.com/Azure/autorest) often do not pass CI static-checking. If they are vendored inside the extension, exclude them from static checking by placing them in the folder: `src/<extension root>/azext_*/vendored_sdks`.
 - Ensure that you include an appropriate owner for your extension in `.github/CODEOWNERS`.
 - Your extension artifact (i.e. `.whl`) will not live in this repository. You can publish your extension to PyPI or somewhere else such as Azure Storage.
 - If you want your extension to appear in the index.json, modify the index.

docs/README.md

@@ -0,0 +1,21 @@
+Azure CLI Extension Documention
+=======================
+Information
+------------
+
+Take a look at: [Azure CLI Extensions](https://github.com/Azure/azure-cli/tree/dev/doc/extensions)
+
+Doc Sections
+------------
+
+- [Authoring](https://github.com/Azure/azure-cli/blob/dev/doc/extensions/authoring.md) - How to author and develop an extension
+
+- [Command Guidelines](https://github.com/Azure/azure-cli/blob/dev/doc/command_guidelines.md) - Command Guidelines
+
+- [Publishing](https://github.com/Azure/azure-cli/blob/dev/doc/extensions/publishing.md) - How to publish an extension
+
+- [Summary Guidelines](https://github.com/Azure/azure-cli/blob/dev/doc/extensions/extension_summary_guidelines.md) - Summary guidelines for extensions to be published
+
+- [Extension Metadata](https://github.com/Azure/azure-cli/blob/dev/doc/extensions/metadata.md) - How to add additional extension metadata
+
+- [FAQ](https://github.com/Azure/azure-cli/blob/dev/doc/extensions/faq.md) - Commonly asked questions

scripts/ci/test_index.py

@@ -16,7 +16,7 @@
 import hashlib
 import shutil
 from wheel.install import WHEEL_INFO_RE
-from util import get_ext_metadata, get_whl_from_url, get_index_data, SKIP_DEP_CHECK
+from util import get_ext_metadata, get_whl_from_url, get_index_data, verify_dependency
 
 
 def get_sha256sum(a_file):
@@ -136,11 +136,12 @@ def test_metadata(self):
                                      "{}".format(item['filename'], json.dumps(metadata, indent=2, sort_keys=True,
                                                                               separators=(',', ': '))))
                 run_requires = metadata.get('run_requires')
-                if run_requires and ext_name not in SKIP_DEP_CHECK:
+                if run_requires:
                     deps = run_requires[0]['requires']
-                    self.assertTrue(all(not dep.startswith('azure-') for dep in deps),
-                                    "Dependencies of {} use disallowed extension dependencies. "
-                                    "Remove these dependencies: {}".format(item['filename'], deps))
+                    self.assertTrue(
+                        all(verify_dependency(dep) for dep in deps),
+                        "Dependencies of {} use disallowed extension dependencies. "
+                        "Remove these dependencies: {}".format(item['filename'], deps))
         shutil.rmtree(extensions_dir)
 
 

scripts/ci/test_source.py

@@ -18,7 +18,7 @@
 from wheel.install import WHEEL_INFO_RE
 from six import with_metaclass
 
-from util import get_ext_metadata, SRC_PATH, SKIP_DEP_CHECK
+from util import get_ext_metadata, verify_dependency, SRC_PATH
 
 
 ALL_TESTS = []
@@ -86,9 +86,9 @@ def test_source_wheels(self):
             ext_name = WHEEL_INFO_RE(filename).groupdict().get('name')
             metadata = get_ext_metadata(ext_dir, ext_file, ext_name)
             run_requires = metadata.get('run_requires')
-            if run_requires and ext_name not in SKIP_DEP_CHECK:
+            if run_requires:
                 deps = run_requires[0]['requires']
-                self.assertTrue(all(not dep.startswith('azure-') for dep in deps),
+                self.assertTrue(all(verify_dependency(dep) for dep in deps),
                                 "Dependencies of {} use disallowed extension dependencies. "
                                 "Remove these dependencies: {}".format(filename, deps))
         shutil.rmtree(built_whl_dir)

scripts/ci/util.py

@@ -8,8 +8,9 @@
 import zipfile
 from wheel.install import WHEEL_INFO_RE
 
-# Extensions to skip dep. check. Aim to keep this list empty.
-SKIP_DEP_CHECK = ['azure-cli-iot-ext']
+# Dependencies that will not be checked.
+# This is for packages starting with 'azure-' but do not use the 'azure' namespace.
+SKIP_DEP_CHECK = ['azure-batch-extensions']
 
 
 def get_repo_root():
@@ -99,3 +100,10 @@ def get_index_data():
             return json.load(f, object_pairs_hook=_catch_dup_keys)
     except ValueError as err:
         raise AssertionError("Invalid JSON in {}: {}".format(INDEX_PATH, err))
+
+
+def verify_dependency(dep):
+    # ex. "azure-batch-extensions (<3.1,>=3.0.0)", "paho-mqtt (==1.3.1)", "pyyaml"
+    # check if 'azure-' dependency, as they use 'azure' namespace.
+    dep_split = dep.split()
+    return not (dep_split and dep_split[0].startswith('azure-') and dep_split[0] not in SKIP_DEP_CHECK)