4/26 release: Up with --repo/--browse, exec (ssh) command, replica co…

…mmands, log streaming commands (#72) * Skeleton code * az containerapp env show * List kube/managed environments * Create kube environment, wait doesn't work yet * Update containerapp stubs (check if it is supported now) * Containerapp env delete, polling not working yet * Added polling for create and delete * Use Microsoft.App RP for show, list, delete command * Create containerapp env using Microsoft.App RP * Add optional containerapp env create arguments * Remove old kube environment code, naming fixes * Containerapp create almost done * Done containerapp create, except for --yaml. Need to test * Containerapp show, list * Fix helptext * Containerapp delete * Containerapp update. Needs secrets api to be implemented, and testing * Add scale command * Various validations, small fixes * listSecrets API for updates, autogen log analytics for env * Use space delimiter for secrets and env variables * Verify sub is registered to Microsoft.ContainerRegistration if creating vnet enabled env, remove logs-type parameter * Containerapp create --yaml * Fix updating registry to do create or update * Fix containerapp update command. Add image-name parameter to support multi container updates. Fix updating registries, containers and secrets * started update with --yaml. Need to do create or update for when an attribute is a list of items * use space delimiter for startup_command and args, instead of comma delimiter * Traffic weights * List and show revisions * az containerapp revision restart, activate, deactivate * Add ability for users to clear args/command in az containerapp update * Various fixes, traffic weights fixes * Verify subnet subscription is registered to Microsoft.ContainerServices * GitHub Actions Update (#17) * Added models. Finished transferring Calvin's previous work. * Updated wrong models. * Updated models in custom.py, added githubactionclient. * Updated envelope to be correct. * Small bug fixes. * Updated error handling. Fixed bugs. Initial working state. * Added better error handling. * Added error messages for tokens with inappropriate access rights. * Added back get_acr_cred. * Fixed problems from merge conflict. * Updated names of imports from ._models.py to fix pylance erros. * Removed random imports. Co-authored-by: Haroon Feisal <[email protected]> * Remove --location since location must be same as managed env * Add options for flag names: --env-vars and --registry-srever * Empty string to clear env_vars * Default revisions_mode to single * Infer acr credentials if it is acr and credentials are not provided * fix help msg * if image is hosted on acr, and no registry server is supplied, infer the registry server * Added subgroups (Ingress, Registry, Secret) and updated revisions (#18) * Added ingress subgroup. * Added help for ingress. * Fixed ingress traffic help. * Added registry commands. * Updated registry remove util to clear secrets if none remaining. Added warning when updating existing registry. Added registry help. * Changed registry delete to remove. * Added error message if user tries to remove non assigned registry. * Changed registry add back to registry set. * Added secret subgroup commands. * Removed yaml support from secret set. * Changed secret add to secret set. Updated consistency between secret set and secret delete. Added secret help. Require at least one secret passed with --secrets for secret commands. * Changed param name for secret delete from --secrets to --secret-names. Updated help. * Changed registry remove to registry delete. * Fixed bug in registry delete. * Added revision mode set and revision copy. * Modified update_containerapp_yaml to support updating from non-current revision. Authored-by: Haroon Feisal <[email protected]> * More p0 fixes (#20) * Remove --registry-login-server, only allow --registry-server * Rename --environment-variables to --env-vars * If no image is supplied, use default quickstart image * Update help text (#21) * Update help text * Update punctuation * master -> main * New 1.0.1 version * Added identity commands + --assign-identity flag to containerapp create (#8) * Added identity show and assign. * Finisheed identity remove. * Added helps, updated identity remove to work with identity names instead of requiring identity resource ids. * Moved helper function to utils. * Require --identities flag when removing identities. * Added message for assign identity with no specified identity. * Added --assign-identity flag to containerapp create. * Moved assign-identity flag to containerapp create. * Fixed small logic error on remove identities when passing duplicate identities. Added warnings for certain edge cases. * Updated param definition for identity assign --identity default. * Added identity examples in help. * Made sure secrets were not removed when assigning identities. Added tolerance for [system] passed with capital letters. * Fixed error from merge. Co-authored-by: Haroon Feisal <[email protected]> * Dapr Commands (#23) * Added ingress subgroup. * Added help for ingress. * Fixed ingress traffic help. * Added registry commands. * Updated registry remove util to clear secrets if none remaining. Added warning when updating existing registry. Added registry help. * Changed registry delete to remove. * Added error message if user tries to remove non assigned registry. * Changed registry add back to registry set. * Added secret subgroup commands. * Removed yaml support from secret set. * Changed secret add to secret set. Updated consistency between secret set and secret delete. Added secret help. Require at least one secret passed with --secrets for secret commands. * Changed param name for secret delete from --secrets to --secret-names. Updated help. * Changed registry remove to registry delete. * Fixed bug in registry delete. * Added revision mode set and revision copy. * Added dapr enable and dapr disable. Need to test more. * Added list, show, set dapr component. Added dapr enable, disable. * Added delete dapr delete. * Added helps and param text. * Changed dapr delete to dapr remove to match with dapr set. * Commented out managed identity for whl file. * Uncommented. Co-authored-by: Haroon Feisal <[email protected]> * Rename --image-name to --container-name * Remove allowInsecure since it was messing with the api parsing * Fix for env var being empty string * Rename to --dapr-instrumentation-key, only infer ACR credentials if --registry-server is provided * Remove az containerapp scale * Fix delete containerapp errors * Remove ingress, dapr flags from az containerapp update/revision copy * Fix revision list -o table * Help text fix * Bump extension to 0.1.2 * Update managed identities and Dapr help text (#25) * Update managed identities and Dapr help text * Update Dapr flags * Add secretref note * Env var options + various bug fixes (#26) * Moved dapr arguments to env as a subgroup. * Added env variable options. * Changed revision mode set to revision set-mode. * Added env var options to revision copy. * Fixed revision copy bug related to env secret refs. * Changed registry and secret delete to remove. Added registry param helps. Removed replica from table output and added trafficWeight. * Updating warning text. * Updated warning text once more. * Made name optional for revision copy if from-revision flag is passed. Co-authored-by: Haroon Feisal <[email protected]> * Fixed style issues, various bug fixes (#27) * Moved dapr arguments to env as a subgroup. * Added env variable options. * Changed revision mode set to revision set-mode. * Added env var options to revision copy. * Fixed revision copy bug related to env secret refs. * Changed registry and secret delete to remove. Added registry param helps. Removed replica from table output and added trafficWeight. * Updating warning text. * Updated warning text once more. * Made name optional for revision copy if from-revision flag is passed. * Fixed whitespace style issues. * Styled clients and utils to pass pylint. * Finished client.py pylint fixes. * Fixed pylint issues. * Fixed flake8 commands and custom. * Fixed flake issues in src. * Added license header to _sdk_models. * Added confirmation for containerapp delete. Co-authored-by: Haroon Feisal <[email protected]> * Update src/containerapp/azext_containerapp/tests/latest/test_containerapp_scenario.py Co-authored-by: Xing Zhou <[email protected]> * Specific Error Types + Bugfixes (Help, remove app-subnet-resource-id, removed env-var alias, added help text for --name) (#28) * Moved dapr arguments to env as a subgroup. * Added env variable options. * Changed revision mode set to revision set-mode. * Added env var options to revision copy. * Fixed revision copy bug related to env secret refs. * Changed registry and secret delete to remove. Added registry param helps. Removed replica from table output and added trafficWeight. * Updating warning text. * Updated warning text once more. * Made name optional for revision copy if from-revision flag is passed. * Fixed whitespace style issues. * Styled clients and utils to pass pylint. * Finished client.py pylint fixes. * Fixed pylint issues. * Fixed flake8 commands and custom. * Fixed flake issues in src. * Added license header to _sdk_models. * Added confirmation for containerapp delete. * Update helps for identity, revision. Removed env-var alias for set-env-vars. Added name param help. * Removed app-subnet-resource-id. * Updated infrastructure subnet param help. * Check if containerapp resource exists before attempting to delete. * Added check before deleting managed env. * Changed error types to be more specific. * Removed check before deletion. Removed comments. Co-authored-by: Haroon Feisal <[email protected]> * Reset to 0.1.0 version, remove unneeded options-list * Update min cli core version * Fixed style issues. (#30) Co-authored-by: Haroon Feisal <[email protected]> * Fix linter issues * Use custom-show-command * Removed --ids from revision, secret, registry list. * Add linter exclusions * Fix polling on delete containerapp * Fix error handling * Add Container App Service * Fix flake linter * Fix help text * Mark extension as preview * Add python 3.9 and 3.10 as supported * Remove registries and secrets from az containerapp update, in favor of registry and secret subgroup * Fix YAML not working * Move import to inside deserialize function * Dapr moved from Template to Configuration * Use aka.ms link for containerapps yaml * Updated dapr enable/disable to current spec. * Fixed oversight. * Remove revisions-mode from containerapp update * Fixed dapr enable property names. (#47) Co-authored-by: Haroon Feisal <[email protected]> * Fix exceptions with using --yaml in containerapp create/update * Rename history msg * Include fqdn in containerapp table output * Added ingress messages. * Revert history msg * Add basic test case * Remove managed-identity support for first release of CLI * Need to investigate test flakiness * Update _help.py removing duplicate help * Added prototype of container up. * Fixed deploy from acr registry image infer credentials issue. * Tried to add source. * Added acr build. * Finished acr build functionality. * Added acr create functionality and pull registry from existing containerapp if it exists. * Fixed bugs. * Check if rg exists and create one with name if it doesn't. * initial containerapp ssh implementation * fix interactive commands (vim); handle ctrl + c instead of exiting * fix style and linter issues * Added disable verbose. Moved utils into utils.py. * fix for ssh for windows clients * fix for unix * Disable verbose now uses sdk poller so it gives running animation. * Added helps for params. Added error handling for non acr registry passed with source. Ignore param disable_warnings for create and no_wait for up. * Updated disable_warnings ignore. Removed disable_warnings from update_containerapp. * add terminal resizing * reorganize code; implement terminal resizing, add startup command param, etc * organize code, remove token from warning output * add validations, add replica commands * use the correct API for fetching default container; remove is_preview * Renamed silent to quiet. * Fixed style issues. * add log streaming, bump version number and add to HISTORY.rst * add basic ssh test * Added workspace name and fqdn to dry_run_str. Added indicators of if the resources are new or existing to dry_run_str. * fix ssh test for windows * Check RP for location when not provided. Open Dockerfile and use EXPOSE for CA port. * fix windows arrow keys after exit * fix typo, add logstream test, remove token from logstream output * Removed print statement. * Updated dockerfile expose automatic ingress feature. * Removed dry run str, added dry run obj instead. * use bearer auth; fix --command bug * add handling for smooth transition to new URL path * Fixed merge conflict. * fix merge conflicts * Create env if name passed and it doesn't exist. * Added missing import from merge. * Added prototype for new environment workflow. * Finished environment logic. * Minor updates before demo. * add 'az containerapp github up' (wip) * various fixes for demo * rearrange github up code * merge haroonf/containerappup * start up refactor * add --repo to up and refactor up * reorganize code more; fix various bugs * fix linter issues, fix lingering exec/tail improvements * update history * update output * bug fixes for --repo * fix --source bug * fix --source * minor bug fixes * Added API change. * Finished API change, added helloworld image auto ingress, checked provisioning state beforehand. * fixes for sisira's comments * fix minor typo * bug fix where commands fail if providing registry creds * Fixed style issues. * Updated help and version text. Co-authored-by: Calvin Chan <[email protected]> Co-authored-by: Haroon Feisal <[email protected]> Co-authored-by: Haroon Feisal <[email protected]> Co-authored-by: Anthony Chu <[email protected]> Co-authored-by: Xing Zhou <[email protected]> Co-authored-by: Sisira Panchagnula <[email protected]>
calvinsID · Apr 22, 2022 · 7f7f1fe · 7f7f1fe
1 parent 4e805bf
commit 7f7f1fe
Show file tree

Hide file tree

Showing 16 changed files with 2,427 additions and 82 deletions.
diff --git a/src/containerapp/HISTORY.rst b/src/containerapp/HISTORY.rst
@@ -3,6 +3,13 @@
 Release History
 ===============
 
+0.3.2
+++++++
+* Added 'az containerapp up' to create or update a container app and all associated resources (container app environment, ACR, Github Actions, resource group, etc.)
+* Open an ssh-like shell in a Container App with 'az containerapp exec'
+* Support for log streaming with 'az containerapp logs show'
+* Replica show and list commands
+
 0.3.1
 ++++++
 * Update "az containerapp github-action add" parameters: replace --docker-file-path with --context-path, add --image.

diff --git a/src/containerapp/azext_containerapp/.flake8 b/src/containerapp/azext_containerapp/.flake8
@@ -0,0 +1,4 @@
+[flake8]
+ignore =
+     W503   # line break before binary operator, not compliant with PEP 8
+     E203   # whitespace before ':', not compliant with PEP 8
diff --git a/src/containerapp/azext_containerapp/_acr_run_polling.py b/src/containerapp/azext_containerapp/_acr_run_polling.py
@@ -0,0 +1,112 @@
+# --------------------------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License. See License.txt in the project root for license information.
+# --------------------------------------------------------------------------------------------
+# pylint: disable=line-too-long, consider-using-f-string
+
+import time
+
+from msrest import Deserializer
+from msrestazure.azure_exceptions import CloudError
+from azure.cli.core.profiles import ResourceType
+from azure.cli.command_modules.acr._constants import get_acr_task_models
+from azure.core.polling import PollingMethod, LROPoller
+
+
+def get_run_with_polling(cmd,
+                         client,
+                         run_id,
+                         registry_name,
+                         resource_group_name):
+    deserializer = Deserializer(
+        {k: v for k, v in get_acr_task_models(cmd).__dict__.items() if isinstance(v, type)})
+
+    def deserialize_run(response):
+        return deserializer('Run', response)
+
+    return LROPoller(
+        client=client,
+        initial_response=client.get(
+            resource_group_name, registry_name, run_id, cls=lambda x, y, z: x),
+        deserialization_callback=deserialize_run,
+        polling_method=RunPolling(
+            cmd=cmd,
+            registry_name=registry_name,
+            run_id=run_id
+        ))
+
+
+class RunPolling(PollingMethod):  # pylint: disable=too-many-instance-attributes
+
+    def __init__(self, cmd, registry_name, run_id, timeout=30):
+        self._cmd = cmd
+        self._registry_name = registry_name
+        self._run_id = run_id
+        self._timeout = timeout
+        self._client = None
+        self._response = None  # Will hold latest received response
+        self._url = None  # The URL used to get the run
+        self._deserialize = None  # The deserializer for Run
+        self.operation_status = ""
+        self.operation_result = None
+
+    def initialize(self, client, initial_response, deserialization_callback):
+        self._client = client._client  # pylint: disable=protected-access
+        self._response = initial_response
+        self._url = initial_response.http_request.url
+        self._deserialize = deserialization_callback
+
+        self._set_operation_status(initial_response)
+
+    def run(self):
+        while not self.finished():
+            time.sleep(self._timeout)
+            self._update_status()
+
+        if self.operation_status not in get_succeeded_run_status(self._cmd):
+            from knack.util import CLIError
+            raise CLIError("The run with ID '{}' finished with unsuccessful status '{}'. "
+                           "Show run details by 'az acr task show-run -r {} --run-id {}'. "
+                           "Show run logs by 'az acr task logs -r {} --run-id {}'.".format(
+                               self._run_id,
+                               self.operation_status,
+                               self._registry_name,
+                               self._run_id,
+                               self._registry_name,
+                               self._run_id
+                           ))
+
+    def status(self):
+        return self.operation_status
+
+    def finished(self):
+        return self.operation_status in get_finished_run_status(self._cmd)
+
+    def resource(self):
+        return self.operation_result
+
+    def _set_operation_status(self, response):
+        if response.http_response.status_code == 200:
+            self.operation_result = self._deserialize(response)
+            self.operation_status = self.operation_result.status
+            return
+        raise CloudError(response)
+
+    def _update_status(self):
+        self._response = self._client._pipeline.run(  # pylint: disable=protected-access
+            self._client.get(self._url), stream=False)
+        self._set_operation_status(self._response)
+
+
+def get_succeeded_run_status(cmd):
+    RunStatus = cmd.get_models('RunStatus', resource_type=ResourceType.MGMT_CONTAINERREGISTRY, operation_group='task_runs')
+    return [RunStatus.succeeded.value]
+
+
+def get_finished_run_status(cmd):
+    RunStatus = cmd.get_models('RunStatus', resource_type=ResourceType.MGMT_CONTAINERREGISTRY, operation_group='task_runs')
+    return [RunStatus.succeeded.value,
+            RunStatus.failed.value,
+            RunStatus.canceled.value,
+            RunStatus.error.value,
+            RunStatus.timeout.value]
diff --git a/src/containerapp/azext_containerapp/_archive_utils.py b/src/containerapp/azext_containerapp/_archive_utils.py
@@ -0,0 +1,243 @@
+# --------------------------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License. See License.txt in the project root for license information.
+# --------------------------------------------------------------------------------------------
+# pylint: disable=consider-using-f-string, consider-using-with, no-member
+
+import tarfile
+import os
+import re
+import codecs
+from io import open
+import requests
+from knack.log import get_logger
+from msrestazure.azure_exceptions import CloudError
+from azure.cli.core.azclierror import (CLIInternalError)
+from azure.cli.core.profiles import ResourceType, get_sdk
+from azure.cli.command_modules.acr._azure_utils import get_blob_info
+from azure.cli.command_modules.acr._constants import TASK_VALID_VSTS_URLS
+
+logger = get_logger(__name__)
+
+
+def upload_source_code(cmd, client,
+                       registry_name,
+                       resource_group_name,
+                       source_location,
+                       tar_file_path,
+                       docker_file_path,
+                       docker_file_in_tar):
+    _pack_source_code(source_location,
+                      tar_file_path,
+                      docker_file_path,
+                      docker_file_in_tar)
+
+    size = os.path.getsize(tar_file_path)
+    unit = 'GiB'
+    for S in ['Bytes', 'KiB', 'MiB', 'GiB']:
+        if size < 1024:
+            unit = S
+            break
+        size = size / 1024.0
+
+    logger.info("Uploading archived source code from '%s'...", tar_file_path)
+    upload_url = None
+    relative_path = None
+    try:
+        source_upload_location = client.get_build_source_upload_url(
+            resource_group_name, registry_name)
+        upload_url = source_upload_location.upload_url
+        relative_path = source_upload_location.relative_path
+    except (AttributeError, CloudError) as e:
+        raise CLIInternalError("Failed to get a SAS URL to upload context. Error: {}".format(e.message)) from e
+
+    if not upload_url:
+        raise CLIInternalError("Failed to get a SAS URL to upload context.")
+
+    account_name, endpoint_suffix, container_name, blob_name, sas_token = get_blob_info(upload_url)
+    BlockBlobService = get_sdk(cmd.cli_ctx, ResourceType.DATA_STORAGE, 'blob#BlockBlobService')
+    BlockBlobService(account_name=account_name,
+                     sas_token=sas_token,
+                     endpoint_suffix=endpoint_suffix,
+                     # Increase socket timeout from default of 20s for clients with slow network connection.
+                     socket_timeout=300).create_blob_from_path(
+                         container_name=container_name,
+                         blob_name=blob_name,
+                         file_path=tar_file_path)
+    logger.info("Sending context ({0:.3f} {1}) to registry: {2}...".format(
+        size, unit, registry_name))
+    return relative_path
+
+
+def _pack_source_code(source_location, tar_file_path, docker_file_path, docker_file_in_tar):
+    logger.info("Packing source code into tar to upload...")
+
+    original_docker_file_name = os.path.basename(docker_file_path.replace("\\", os.sep))
+    ignore_list, ignore_list_size = _load_dockerignore_file(source_location, original_docker_file_name)
+    common_vcs_ignore_list = {'.git', '.gitignore', '.bzr', 'bzrignore', '.hg', '.hgignore', '.svn'}
+
+    def _ignore_check(tarinfo, parent_ignored, parent_matching_rule_index):
+        # ignore common vcs dir or file
+        if tarinfo.name in common_vcs_ignore_list:
+            logger.info("Excluding '%s' based on default ignore rules", tarinfo.name)
+            return True, parent_matching_rule_index
+
+        if ignore_list is None:
+            # if .dockerignore doesn't exists, inherit from parent
+            # eg, it will ignore the files under .git folder.
+            return parent_ignored, parent_matching_rule_index
+
+        for index, item in enumerate(ignore_list):
+            # stop checking the remaining rules whose priorities are lower than the parent matching rule
+            # at this point, current item should just inherit from parent
+            if index >= parent_matching_rule_index:
+                break
+            if re.match(item.pattern, tarinfo.name):
+                logger.debug(".dockerignore: rule '%s' matches '%s'.",
+                             item.rule, tarinfo.name)
+                return item.ignore, index
+
+        logger.debug(".dockerignore: no rule for '%s'. parent ignore '%s'",
+                     tarinfo.name, parent_ignored)
+        # inherit from parent
+        return parent_ignored, parent_matching_rule_index
+
+    with tarfile.open(tar_file_path, "w:gz") as tar:
+        # need to set arcname to empty string as the archive root path
+        _archive_file_recursively(tar,
+                                  source_location,
+                                  arcname="",
+                                  parent_ignored=False,
+                                  parent_matching_rule_index=ignore_list_size,
+                                  ignore_check=_ignore_check)
+
+        # Add the Dockerfile if it's specified.
+        # In the case of run, there will be no Dockerfile.
+        if docker_file_path:
+            docker_file_tarinfo = tar.gettarinfo(
+                docker_file_path, docker_file_in_tar)
+            with open(docker_file_path, "rb") as f:
+                tar.addfile(docker_file_tarinfo, f)
+
+
+class IgnoreRule:  # pylint: disable=too-few-public-methods
+    def __init__(self, rule):
+
+        self.rule = rule
+        self.ignore = True
+        # ! makes exceptions to exclusions
+        if rule.startswith('!'):
+            self.ignore = False
+            rule = rule[1:]  # remove !
+            # load path without leading slash in linux and windows
+            # environments (interferes with dockerignore file)
+            if rule.startswith('/'):
+                rule = rule[1:]  # remove beginning '/'
+
+        self.pattern = "^"
+        tokens = rule.split('/')
+        token_length = len(tokens)
+        for index, token in enumerate(tokens, 1):
+            # ** matches any number of directories
+            if token == "**":
+                self.pattern += ".*"  # treat **/ as **
+            else:
+                # * matches any sequence of non-seperator characters
+                # ? matches any single non-seperator character
+                # . matches dot character
+                self.pattern += token.replace(
+                    "*", "[^/]*").replace("?", "[^/]").replace(".", "\\.")
+                if index < token_length:
+                    self.pattern += "/"  # add back / if it's not the last
+        self.pattern += "$"
+
+
+def _load_dockerignore_file(source_location, original_docker_file_name):
+    # reference: https://docs.docker.com/engine/reference/builder/#dockerignore-file
+    docker_ignore_file = os.path.join(source_location, ".dockerignore")
+    docker_ignore_file_override = None
+    if original_docker_file_name != "Dockerfile":
+        docker_ignore_file_override = os.path.join(
+            source_location, "{}.dockerignore".format(original_docker_file_name))
+        if os.path.exists(docker_ignore_file_override):
+            logger.info("Overriding .dockerignore with %s", docker_ignore_file_override)
+            docker_ignore_file = docker_ignore_file_override
+
+    if not os.path.exists(docker_ignore_file):
+        return None, 0
+
+    encoding = "utf-8"
+    header = open(docker_ignore_file, "rb").read(len(codecs.BOM_UTF8))
+    if header.startswith(codecs.BOM_UTF8):
+        encoding = "utf-8-sig"
+
+    ignore_list = []
+    if docker_ignore_file == docker_ignore_file_override:
+        ignore_list.append(IgnoreRule(".dockerignore"))
+
+    for line in open(docker_ignore_file, 'r', encoding=encoding).readlines():
+        rule = line.rstrip()
+
+        # skip empty line and comment
+        if not rule or rule.startswith('#'):
+            continue
+
+        # the ignore rule at the end has higher priority
+        ignore_list = [IgnoreRule(rule)] + ignore_list
+
+    return ignore_list, len(ignore_list)
+
+
+def _archive_file_recursively(tar, name, arcname, parent_ignored, parent_matching_rule_index, ignore_check):
+    # create a TarInfo object from the file
+    tarinfo = tar.gettarinfo(name, arcname)
+
+    if tarinfo is None:
+        raise CLIInternalError("tarfile: unsupported type {}".format(name))
+
+    # check if the file/dir is ignored
+    ignored, matching_rule_index = ignore_check(
+        tarinfo, parent_ignored, parent_matching_rule_index)
+
+    if not ignored:
+        # append the tar header and data to the archive
+        if tarinfo.isreg():
+            with open(name, "rb") as f:
+                tar.addfile(tarinfo, f)
+        else:
+            tar.addfile(tarinfo)
+
+    # even the dir is ignored, its child items can still be included, so continue to scan
+    if tarinfo.isdir():
+        for f in os.listdir(name):
+            _archive_file_recursively(tar, os.path.join(name, f), os.path.join(arcname, f),
+                                      parent_ignored=ignored, parent_matching_rule_index=matching_rule_index,
+                                      ignore_check=ignore_check)
+
+
+def check_remote_source_code(source_location):
+    lower_source_location = source_location.lower()
+
+    # git
+    if lower_source_location.startswith("git@") or lower_source_location.startswith("git://"):
+        return source_location
+
+    # http
+    if lower_source_location.startswith("https://") or lower_source_location.startswith("http://") \
+       or lower_source_location.startswith("github.com/"):
+        isVSTS = any(url in lower_source_location for url in TASK_VALID_VSTS_URLS)
+        if isVSTS or re.search(r"\.git(?:#.+)?$", lower_source_location):
+            # git url must contain ".git" or be from VSTS/Azure DevOps.
+            # This is because Azure DevOps doesn't follow the standard git server convention of putting
+            # .git at the end of their URLs, so we have to special case them.
+            return source_location
+        if not lower_source_location.startswith("github.com/"):
+            # Others are tarball
+            if requests.head(source_location).status_code < 400:
+                return source_location
+            raise CLIInternalError("'{}' doesn't exist.".format(source_location))
+
+    # oci
+    if lower_source_location.startswith("oci://"):
+        return source_location
+    raise CLIInternalError("'{}' doesn't exist.".format(source_location))