samconfig debug level logging fixed; documentation updated #2891

ssenchenko · 2021-05-19T23:15:16Z

Which issue(s) does this change fix?

#2747

Why is this change necessary?

Current SAM CLI behaviour while looking up the samconfig.toml is confusing for customers.

How does it address the issue?

printing out configuration file location or a warning if samconfig.toml hasn't been found for each command which involves configuration file parsing
fixing debug level logging to provide accurate information on the configuration file parameters
updating sam cli help messages
updating configuration file doc

What side effects does this change have?

None found

Checklist

Add input/output type hints to new functions/methods
Write design document (Do I need to write a design document?)
Write unit tests
Write/update functional tests
Write/update integration tests
make pr passes
make update-reproducible-reqs if dependencies were changed
Write documentation

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

ssenchenko · 2021-05-19T23:17:45Z

samcli/cli/cli_config_file.py

@@ -57,14 +57,12 @@ def __call__(self, config_path, config_env, cmd_names):

        samconfig = SamConfig(config_file_dir, config_file_name)

-        # Enable debug level logging by environment variable "SAM_DEBUG"
-        if os.environ.get("SAM_DEBUG", "").lower() == "true":


It never worked unless SAM_DEBUG was set manually apart from --debug flag because click doesn't create/set environment variables

--debug also loads from this envvar. In case this callback is executed before callback of --debug, the log level will not be properly set and here the debug log will not be shown.

It used to be the case but now it is not. Now --debug callback is executed before. I kept SAM_DEBUG as a back door though. Hopefully, we won't ever need it because it makes UX confusing

ssenchenko · 2021-05-19T23:22:55Z

samcli/cli/main.py

@@ -74,10 +74,6 @@ def print_cmdline_args(func):
    """

    def wrapper(*args, **kwargs):
-        if kwargs.get("config_file") and kwargs.get("config_env"):


It was inaccurate. When the template file was used outside of the project source directory, instead of giving a warning that the config was not found, it output samconfig.toml which created an illusion that the file was found.

I agree with your point that it gives an impression that the file is found. Here it gives the debugging information about the config file and config env. If the file is not found, the debug will show up later. I suggest changing the log to better describe this. Or combine it into the expanded cli parameters.

I deleted this code because I added file directory echo when config file is parsed and because the config env name was already in debug. It didn't get to output without SAM_DEBUG but now it does.

wchengru · 2021-05-19T23:38:26Z

samcli/cli/cli_config_file.py

+        if samconfig.exists():
+            click.echo(f"Config file location: {samconfig.path()}")
+        else:
+            click.secho(f"Config file '{samconfig.path()}' does not exist", fg="yellow", err=True)


The samconfig is not necessary in the project folder. I would prefer not to put it in error level.

wchengru · 2021-05-19T23:48:01Z

samcli/cli/options.py

    return click.option(
        "--debug",
+        is_eager=True,


We are using is_eager in some other options. Can you please check the callback order is correct if more than one is_eager options?

Yes. The order is okay. There is no problem in this case because --debug doesn't do anything but sets the log level.
Here is the list of eager options:

help, version (by default)

info

config-file, config-env, and the full block

template-file

guided

sriram-mv · 2021-05-26T17:12:34Z

Overall this looks good, my only concern is if showcasing a warning on configuration file not being found might confuse users who are not using configuration files already. But overall, this is a good improvement.

sriram-mv

Just some thoughts on the help text, otherwise looks good!

sriram-mv · 2021-05-26T17:23:17Z

samcli/cli/cli_config_file.py

-        "see: "
+        "Its default value is 'samconfig.toml' in project root directory. Project root directory is defined by the "
+        "template file location. When you use config file and specify --template-file SAM CLI expects samconfig.toml "
+        "and the template file to be in the same directory. Alternatively, use --config-file to point to samconfig.toml"


"When you use the default config file and specify --template-file SAM CLI expects samconfig.toml and the template file to be in the same directory. Alternatively, if --config-file is explicitly specified, it can point to a custom samconfig.toml location"

How does that sound?

Sounds better indeed. Thanks!

sriram-mv · 2021-05-26T17:23:50Z

samcli/cli/main.py

@@ -115,6 +111,15 @@ def cli(ctx):

    The AWS Serverless Application Model extends AWS CloudFormation to provide a simplified way of defining the
    Amazon API Gateway APIs, AWS Lambda functions, and Amazon DynamoDB tables needed by your serverless application.
+
+    SAM CLI commands run in the project root directory which is the directory with SAM template file
+    (template.{yml|yaml|json}). If no template file is specified explicitely, SAM CLI looks it up in the current


sriram-mv · 2021-05-26T17:25:05Z

samcli/commands/_utils/options.py

@@ -217,7 +217,12 @@ def template_click_option(include_build=True):
        callback=partial(get_or_default_template_file_name, include_build=include_build),
        show_default=True,
        is_eager=True,
-        help="AWS SAM template which references built artifacts for resources in the template. (if applicable)"
+        help="AWS SAM template which references built artifacts for resources in the template. "


(if applicable) still needs to be present.

sriram-mv · 2021-05-26T17:25:55Z

samcli/commands/_utils/options.py

+        "Template file defines the root directory of the project and allows to point SAM CLI to the directory "
+        "for build, local invocation etc. If template file is not specified explicitly SAM CLI expects it to be "
+        "in the current working directory (where it is running). When you use config file and specify --template-file "
+        "SAM CLI expects samconfig.toml and the template file to be in the same directory."


"When you use the default config file and specify --template-file SAM CLI expects samconfig.toml and the template file to be in the same directory. Alternatively, if --config-file is explicitly specified, it can point to a custom samconfig.toml location"

How does that sound?

Thanks, that's better 😃

aahung

LGTM!

aahung · 2021-05-26T20:08:09Z

samcli/cli/main.py

+    SAM CLI options can be either passed directly to the commands and/or stored in the config file (samconfig.toml),
+    which is expected to be in the project root directory by default. It is also possible to specify a custom directory
+    for the config file if necessary.
+
    You can find more in-depth guide about the SAM specification here:
    https://github.com/awslabs/serverless-application-model.


unrelated, this link is outdated

Suggested change

https://github.com/awslabs/serverless-application-model.

https://github.com/aws/serverless-application-model.

aahung · 2021-05-26T20:11:43Z

tests/integration/buildcmd/build_integ_base.py

+        if process_stdout.startswith("Config file"):
+            *_, process_stdout = process_stdout.partition("\n")


Do we need "if" here since "Config file" is always printed?

Well it's not exactly every time. There are commands which don't use it so it's better be on the safe side

sriram-mv · 2021-05-27T22:08:41Z

samcli/cli/cli_config_file.py

@@ -236,8 +234,10 @@ def decorator_customize_config_file(f):
    config_file_param_decls = ("--config-file",)
    config_file_attrs["help"] = (
        "The path and file name of the configuration file containing default parameter values to use. "
-        "Its default value is 'samconfig.toml' in project directory. For more information about configuration files, "
-        "see: "
+        "Its default value is 'samconfig.toml' in project root directory. Project root directory is defined by the "


Final tweak, I think the help text being in 3rd person makes it sound more uniform, apologies, I didnt note this on my first pass. But I'm approving the PR, feel free to change the help text to be in 3rd person format.

Changing it. Thanks

aahung · 2021-05-28T17:06:46Z

samcli/commands/_utils/options.py

-        help="AWS SAM template which references built artifacts for resources in the template. (if applicable)"
+        help="AWS SAM template which references built artifacts for resources in the template (if applicable). "
+        "Template file defines the root directory of the project and allows to point SAM CLI to the directory "
+        "for build, local invocation etc. If template file is not specified explicitly SAM CLI expects it to be "


Suggested change

"for build, local invocation etc. If template file is not specified explicitly SAM CLI expects it to be "

"for build, local invocation, etc. If template file is not specified explicitly SAM CLI expects it to be "

aahung · 2021-05-28T17:09:41Z

samcli/commands/_utils/options.py

+        "for build, local invocation etc. If template file is not specified explicitly SAM CLI expects it to be "
+        "in the current working directory (where it is running). When you use config file and specify --template-file "
+        "SAM CLI expects samconfig.toml and the template file to be in the same directory."
+        "Alternatively, if --config-file is explicitly specified, it can point to a custom samconfig.toml location."


"explicitly" seems unnecessary because --config-file, the CLI option, is either specified or not specified. There is no implicitly specified.

Suggested change

"Alternatively, if --config-file is explicitly specified, it can point to a custom samconfig.toml location."

"Alternatively, if --config-file is specified, it can point to a custom samconfig.toml location."

…ws#2891)" This reverts commit 2a13a69.

…2891)" (#2918) This reverts commit 2a13a69.

* samconfig debug level logging fixed; documentation updated * integration tests fix * help text improved Co-authored-by: Slava Senchenko <[email protected]>

…ws#2891)" (aws#2918) This reverts commit 2a13a69.

* samconfig debug level logging fixed; documentation updated * integration tests fix * help text improved Co-authored-by: Slava Senchenko <[email protected]>

…ws#2891)" (aws#2918) This reverts commit 2a13a69.

* samconfig debug level logging fixed; documentation updated * integration tests fix * help text improved Co-authored-by: Slava Senchenko <[email protected]>

…ws#2891)" (aws#2918) This reverts commit 2a13a69.

* chore: Use BUILD_TAG and JENKINS_URL to identify Jenkins env (#2805) * chore: Use BUILD_TAG instead of JENKINS_URL to identify Jenkins env * Keep JENKINS_URL * fix: Java Gradle 7 Build Test Data Issue (#2816) * Updated Deprecated compile to implementation * Updated Tabs to Spaces * Updated for Kotlin * Updated More Tabs to Spaces * Request for Comments: Auto Create ECR Repos in Guided Deploy (#2675) * Added Auto Create ECR Design Doc * Updated Format * Addressed feedback * fix(bug): Pass boto_session to SAM Translator library (#2759) When validating the sam template, SAM CLI requires credentials to get the IAM Manged Policies. SAM Translator also requires the region in order to figure out the parition. Previously, SAM Translator assumed this to be on the Env but SAM CLI could get this information from a command line argument or a profile. This commit passes the boto_session into the SAM Translator lib (v1.35.0 or later), so that SAM Translator can figure out the partition from the information passed to SAM CLI. Co-authored-by: Jacob Fuss <[email protected]> Co-authored-by: Mathieu Grandis <[email protected]> * feat: add option --container-host and --container-host-interface to sam local commands (#2806) * chore: bump version to 1.23.0 (#2824) Co-authored-by: Xia Zhao <[email protected]> * refactor: Extract git-clone functionality out of InitTemplates class (#2821) * [Refactor] extract git-clone functionality out of InitTemplates class to its own class * apply review comments * typo * apply review comments * chore: add command line options to pyinstaller build script (#2829) * chore: add command line options to pyinstaller build script * Update quotes * fix the dist folder name * update logs * trigger appveyor build * ignoring temp dirs used by dotnet (#2839) Co-authored-by: Slava Senchenko <[email protected]> * chore: Add GitHub actions to automate our issues workflow (#2521) * add github actions to automate our github issue workflow * reformat * update name format * update response message to be more precise * updated with the correct sam bot login name * updated with the correct token name * updated label name and bot name Co-authored-by: Mathieu Grandis <[email protected]> * Point numpy version to <1.20.3 (#2868) * Point numpy version to <1.19 to avoid PEP 317 failure * Update integ test python requirements which contain numpy * Fixing to numpy 1.20.2 * Revert "Fixing to numpy 1.20.2" This reverts commit a03f4d77e4b1588ecc3d0cbbe0f4c7c80ef60571. * Fixing numpy version to <1.20.3 * chore: Overhaul the development guide (#2827) * Validate default template.json (#2855) Issue: https://github.com/aws/aws-sam-cli/issues/2355 Added integration tests for `validate` command Co-authored-by: Slava Senchenko <[email protected]> Co-authored-by: _sam <[email protected]> * fix: package/deploy failure when Location/TemplateURL is virtual host S3 URL (#2785) * feat: Supports uncompression local layer zips in sam local (#2877) * refactor: refactor logs command library (#2862) * refactor logs command library * re-organize due to click usage * address comments * adding pylint disable for console consumer * make pylint happy with python 3.6 Co-authored-by: Mathieu Grandis <[email protected]> * sam init - Enable --app-template argument for Image package-type (#2864) * Enable --app-template argument for Image package-type while generating a new SAM project using 'sam init' * Fix the exception message * normalize pathes in UT to pass on windows * normalize project-template local path * fix: Ignore `.aws-sam` in sam build cache checksum (#2881) * feat: Allow dir_checksum() to accept a ignore_list * feat: Ignore .aws-sam when calculate cache md5 * fix: Fix crash when nested CFN stack has dict TemplateURL (unresolved intrinsics) (#2879) * fix: Fix crash when nested CFN stack has dict TemplateURL * Interactive flow question default answer from toml (#2850) * get questions' default answers from toml * make black happy * add more docs * rename question's attribute 'default_from_toml' to 'defaultFromToml' and rename 'valueof' to 'key' and add some docs * Add preload_value * Allow to pass toml file to interactive flow run() * Update related classes to utilize proload value context object * Update test * Add missing docstring * Remove samconfig change * Rename extra_context to context because it is required field now * Remove toml logics from this PR * Update comment Co-authored-by: Sam Liu <[email protected]> Co-authored-by: _sam <[email protected]> * Don't use layer.name in LayerBuildDefinition.__str__ (#2873) * Watchdog error (#2902) * chore: updating version of watchdog. Co-authored-by: Tarun Mall <[email protected]> * chore: Update aws_lambda_builders to 1.4.0 (#2903) * chore: Update aws_lambda_builders to 1.4.0 * Update integration tests for new maven behavior * Add integ test for PEP 600 tags * chore: Adds missing unit tests for LayerBuildDefinition in build_graph (#2883) * Adds missing unit tests for LayerBuildDefinition in build_graph * fix black formatting * fix: Build graph tests using assertTrue instead of assertEqual + added assertions (#2909) * samconfig debug level logging fixed; documentation updated (#2891) * samconfig debug level logging fixed; documentation updated * integration tests fix * help text improved Co-authored-by: Slava Senchenko <[email protected]> * chore: update to aws-sam-translator 1.36.0 (#2917) * Revert "samconfig debug level logging fixed; documentation updated (#2891)" (#2918) This reverts commit 2a13a69822660538c478118125eef50d0164995a. * chore: bump version to 1.24.0 (#2919) * fix: Windows default validate template integration test (#2924) * Enabled ability to provide tags as list in samconfig.toml file (#2912) * Enabled ability to provide tags as list in samconfig.toml file * Removed trailing white spaces and reformatted code * Added integration test for tags as list deploy command * Added integration test for tags as string from samconfig.toml Co-authored-by: Mohamed Elasmar <[email protected]> Co-authored-by: Wing Fung Lau <[email protected]> * fix: Add configparser to PyInstaller hiddenimports to resolve dependency issue from botocore (#2932) * Revert "Enabled ability to provide tags as list in samconfig.toml file (#2912)" (#2933) This reverts commit 104b5e5c528ef7e1ad0e83a5ba42316836a21e83. * chore: bump version to 1.24.1 (#2938) * chore: Update requests to 2.25.1 to remove the locking on urllib3 to 1.25 (#2929) * Updating tomlkit version as we need fix of the dataloss bug during copy() method use on Table object (#2939) * Updating tomlkit version as we need fix of the dataloss bug during copy() method use on Table object * Fixing types for tomlkit * Adding integration test for tomlkit not able to parse boolean issue. * Updating THIRD-PARTY-LICENSES file. * Parameterizing integ test filename Co-authored-by: Tarun Mall <[email protected]> Co-authored-by: _sam <[email protected]> * test: Fix the integration validate tests on Windows (#2940) * ci: Pin boto3-stubs to 1.17.90 due to a bug in 1.17.91 (#2942) * resolve pseudo region in build amd deploy comands (#2884) * resolve pseudo region from command argument or envvar if available * Revert "resolve pseudo region from command argument or envvar if available" This reverts commit abc0b2b62526f517dd633186861087fefb0f8b6e. * pass the aws-region to the BuildContext, DeployContext and Deploy command * Add integration tests * Make black happy * Temporary skip SAR build INTEGRATION TEST till we figure out the credeential issue * skip SAR tests when no credentials are available * Use the constant IntrinsicsSymbolTable.AWS_REGION instead of the string 'AWS::Region' * expand build SAR integration tests to four(all combinations of use-containr and us-east-2 region) * refactoring, merge stack_names and stack_names_with_regions together Co-authored-by: _sam <[email protected]> * fix: Catch more errors when building an image (#2857) * chore: fix canary/integration test issue (#2945) * feat: Allow tags as list input from samconfig.toml file (#2956) * Enabled ability to provide tags as list in samconfig.toml file * Removed trailing white spaces and reformatted code * Added integration test for tags as list deploy command * Added integration test for tags as string from samconfig.toml * Fixed Appveyer error by removing s3 info Co-authored-by: Mohamed Elasmar <[email protected]> Co-authored-by: Wing Fung Lau <[email protected]> * fix: Deploy integration tests for toml tags as a list (#2965) * chore: Increase awareness of same file warning during package (#2946) * chore: increase awareness of same file warning during package * fix formatting & grammar Co-authored-by: Mathieu Grandis <[email protected]> * fix: Allow the base64Encoded field in REST Api, skip validation of unknown fields and validate missing statusCode for Http Api (#2941) * fix API Gateway emulator: - skip validating the non allowed fields for Http Api Gateway, as it always skip the unknown fields - add base64Encoded as an allowed field for Rest Api gateway - base64 decoding will be always done for Http API gateway if the lambda response isBase64Encoded is true regardless the content-type - validate if statusCode is missing in case of Http API, and payload version 1.0 * - accept "true", "True", "false", "False" as valid isBase64Encoded values. - Validate on other isBase64Encoded Values - add more integration && unit test cases * fix lint && black issues * use smaller image to test Base64 response * fix: pass copy of environment variables for keeping cache valid (#2943) * fix: pass copy of environment variables for keeping cache valid * add integ tests * update docs * make black happy Co-authored-by: Qingchuan Ma <[email protected]> * fix: Skip build of Docker image if ImageUri is a valid ECR URL (#2934) (#2935) * Add condition to managed bucket policy (#2999) * Update appveyor.yml to do docker login on both dockerhub and Public ECR (#3005) (#3006) Co-authored-by: Wing Fung Lau <[email protected]> * chore: bump version to 1.25.0 (#3007) Co-authored-by: Sriram Madapusi Vasudevan <[email protected]> * temp: reduce python testing matrix (#3008) * temp: disable testing against python 3.8, and enabled 3.7 (#3009) * temp: disable testing against python 3.8, and enabled 3.7 * temp: disable testing against python 3.8, and enabled 3.7 & 3.6 * fix: enable all runtimes in python testing matrix (#3011) * revert: enable all runtimes in python testing matrix * fix indentation for yml * chore: update to aws-sam-translator 1.37.0 (#3019) * chore: bump version to 1.26.0 (#3020) * chore: Improved --resolve-s3 option documentation and deployment without s3 error messages (#2983) * Improve documentation on --resolve-s3 option and improve s3 failure messages * Changed indentation for integration test on s3 error message * Fixed a typo in description * Improve spacing on help text for resolve-s3 option * feature: new SAM command supporting on CDK, sam init, sam package, sam deploy (#2994) * Cdk support package and deploy (#352) * Refactor project type click option * Refactor IAC helper * Update callbacks handling --cdk-app and --template * Add methods for stack in iac interface; Update CFN plugin to link image assets * Refactor option validations and update package cli interface * Update commands to include iac option validations * Fix iac validation * sam package for CDK * sam package & deploy for CDK * Update option validations to deal with guided deploy * Update test for guided deploy for CDK * Upgrade lambda builder * chore: Update aws_lambda_builders to 1.4.0 (#2903) * chore: Update aws_lambda_builders to 1.4.0 * Update integration tests for new maven behavior * Add integ test for PEP 600 tags * Update to update asset parameter after pacakage * Update iac cdk unit tests * Update iac cdk unit tests * resolve PR comments * resolve PR comments Co-authored-by: _sam <[email protected]> Co-authored-by: Mohamed Elasmar <[email protected]> * SAM CLI CDK init flow: (#344) * SAM CLI CDK init flow: interactive and no interactive * fix current test cases * black reformat * Allow clone from non-master branch * trigger tests * Resolve comments * Resolve comments, fix cdk runtime list, and improve docstring and error message * fix pylint * fix pylint * Update exception name for CDK project errors * Trigger appveyor * ci: Pin boto3-stubs to 1.17.90 due to a bug in 1.17.91 (#2942) * black reformat * Cdk support package and deploy fix (#2996) * Fix --resolve-s3 --s3-bucket validation under guided flow * Fix package resource assets * Add debug * Trigger test with debug * restart docker service in linux * revert - restart docker service in linux * Update appveyor.yml to log into ECR * Revert "Update appveyor.yml to log into ECR" This reverts commit e948298f1279c973fb8b596d39942afb18a32626. * Update appveyor.yml to log into Public ECR * Update appveyor.yml to explicitly specify server for logging in dockerhub * Disable python3.7, 3.6 to run integ test without pull limitation * fix rapid version regex * Update regex * fix integ test options * fix parsing the Lambda Function Image Uri * try fixing another integ test issue * resolve the resources assets * fix two log diff error * Fix recognizing assets in CFN project * Fix artifact_exporter unit test * Fix handling packageable resources in Metadata * Fix handling of Metadata resource in artifact exporter * Fix integ test - test_deploy_without_stack_name * Handling missing stack_name in iac_validator * Add more tests * Improve package regression log * Increase rerun number on two flaky tests test_all_containers_are_initialized_before_any_invoke/test_no_new_created_containers_after_lambda_function_invoke * Fix handling of multiple assets in one resource * Fix Handling of Metadata section * enable integration test for python 3.6 * enable integration test for python 3.7 * kick off tests Co-authored-by: Wing Fung Lau <[email protected]> Co-authored-by: _sam <[email protected]> Co-authored-by: Mohamed Elasmar <[email protected]> * chore: bump SAM CLI version * Merge cdk develop branch into beta release branch (#3047) * Cdk support package and deploy (#352) * Refactor project type click option * Refactor IAC helper * Update callbacks handling --cdk-app and --template * Add methods for stack in iac interface; Update CFN plugin to link image assets * Refactor option validations and update package cli interface * Update commands to include iac option validations * Fix iac validation * sam package for CDK * sam package & deploy for CDK * Update option validations to deal with guided deploy * Update test for guided deploy for CDK * Upgrade lambda builder * chore: Update aws_lambda_builders to 1.4.0 (#2903) * chore: Update aws_lambda_builders to 1.4.0 * Update integration tests for new maven behavior * Add integ test for PEP 600 tags * Update to update asset parameter after pacakage * Update iac cdk unit tests * Update iac cdk unit tests * resolve PR comments * resolve PR comments Co-authored-by: _sam <[email protected]> Co-authored-by: Mohamed Elasmar <[email protected]> * SAM CLI CDK init flow: (#344) * SAM CLI CDK init flow: interactive and no interactive * fix current test cases * black reformat * Allow clone from non-master branch * trigger tests * Resolve comments * Resolve comments, fix cdk runtime list, and improve docstring and error message * fix pylint * fix pylint * Update exception name for CDK project errors * Trigger appveyor * ci: Pin boto3-stubs to 1.17.90 due to a bug in 1.17.91 (#2942) * black reformat * Cdk support package and deploy fix (#2996) * Fix --resolve-s3 --s3-bucket validation under guided flow * Fix package resource assets * Add debug * Trigger test with debug * restart docker service in linux * revert - restart docker service in linux * Update appveyor.yml to log into ECR * Revert "Update appveyor.yml to log into ECR" This reverts commit e948298f1279c973fb8b596d39942afb18a32626. * Update appveyor.yml to log into Public ECR * Update appveyor.yml to explicitly specify server for logging in dockerhub * Disable python3.7, 3.6 to run integ test without pull limitation * fix rapid version regex * Update regex * fix integ test options * fix parsing the Lambda Function Image Uri * try fixing another integ test issue * resolve the resources assets * fix two log diff error * Fix recognizing assets in CFN project * Fix artifact_exporter unit test * Fix handling packageable resources in Metadata * Fix handling of Metadata resource in artifact exporter * Fix integ test - test_deploy_without_stack_name * Handling missing stack_name in iac_validator * Add more tests * Improve package regression log * Increase rerun number on two flaky tests test_all_containers_are_initialized_before_any_invoke/test_no_new_created_containers_after_lambda_function_invoke * Fix handling of multiple assets in one resource * Fix Handling of Metadata section * enable integration test for python 3.6 * enable integration test for python 3.7 * kick off tests * fix: interactive creating CDK project won't direct to the correct resource (#3044) Co-authored-by: Wing Fung Lau <[email protected]> Co-authored-by: _sam <[email protected]> Co-authored-by: Mohamed Elasmar <[email protected]> * feat: Add SAM Pipeline commands (#3085) * sam pipeline bootstrap (#2811) * two-stages-pipeline plugin * typos * add docstring * make mypy happy * removing swap file * delete the two_stages_pipeline plugin as the pipeline-bootstrap command took over its responsibility * remove 'get_template_function_runtimes' function as the decision is made to not process the SAM template during pipeline init which was the only place we use the function * sam pipeline bootstrap command * move the pipelineconfig.toml file to .aws-sam * UX - rewriting Co-authored-by: Chris Rehn <[email protected]> * UX improvements * make black happy * apply review comments * UX - rewriting Co-authored-by: Chris Rehn <[email protected]> * refactor * Apply review comments * use python way of array elements assignments * Update samcli/lib/pipeline/bootstrap/stage.py Co-authored-by: _sam <[email protected]> * apply review comments * typo * read using utf-8 * create and user a safe version of the save_config method * apply review comments * rename _get_command_name to _get_command_names * don't save generated ARNs for now, will save during init * Revert "don't save generated ARNs for now, will save during init" This reverts commit d184e164022d9560131c62a826436edbc93da189. * Notify the user to rotate periodically rotate the IAM credentials * typo * Use AES instead of KMS for S3 SSE * rename Ecr to ECR and Iam to IAM * Grant lambda service explicit permissions to thhe ECR instead of relying on giving this permissions on ad-hoc while creating the container images Co-authored-by: Chris Rehn <[email protected]> Co-authored-by: _sam <[email protected]> * sam pipeline init command (#2831) * sam pipeline init command * apply review comments * apply review comments * display a message that we have successfully created the pipeline configuration file(s). * doc typo * Let 'sam pipeline init' prefills pipeline's infrastructure resources… (#2894) * Let 'sam pipeline init' prefills pipeline's infrastructure resources' values from 'sam pipeline bootstrap' results. * save bootstrapped sateg region * make black happy * exclude non-dict keys from samconfig.get_env_names method. * Rename the pipeline 'Stage' concept to 'Environment' (#2908) * Rename the pipeline 'Stage' concept to 'Environment' * typo * Rename --environment-name argument to --environment * Sam pipelines ux rename ecr repo to image repository (#2910) * Rename ecr-repo to image-repository * UT Fixes * typo * typo * feat: Support creating pipeline files directly into . without hooks (#2911) * feat: Support creating pipeline files directly into . without hooks * Integration test for pipeline init and pipeline bootstrap (#2841) * Expose Environment._get_stack_name for integ test to predict stack name * Add integ test for pipeline bootstrap * Add init integ test * small UX improvements: (#2914) * small UX improvements: 1. show a message when the user cancels a bootstrapping command. 2. Don't prompt for CI/CD provider or provider templates if there is only one choice. 3. Make PipelineFileAlreadyExistsError a UserError. 4. use the Colored class instead of fg='color' when prompting a colored message. 5. Fix a bug where we were not allowing empty response for not required questions. * Fix Integration Test: We now don't ask the user to select a provider's pipeline template if there is only one * Add docs for PipelineFileAlreadyExistsError * make black happy * Sam pipelines s3 security (#2975) * Deny non https requests for the artifacts S3 bucket * enable bucket serverside logging * add integration tests for artifacts bucket SSL-only requests and access logging * typo * Ensure the ArtifactsLoggingBucket denies non ssl requests (#2976) * Sam pipelines ux round 3 (#2979) * rename customer facing message 'CI/CD provider' to 'CI/CD system' * add a note about what 'Environment Name' is during the pipeline bootstrap guided context * Apply suggestions from code review typo Co-authored-by: Chris Rehn <[email protected]> Co-authored-by: Chris Rehn <[email protected]> * let pipeline IAM user assume only IAM roles tagged with Role=pipeline-execution-role (#2982) * Adding AWS_ prefix to displayed out. (#2993) Co-authored-by: Tarun Mall <[email protected]> * Add region to pipeline bootstrap interactive flow (#2997) * Ask AWS region in bootstrap interactive flow * Read default region from boto session first * Fix a unit test * Inform write to pipelineconfig.toml at the end of bootstrap (#3002) * Print info about pipelineconfig.toml after resources are bootstrapped * Update samcli/commands/pipeline/bootstrap/cli.py Co-authored-by: Chris Rehn <[email protected]> Co-authored-by: Chris Rehn <[email protected]> * List detected env names in pipeline init when prompt to input the env name (#3000) * Allow question.question can be resolved using key path * Pass the list of env names message (environment_names_message) into pipeline init interactive flow context * Update samcli/commands/pipeline/init/interactive_init_flow.py Co-authored-by: Chris Rehn <[email protected]> * Fix unit test (trigger pr builds) * Fix integ test * Fix integ test Co-authored-by: Chris Rehn <[email protected]> * Adding account id to bootstrap message. (#2998) * Adding account id to bootstrap message. * adding docstring * Addressing PR comments. * Adding unit tests. * Fixing unit tests. Co-authored-by: Tarun Mall <[email protected]> * Cfn creds fix (#3014) * Removing pipeline user creds from cfn output. This maintains same user exp. Co-authored-by: Tarun Mall <[email protected]> * Ux bootstrap revamp 20210706 (#3021) * Add intro paragraph to bootstrap * Add switch account prompt * Revamp stage definition prompt * Revamp existing resources prompt * Revamp security prompt * Allow answers to be changed later * Add exit message for bootstrap * Add exit message for bootstrap (1) * Add indentation to review values * Add "Below is the summary of the answers:" * Sweep pylint errors * Update unit tests * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/cli.py Co-authored-by: Chris Rehn <[email protected]> * Update unit tests * Add bold to other literals Co-authored-by: Chris Rehn <[email protected]> * Adding account condition for CFN execution role. (#3027) Co-authored-by: Tarun Mall <[email protected]> * pipeline UX revamp 20210707 (#3031) * Allow running bootstrap inside pipeline init * Select account credential source within bootstrap * Add bootstrap decorations within pipeline init * Removing ip range option from bootstrap. (#3036) * Removing ip range option from bootstrap. * Fixing unit test from UX PR. Co-authored-by: Tarun Mall <[email protected]> * Fix toml file incorrect read/write in init --bootstrap (#3037) * Temporarily removing account fix. (#3038) Co-authored-by: Tarun Mall <[email protected]> * Rename environment to stage (#3040) * Improve account source selection (#3042) * Fixing various cosmetics UX issues with pipeline workflow. (#3046) * Fixing credential to credentials * Forcing text color to yellow. * Adding new line after stage diagram. * Adding extra line after checking bootstrap message. * Renaming config -> configuration * account source -> credential source * Removing old message. * Fixing indentation in list. * Fixing bunch of indentation. * fixing f string Co-authored-by: Tarun Mall <[email protected]> * Auto skip questions if stage detected (#3045) * Autofill question if default value is presented * Allow to use index to select stage names (#3051) * Updating message when bootstrap stages are missing. (#3058) * Updating message when bootstrap stages are missing. * Fixing indendation Co-authored-by: Tarun Mall <[email protected]> * Fixing bootstrap integ tests. (#3061) * Fixing bootstrap integ tests. * Cleaning up some integ tests. * Using environment variables when running integ test on CI. * Using expression instead of full loop. * Adding instruction to use default profile on local. Co-authored-by: Tarun Mall <[email protected]> * Fix bootstrap test region (#3064) * Fix bootstrap region in integ test * Fix regions in non-interactive mode as well * Add more pipeline init integ test (#3065) * Fix existing pipeline init integ test * Add more pipeline init integ tests * Config file bug (#3066) * Validating config file after bootstrap stack creation. * Validating config file after bootstrap. Co-authored-by: Tarun Mall <[email protected]> * Fix pipeline init integ test because of pipelineconfig file exists (#3067) * Make stage name randomized to avoid race condition among multi canary runs (#3078) * Load number of stages from pipeline template (#3059) * Load number of stages from templates * Rename variable and add debug log * Add encoding to open() * Allow roles with Tag aws-sam-pipeline-codebuild-service-role to assume PipelineExecutionRole (#2950) * pipeline init UX: Ask to confirm when file exists (#3079) * Ask to confirm overriding if files already exist, or save to another directory * Add doc links (#3087) * Adding accidentally removed tests back. (#3088) Co-authored-by: Tarun Mall <[email protected]> Co-authored-by: elbayaaa <[email protected]> Co-authored-by: Chris Rehn <[email protected]> Co-authored-by: Ahmed Elbayaa <[email protected]> Co-authored-by: Tarun <[email protected]> Co-authored-by: Tarun Mall <[email protected]> * chore: bump aws-lambda-builder version to 1.5.0 (#3086) * chore: update to aws-sam-translator 1.38.0 (#3073) * ci: Update expected Jenkins file in pipeline integ test (#3090) * chore: Refine pipeline help text and update unit test (#3091) * Update --bucket help text * Update --stage help text * Update help text * Update help text * Update help text * Update help text * Update help text * Update jenkins generated files * Update some intro texts * Remove trialing spaces * Clearing pipeline integ test buckets with versioned objects. (#3094) * Clearing pipeline integ test buckets with versioned objects. * Fixing black formatting. Co-authored-by: Tarun Mall <[email protected]> * Fixing bug in bucket cleanup. (#3096) Co-authored-by: Tarun Mall <[email protected]> * Deleting bucket (#3097) Co-authored-by: Tarun Mall <[email protected]> * Revert "temp: disable testing against python 3.8, and enabled 3.7 (#3009)" (#3098) This reverts commit fe832185be09acb199b2a09ad73bf59e1553d131. Co-authored-by: Tarun Mall <[email protected]> * chore: bump SAM CLI version to 1.27.0 (#3101) * Add pipeline to pyinstaller (#3103) * Adding pipeline to pyinstaller. Co-authored-by: Tarun Mall <[email protected]> * Including stage resource yaml in pip. (#3106) * Including stage resource yaml in pip. * Bumping patch version Co-authored-by: Tarun Mall <[email protected]> * ci: Speed up unit test by caching the git clone (#3060) * ci: Speed up unit test by caching the git clone * Revert "Revert "temp: disable testing against python 3.8, and enabled 3.7"" (#3102) This reverts commit 1916bfa354b5d2612bd1bf9efd54a77e2bc66ff6. Revert "Revert "temp: disable testing against python 3.8, and enabled 3.7 (#3009)" (#3098)" (#3102) Co-authored-by: Tarun Mall <[email protected]> * fix: fixing pipeline init integration test. (#3123) * fix: fixing pipeline init integration test so that it don't break every time we update our template. * black formatting. * cleaning up not needed file. Co-authored-by: Tarun Mall <[email protected]> * chore: upgrade pylint to 2.9.0 (#3119) * chore: fix pylint failures in python3.9 * chore: fix pylint failures in python3.9 * chore: bump pylint version to 2.9.0 * fix typo * Add disabling reasons on new rules * fix: integration test case related to recent fix on ruby (#3124) * fix: add dockerhub default login server, improve logs to check docker pull limitation (#3137) * fix: add sample payload for 'sam local generate-event stepfunctions error' (#3043) * add sample payload for 'sam local generate-event stepfunctions error' * add better default for error * chore: Use BUILD_TAG and JENKINS_URL to identify Jenkins env (#2805) * chore: Use BUILD_TAG instead of JENKINS_URL to identify Jenkins env * Keep JENKINS_URL * Request for Comments: Auto Create ECR Repos in Guided Deploy (#2675) * Added Auto Create ECR Design Doc * Updated Format * Addressed feedback * fix(bug): Pass boto_session to SAM Translator library (#2759) When validating the sam template, SAM CLI requires credentials to get the IAM Manged Policies. SAM Translator also requires the region in order to figure out the parition. Previously, SAM Translator assumed this to be on the Env but SAM CLI could get this information from a command line argument or a profile. This commit passes the boto_session into the SAM Translator lib (v1.35.0 or later), so that SAM Translator can figure out the partition from the information passed to SAM CLI. Co-authored-by: Jacob Fuss <[email protected]> Co-authored-by: Mathieu Grandis <[email protected]> * feat: add option --container-host and --container-host-interface to sam local commands (#2806) * chore: bump version to 1.23.0 (#2824) Co-authored-by: Xia Zhao <[email protected]> * refactor: Extract git-clone functionality out of InitTemplates class (#2821) * [Refactor] extract git-clone functionality out of InitTemplates class to its own class * apply review comments * typo * apply review comments * ignoring temp dirs used by dotnet (#2839) Co-authored-by: Slava Senchenko <[email protected]> * chore: Add GitHub actions to automate our issues workflow (#2521) * add github actions to automate our github issue workflow * reformat * update name format * update response message to be more precise * updated with the correct sam bot login name * updated with the correct token name * updated label name and bot name Co-authored-by: Mathieu Grandis <[email protected]> * Point numpy version to <1.20.3 (#2868) * Point numpy version to <1.19 to avoid PEP 317 failure * Update integ test python requirements which contain numpy * Fixing to numpy 1.20.2 * Revert "Fixing to numpy 1.20.2" This reverts commit a03f4d77e4b1588ecc3d0cbbe0f4c7c80ef60571. * Fixing numpy version to <1.20.3 * chore: Overhaul the development guide (#2827) * Validate default template.json (#2855) Issue: https://github.com/aws/aws-sam-cli/issues/2355 Added integration tests for `validate` command Co-authored-by: Slava Senchenko <[email protected]> Co-authored-by: _sam <[email protected]> * fix: package/deploy failure when Location/TemplateURL is virtual host S3 URL (#2785) * feat: Supports uncompression local layer zips in sam local (#2877) * refactor: refactor logs command library (#2862) * refactor logs command library * re-organize due to click usage * address comments * adding pylint disable for console consumer * make pylint happy with python 3.6 Co-authored-by: Mathieu Grandis <[email protected]> * sam init - Enable --app-template argument for Image package-type (#2864) * Enable --app-template argument for Image package-type while generating a new SAM project using 'sam init' * Fix the exception message * normalize pathes in UT to pass on windows * normalize project-template local path * fix: Ignore `.aws-sam` in sam build cache checksum (#2881) * feat: Allow dir_checksum() to accept a ignore_list * feat: Ignore .aws-sam when calculate cache md5 * fix: Fix crash when nested CFN stack has dict TemplateURL (unresolved intrinsics) (#2879) * fix: Fix crash when nested CFN stack has dict TemplateURL * Interactive flow question default answer from toml (#2850) * get questions' default answers from toml * make black happy * add more docs * rename question's attribute 'default_from_toml' to 'defaultFromToml' and rename 'valueof' to 'key' and add some docs * Add preload_value * Allow to pass toml file to interactive flow run() * Update related classes to utilize proload value context object * Update test * Add missing docstring * Remove samconfig change * Rename extra_context to context because it is required field now * Remove toml logics from this PR * Update comment Co-authored-by: Sam Liu <[email protected]> Co-authored-by: _sam <[email protected]> * Don't use layer.name in LayerBuildDefinition.__str__ (#2873) * Watchdog error (#2902) * chore: updating version of watchdog. Co-authored-by: Tarun Mall <[email protected]> * chore: Adds missing unit tests for LayerBuildDefinition in build_graph (#2883) * Adds missing unit tests for LayerBuildDefinition in build_graph * fix black formatting * fix: Build graph tests using assertTrue instead of assertEqual + added assertions (#2909) * samconfig debug level logging fixed; documentation updated (#2891) * samconfig debug level logging fixed; documentation updated * integration tests fix * help text improved Co-authored-by: Slava Senchenko <[email protected]> * chore: update to aws-sam-translator 1.36.0 (#2917) * Revert "samconfig debug level logging fixed; documentation updated (#2891)" (#2918) This reverts commit 2a13a69822660538c478118125eef50d0164995a. * chore: bump version to 1.24.0 (#2919) * fix: Windows default validate template integration test (#2924) * Enabled ability to provide tags as list in samconfig.toml file (#2912) * Enabled ability to provide tags as list in samconfig.toml file * Removed trailing white spaces and reformatted code * Added integration test for tags as list deploy command * Added integration test for tags as string from samconfig.toml Co-authored-by: Mohamed Elasmar <[email protected]> Co-authored-by: Wing Fung Lau <[email protected]> * fix: Add configparser to PyInstaller hiddenimports to resolve dependency issue from botocore (#2932) * Revert "Enabled ability to provide tags as list in samconfig.toml file (#2912)" (#2933) This reverts commit 104b5e5c528ef7e1ad0e83a5ba42316836a21e83. * chore: bump version to 1.24.1 (#2938) * chore: Update requests to 2.25.1 to remove the locking on urllib3 to 1.25 (#2929) * Updating tomlkit version as we need fix of the dataloss bug during copy() method use on Table object (#2939) * Updating tomlkit version as we need fix of the dataloss bug during copy() method use on Table object * Fixing types for tomlkit * Adding integration test for tomlkit not able to parse boolean issue. * Updating THIRD-PARTY-LICENSES file. * Parameterizing integ test filename Co-authored-by: Tarun Mall <[email protected]> Co-authored-by: _sam <[email protected]> * test: Fix the integration validate tests on Windows (#2940) * resolve pseudo region in build amd deploy comands (#2884) * resolve pseudo region from command argument or envvar if available * Revert "resolve pseudo region from command argument or envvar if available" This reverts commit abc0b2b62526f517dd633186861087fefb0f8b6e. * pass the aws-region to the BuildContext, DeployContext and Deploy command * Add integration tests * Make black happy * Temporary skip SAR build INTEGRATION TEST till we figure out the credeential issue * skip SAR tests when no credentials are available * Use the constant IntrinsicsSymbolTable.AWS_REGION instead of the string 'AWS::Region' * expand build SAR integration tests to four(all combinations of use-containr and us-east-2 region) * refactoring, merge stack_names and stack_names_with_regions together Co-authored-by: _sam <[email protected]> * fix: Catch more errors when building an image (#2857) * chore: fix canary/integration test issue (#2945) * feat: Allow tags as list input from samconfig.toml file (#2956) * Enabled ability to provide tags as list in samconfig.toml file * Removed trailing white spaces and reformatted code * Added integration test for tags as list deploy command * Added integration test for tags as string from samconfig.toml * Fixed Appveyer error by removing s3 info Co-authored-by: Mohamed Elasmar <[email protected]> Co-authored-by: Wing Fung Lau <[email protected]> * fix: Deploy integration tests for toml tags as a list (#2965) * chore: Increase awareness of same file warning during package (#2946) * chore: increase awareness of same file warning during package * fix formatting & grammar Co-authored-by: Mathieu Grandis <[email protected]> * fix: Allow the base64Encoded field in REST Api, skip validation of unknown fields and validate missing statusCode for Http Api (#2941) * fix API Gateway emulator: - skip validating the non allowed fields for Http Api Gateway, as it always skip the unknown fields - add base64Encoded as an allowed field for Rest Api gateway - base64 decoding will be always done for Http API gateway if the lambda response isBase64Encoded is true regardless the content-type - validate if statusCode is missing in case of Http API, and payload version 1.0 * - accept "true", "True", "false", "False" as valid isBase64Encoded values. - Validate on other isBase64Encoded Values - add more integration && unit test cases * fix lint && black issues * use smaller image to test Base64 response * fix: pass copy of environment variables for keeping cache valid (#2943) * fix: pass copy of environment variables for keeping cache valid * add integ tests * update docs * make black happy Co-authored-by: Qingchuan Ma <[email protected]> * fix: Skip build of Docker image if ImageUri is a valid ECR URL (#2934) (#2935) * Add condition to managed bucket policy (#2999) * chore: bump version to 1.25.0 (#3007) Co-authored-by: Sriram Madapusi Vasudevan <[email protected]> * temp: reduce python testing matrix (#3008) * temp: disable testing against python 3.8, and enabled 3.7 (#3009) * temp: disable testing against python 3.8, and enabled 3.7 * temp: disable testing against python 3.8, and enabled 3.7 & 3.6 * chore: update to aws-sam-translator 1.37.0 (#3019) * chore: bump version to 1.26.0 (#3020) * chore: Improved --resolve-s3 option documentation and deployment without s3 error messages (#2983) * Improve documentation on --resolve-s3 option and improve s3 failure messages * Changed indentation for integration test on s3 error message * Fixed a typo in description * Improve spacing on help text for resolve-s3 option * feat: Add SAM Pipeline commands (#3085) * sam pipeline bootstrap (#2811) * two-stages-pipeline plugin * typos * add docstring * make mypy happy * removing swap file * delete the two_stages_pipeline plugin as the pipeline-bootstrap command took over its responsibility * remove 'get_template_function_runtimes' function as the decision is made to not process the SAM template during pipeline init which was the only place we use the function * sam pipeline bootstrap command * move the pipelineconfig.toml file to .aws-sam * UX - rewriting Co-authored-by: Chris Rehn <[email protected]> * UX improvements * make black happy * apply review comments * UX - rewriting Co-authored-by: Chris Rehn <[email protected]> * refactor * Apply review comments * use python way of array elements assignments * Update samcli/lib/pipeline/bootstrap/stage.py Co-authored-by: _sam <[email protected]> * apply review comments * typo * read using utf-8 * create and user a safe version of the save_config method * apply review comments * rename _get_command_name to _get_command_names * don't save generated ARNs for now, will save during init * Revert "don't save generated ARNs for now, will save during init" This reverts commit d184e164022d9560131c62a826436edbc93da189. * Notify the user to rotate periodically rotate the IAM credentials * typo * Use AES instead of KMS for S3 SSE * rename Ecr to ECR and Iam to IAM * Grant lambda service explicit permissions to thhe ECR instead of relying on giving this permissions on ad-hoc while creating the container images Co-authored-by: Chris Rehn <[email protected]> Co-authored-by: _sam <[email protected]> * sam pipeline init command (#2831) * sam pipeline init command * apply review comments * apply review comments * display a message that we have successfully created the pipeline configuration file(s). * doc typo * Let 'sam pipeline init' prefills pipeline's infrastructure resources… (#2894) * Let 'sam pipeline init' prefills pipeline's infrastructure resources' values from 'sam pipeline bootstrap' results. * save bootstrapped sateg region * make black happy * exclude non-dict keys from samconfig.get_env_names method. * Rename the pipeline 'Stage' concept to 'Environment' (#2908) * Rename the pipeline 'Stage' concept to 'Environment' * typo * Rename --environment-name argument to --environment * Sam pipelines ux rename ecr repo to image repository (#2910) * Rename ecr-repo to image-repository * UT Fixes * typo * typo * feat: Support creating pipeline files directly into . without hooks (#2911) * feat: Support creating pipeline files directly into . without hooks * Integration test for pipeline init and pipeline bootstrap (#2841) * Expose Environment._get_stack_name for integ test to predict stack name * Add integ test for pipeline bootstrap * Add init integ test * small UX improvements: (#2914) * small UX improvements: 1. show a message when the user cancels a bootstrapping command. 2. Don't prompt for CI/CD provider or provider templates if there is only one choice. 3. Make PipelineFileAlreadyExistsError a UserError. 4. use the Colored class instead of fg='color' when prompting a colored message. 5. Fix a bug where we were not allowing empty response for not required questions. * Fix Integration Test: We now don't ask the user to select a provider's pipeline template if there is only one * Add docs for PipelineFileAlreadyExistsError * make black happy * Sam pipelines s3 security (#2975) * Deny non https requests for the artifacts S3 bucket * enable bucket serverside logging * add integration tests for artifacts bucket SSL-only requests and access logging * typo * Ensure the ArtifactsLoggingBucket denies non ssl requests (#2976) * Sam pipelines ux round 3 (#2979) * rename customer facing message 'CI/CD provider' to 'CI/CD system' * add a note about what 'Environment Name' is during the pipeline bootstrap guided context * Apply suggestions from code review typo Co-authored-by: Chris Rehn <[email protected]> Co-authored-by: Chris Rehn <[email protected]> * let pipeline IAM user assume only IAM roles tagged with Role=pipeline-execution-role (#2982) * Adding AWS_ prefix to displayed out. (#2993) Co-authored-by: Tarun Mall <[email protected]> * Add region to pipeline bootstrap interactive flow (#2997) * Ask AWS region in bootstrap interactive flow * Read default region from boto session first * Fix a unit test * Inform write to pipelineconfig.toml at the end of bootstrap (#3002) * Print info about pipelineconfig.toml after resources are bootstrapped * Update samcli/commands/pipeline/bootstrap/cli.py Co-authored-by: Chris Rehn <[email protected]> Co-authored-by: Chris Rehn <[email protected]> * List detected env names in pipeline init when prompt to input the env name (#3000) * Allow question.question can be resolved using key path * Pass the list of env names message (environment_names_message) into pipeline init interactive flow context * Update samcli/commands/pipeline/init/interactive_init_flow.py Co-authored-by: Chris Rehn <[email protected]> * Fix unit test (trigger pr builds) * Fix integ test * Fix integ test Co-authored-by: Chris Rehn <[email protected]> * Adding account id to bootstrap message. (#2998) * Adding account id to bootstrap message. * adding docstring * Addressing PR comments. * Adding unit tests. * Fixing unit tests. Co-authored-by: Tarun Mall <[email protected]> * Cfn creds fix (#3014) * Removing pipeline user creds from cfn output. This maintains same user exp. Co-authored-by: Tarun Mall <[email protected]> * Ux bootstrap revamp 20210706 (#3021) * Add intro paragraph to bootstrap * Add switch account prompt * Revamp stage definition prompt * Revamp existing resources prompt * Revamp security prompt * Allow answers to be changed later * Add exit message for bootstrap * Add exit message for bootstrap (1) * Add indentation to review values * Add "Below is the summary of the answers:" * Sweep pylint errors * Update unit tests * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/cli.py Co-authored-by: Chris Rehn <[email protected]> * Update unit tests * Add bold to other literals Co-authored-by: Chris Rehn <[email protected]> * Adding account condition for CFN execution role. (#3027) Co-authored-by: Tarun Mall <[email protected]> * pipeline UX revamp 20210707 (#3031) * Allow running bootstrap inside pipeline init * Select account credential source within bootstrap * Add bootstrap decorations within pipeline init * Removing ip range option from bootstrap. (#3036) * Removing ip range option from bootstrap. * Fixing unit test from UX PR. Co-authored-by: Tarun Mall <[email protected]> * Fix toml file incorrect read/write in init --bootstrap (#3037) * Temporarily removing account fix. (#3038) Co-authored-by: Tarun Mall <[email protected]> * Rename environment to stage (#3040) * Improve account source selection (#3042) * Fixing various cosmetics UX issues with pipeline workflow. (#3046) * Fixing credential to credentials * Forcing text color to yellow. * Adding new line after stage diagram. * Adding extra line after checking bootstrap message. * Renaming config -> configuration * account source -> credential source * Removing old message. * Fixing indentation in list. * Fixing bunch of indentation. * fixing f string Co-authored-by: Tarun Mall <[email protected]> * Auto skip questions if stage detected (#3045) * Autofill question if default value is presented * Allow to use index to select stage names (#3051) * Updating message when bootstrap stages are missing. (#3058) * Updating message when bootstrap stages are missing. * Fixing indendation Co-authored-by: Tarun Mall <[email protected]> * Fixing bootstrap integ tests. (#3061) * Fixing bootstrap integ tests. * Cleaning up some integ tests. * Using environment variables when running integ test on CI. * Using expression instead of full loop. * Adding instruction to use default profile on local. Co-authored-by: Tarun Mall <[email protected]> * Fix bootstrap test region (#3064) * Fix bootstrap region in integ test * Fix regions in non-interactive mode as well * Add more pipeline init integ test (#3065) * Fix existing pipeline init integ test * Add more pipeline init integ tests * Config file bug (#3066) * Validating config file after bootstrap stack creation. * Validating config file after bootstrap. Co-authored-by: Tarun Mall <[email protected]> * Fix pipeline init integ test because of pipelineconfig file exists (#3067) * Make stage name randomized to avoid race condition among multi canary runs (#3078) * Load number of stages from pipeline template (#3059) * Load number of stages from templates * Rename variable and add debug log * Add encoding to open() * Allow roles with Tag aws-sam-pipeline-codebuild-service-role to assume PipelineExecutionRole (#2950) * pipeline init UX: Ask to confirm when file exists (#3079) * Ask to confirm overriding if files already exist, or save to another directory * Add doc links (#3087) * Adding accidentally removed tests back. (#3088) Co-authored-by: Tarun Mall <[email protected]> Co-authored-by: elbayaaa <[email protected]> Co-authored-by: Chris Rehn <[email protected]> Co-authored-by: Ahmed Elbayaa <[email protected]> Co-authored-by: Tarun <[email protected]> Co-authored-by: Tarun Mall <[email protected]> * chore: bump aws-lambda-builder version to 1.5.0 (#3086) * chore: update to aws-sam-translator 1.38.0 (#3073) * ci: Update expected Jenkins file in pipeline integ test (#3090) * chore: Refine pipeline help text and update unit test (#3091) * Update --bucket help text * Update --stage help text * Update help text * Update help text * Update help text * Update help text * Update help text * Update jenkins generated files * Update some intro texts * Remove trialing spaces * Clearing pipeline integ test buckets with versioned objects. (#3094) * Clearing pipeline integ test buckets with versioned objects. * Fixing black formatting. Co-authored-by: Tarun Mall <[email protected]> * Fixing bug in bucket cleanup. (#3096) Co-authored-by: Tarun Mall <[email protected]> * Deleting bucket (#3097) Co-authored-by: Tarun Mall <[email protected]> * chore: bump SAM CLI version to 1.27.0 (#3101) * Add pipeline to pyinstaller (#3103) * Adding pipeline to pyinstaller. Co-authored-by: Tarun Mall <[email protected]> * Including stage resource yaml in pip. (#3106) * Including stage resource yaml in pip. * Bumping patch version Co-authored-by: Tarun Mall <[email protected]> * ci: Speed up unit test by caching the git clone (#3060) * ci: Speed up unit test by caching the git clone * Revert "Revert "temp: disable testing against python 3.8, and enabled 3.7"" (#3102) This reverts commit 1916bfa354b5d2612bd1bf9efd54a77e2bc66ff6. Revert "Revert "temp: disable testing against python 3.8, and enabled 3.7 (#3009)" (#3098)" (#3102) Co-authored-by: Tarun Mall <[email protected]> * fix: fixing pipeline init integration test. (#3123) * fix: fixing pipeline init integration test so that it don't break every time we update our template. * black formatting. * cleaning up not needed file. Co-authored-by: Tarun Mall <[email protected]> * chore: upgrade pylint to 2.9.0 (#3119) * chore: fix pylint failures in python3.9 * chore: fix pylint failures in python3.9 * chore: bump pylint version to 2.9.0 * fix typo * Add disabling reasons on new rules * fix: integration test case related to recent fix on ruby (#3124) * fix: add dockerhub default login server, improve logs to check docker pull limitation (#3137) * fix: add sample payload for 'sam local generate-event stepfunctions error' (#3043) * add sample payload for 'sam local generate-event stepfunctions error' * add better default for error * fix conflicts * chore: removed unused code which was using pre-defined managed policy… (#3030) * chore: removed unused code which was using pre-defined managed policy list and used in a sam translator wrapper, but the code path is not used. * make black * feat(public-ecr): Download Emulation images (#3152) Co-authored-by: Jacob Fuss <[email protected]> Co-authored-by: Mathieu Grandis <[email protected]> * resolve PR comments * fix(integ): Use images that are in public ecr (#3162) Co-authored-by: Jacob Fuss <[email protected]> * Add ECR credentials for windows test (#3160) * Add ECR credentials for windows test * Remove the dockerhub env vars * fix(integ): Fix Invalid image tag errors (#3163) Co-authored-by: Jacob Fuss <[email protected]> * Install aws cli in the windows test jobs (#3164) * Add ECR credentials for windows test * Remove the dockerhub env vars * install aws cli in the windows test jobs * fix(integ): Add missing image to have deploy integ tests work (#3165) Co-authored-by: Jacob Fuss <[email protected]> * chore: Update dateparser to 1.0, update TestBuildCommand_PythonFunctions_Images test requirement (#3172) * chore: Update dateparser to 1.0 * Move public ECR limited test cases to Canary tests * Python39 support for samcli (#3173) * Python39 support for samcli (#354) * Python39 support for samcli * Updated reproducible-linux.txt and lambda_build_container.py for test purpose * Revert files after testing * updated integ test * updated appveyor * updated to appveyor * Update python3.9 appveyor config * update windows python3.9 executable path * update appveyor * fix lint and windows python appveyor script * bump version of lambda-builder to 1.6.0 Co-authored-by: jonife <[email protected]> * chore: bump SAM CLI version to 1.28.0 (#3174) * skip CDK build integration test cases to run during make-pr because of the limitation of ecr * run black Co-authored-by: _sam <[email protected]> Co-authored-by: Cosh_ <[email protected]> Co-authored-by: Jacob Fuss <[email protected]> Co-authored-by: Jacob Fuss <[email protected]> Co-authored-by: Mathieu Grandis <[email protected]> Co-authored-by: Xia Zhao <[email protected]> Co-authored-by: Xia Zhao <[email protected]> Co-authored-by: elbayaaa <[email protected]> Co-authored-by: Raymond Wang <[email protected]> Co-authored-by: Slava Senchenko <[email protected]> Co-authored-by: Slava Senchenko <[email protected]> Co-authored-by: mingkun2020 <[email protected]> Co-authored-by: Wing Fung Lau <[email protected]> Co-authored-by: Mehmet Nuri Deveci <[email protected]> Co-authored-by: Sam Liu <[email protected]> Co-authored-by: Arturo García <[email protected]> Co-authored-by: Tarun <[email protected]> Co-authored-by: Tarun Mall <[email protected]> Co-authored-by: Qingchuan Ma <[email protected]> Co-authored-by: hnnasit <[email protected]> Co-authored-by: Renato Valenzuela <[email protected]> Co-authored-by: Alexis Facques <[email protected]> Co-authored-by: Sriram Madapusi Vasudevan <[email protected]> Co-authored-by: Chris Rehn <[email protected]> Co-authored-by: Ahmed Elbayaa <[email protected]> Co-authored-by: Ruperto Torres <[email protected]> Co-authored-by: jonife <[email protected]>

* chore: Use BUILD_TAG and JENKINS_URL to identify Jenkins env (#2805) * chore: Use BUILD_TAG instead of JENKINS_URL to identify Jenkins env * Keep JENKINS_URL * fix: Java Gradle 7 Build Test Data Issue (#2816) * Updated Deprecated compile to implementation * Updated Tabs to Spaces * Updated for Kotlin * Updated More Tabs to Spaces * Request for Comments: Auto Create ECR Repos in Guided Deploy (#2675) * Added Auto Create ECR Design Doc * Updated Format * Addressed feedback * fix(bug): Pass boto_session to SAM Translator library (#2759) When validating the sam template, SAM CLI requires credentials to get the IAM Manged Policies. SAM Translator also requires the region in order to figure out the parition. Previously, SAM Translator assumed this to be on the Env but SAM CLI could get this information from a command line argument or a profile. This commit passes the boto_session into the SAM Translator lib (v1.35.0 or later), so that SAM Translator can figure out the partition from the information passed to SAM CLI. Co-authored-by: Jacob Fuss <[email protected]> Co-authored-by: Mathieu Grandis <[email protected]> * feat: add option --container-host and --container-host-interface to sam local commands (#2806) * chore: bump version to 1.23.0 (#2824) Co-authored-by: Xia Zhao <[email protected]> * refactor: Extract git-clone functionality out of InitTemplates class (#2821) * [Refactor] extract git-clone functionality out of InitTemplates class to its own class * apply review comments * typo * apply review comments * chore: add command line options to pyinstaller build script (#2829) * chore: add command line options to pyinstaller build script * Update quotes * fix the dist folder name * update logs * trigger appveyor build * ignoring temp dirs used by dotnet (#2839) Co-authored-by: Slava Senchenko <[email protected]> * chore: Add GitHub actions to automate our issues workflow (#2521) * add github actions to automate our github issue workflow * reformat * update name format * update response message to be more precise * updated with the correct sam bot login name * updated with the correct token name * updated label name and bot name Co-authored-by: Mathieu Grandis <[email protected]> * Point numpy version to <1.20.3 (#2868) * Point numpy version to <1.19 to avoid PEP 317 failure * Update integ test python requirements which contain numpy * Fixing to numpy 1.20.2 * Revert "Fixing to numpy 1.20.2" This reverts commit a03f4d77e4b1588ecc3d0cbbe0f4c7c80ef60571. * Fixing numpy version to <1.20.3 * chore: Overhaul the development guide (#2827) * Validate default template.json (#2855) Issue: https://github.com/aws/aws-sam-cli/issues/2355 Added integration tests for `validate` command Co-authored-by: Slava Senchenko <[email protected]> Co-authored-by: _sam <[email protected]> * fix: package/deploy failure when Location/TemplateURL is virtual host S3 URL (#2785) * feat: Supports uncompression local layer zips in sam local (#2877) * refactor: refactor logs command library (#2862) * refactor logs command library * re-organize due to click usage * address comments * adding pylint disable for console consumer * make pylint happy with python 3.6 Co-authored-by: Mathieu Grandis <[email protected]> * sam init - Enable --app-template argument for Image package-type (#2864) * Enable --app-template argument for Image package-type while generating a new SAM project using 'sam init' * Fix the exception message * normalize pathes in UT to pass on windows * normalize project-template local path * fix: Ignore `.aws-sam` in sam build cache checksum (#2881) * feat: Allow dir_checksum() to accept a ignore_list * feat: Ignore .aws-sam when calculate cache md5 * fix: Fix crash when nested CFN stack has dict TemplateURL (unresolved intrinsics) (#2879) * fix: Fix crash when nested CFN stack has dict TemplateURL * Interactive flow question default answer from toml (#2850) * get questions' default answers from toml * make black happy * add more docs * rename question's attribute 'default_from_toml' to 'defaultFromToml' and rename 'valueof' to 'key' and add some docs * Add preload_value * Allow to pass toml file to interactive flow run() * Update related classes to utilize proload value context object * Update test * Add missing docstring * Remove samconfig change * Rename extra_context to context because it is required field now * Remove toml logics from this PR * Update comment Co-authored-by: Sam Liu <[email protected]> Co-authored-by: _sam <[email protected]> * Don't use layer.name in LayerBuildDefinition.__str__ (#2873) * Watchdog error (#2902) * chore: updating version of watchdog. Co-authored-by: Tarun Mall <[email protected]> * chore: Update aws_lambda_builders to 1.4.0 (#2903) * chore: Update aws_lambda_builders to 1.4.0 * Update integration tests for new maven behavior * Add integ test for PEP 600 tags * chore: Adds missing unit tests for LayerBuildDefinition in build_graph (#2883) * Adds missing unit tests for LayerBuildDefinition in build_graph * fix black formatting * fix: Build graph tests using assertTrue instead of assertEqual + added assertions (#2909) * samconfig debug level logging fixed; documentation updated (#2891) * samconfig debug level logging fixed; documentation updated * integration tests fix * help text improved Co-authored-by: Slava Senchenko <[email protected]> * chore: update to aws-sam-translator 1.36.0 (#2917) * Revert "samconfig debug level logging fixed; documentation updated (#2891)" (#2918) This reverts commit 2a13a69822660538c478118125eef50d0164995a. * chore: bump version to 1.24.0 (#2919) * fix: Windows default validate template integration test (#2924) * Enabled ability to provide tags as list in samconfig.toml file (#2912) * Enabled ability to provide tags as list in samconfig.toml file * Removed trailing white spaces and reformatted code * Added integration test for tags as list deploy command * Added integration test for tags as string from samconfig.toml Co-authored-by: Mohamed Elasmar <[email protected]> Co-authored-by: Wing Fung Lau <[email protected]> * fix: Add configparser to PyInstaller hiddenimports to resolve dependency issue from botocore (#2932) * Revert "Enabled ability to provide tags as list in samconfig.toml file (#2912)" (#2933) This reverts commit 104b5e5c528ef7e1ad0e83a5ba42316836a21e83. * chore: bump version to 1.24.1 (#2938) * chore: Update requests to 2.25.1 to remove the locking on urllib3 to 1.25 (#2929) * Updating tomlkit version as we need fix of the dataloss bug during copy() method use on Table object (#2939) * Updating tomlkit version as we need fix of the dataloss bug during copy() method use on Table object * Fixing types for tomlkit * Adding integration test for tomlkit not able to parse boolean issue. * Updating THIRD-PARTY-LICENSES file. * Parameterizing integ test filename Co-authored-by: Tarun Mall <[email protected]> Co-authored-by: _sam <[email protected]> * test: Fix the integration validate tests on Windows (#2940) * ci: Pin boto3-stubs to 1.17.90 due to a bug in 1.17.91 (#2942) * resolve pseudo region in build amd deploy comands (#2884) * resolve pseudo region from command argument or envvar if available * Revert "resolve pseudo region from command argument or envvar if available" This reverts commit abc0b2b62526f517dd633186861087fefb0f8b6e. * pass the aws-region to the BuildContext, DeployContext and Deploy command * Add integration tests * Make black happy * Temporary skip SAR build INTEGRATION TEST till we figure out the credeential issue * skip SAR tests when no credentials are available * Use the constant IntrinsicsSymbolTable.AWS_REGION instead of the string 'AWS::Region' * expand build SAR integration tests to four(all combinations of use-containr and us-east-2 region) * refactoring, merge stack_names and stack_names_with_regions together Co-authored-by: _sam <[email protected]> * fix: Catch more errors when building an image (#2857) * chore: fix canary/integration test issue (#2945) * feat: Allow tags as list input from samconfig.toml file (#2956) * Enabled ability to provide tags as list in samconfig.toml file * Removed trailing white spaces and reformatted code * Added integration test for tags as list deploy command * Added integration test for tags as string from samconfig.toml * Fixed Appveyer error by removing s3 info Co-authored-by: Mohamed Elasmar <[email protected]> Co-authored-by: Wing Fung Lau <[email protected]> * fix: Deploy integration tests for toml tags as a list (#2965) * chore: Increase awareness of same file warning during package (#2946) * chore: increase awareness of same file warning during package * fix formatting & grammar Co-authored-by: Mathieu Grandis <[email protected]> * fix: Allow the base64Encoded field in REST Api, skip validation of unknown fields and validate missing statusCode for Http Api (#2941) * fix API Gateway emulator: - skip validating the non allowed fields for Http Api Gateway, as it always skip the unknown fields - add base64Encoded as an allowed field for Rest Api gateway - base64 decoding will be always done for Http API gateway if the lambda response isBase64Encoded is true regardless the content-type - validate if statusCode is missing in case of Http API, and payload version 1.0 * - accept "true", "True", "false", "False" as valid isBase64Encoded values. - Validate on other isBase64Encoded Values - add more integration && unit test cases * fix lint && black issues * use smaller image to test Base64 response * fix: pass copy of environment variables for keeping cache valid (#2943) * fix: pass copy of environment variables for keeping cache valid * add integ tests * update docs * make black happy Co-authored-by: Qingchuan Ma <[email protected]> * fix: Skip build of Docker image if ImageUri is a valid ECR URL (#2934) (#2935) * Add condition to managed bucket policy (#2999) * Update appveyor.yml to do docker login on both dockerhub and Public ECR (#3005) (#3006) Co-authored-by: Wing Fung Lau <[email protected]> * chore: bump version to 1.25.0 (#3007) Co-authored-by: Sriram Madapusi Vasudevan <[email protected]> * temp: reduce python testing matrix (#3008) * temp: disable testing against python 3.8, and enabled 3.7 (#3009) * temp: disable testing against python 3.8, and enabled 3.7 * temp: disable testing against python 3.8, and enabled 3.7 & 3.6 * fix: enable all runtimes in python testing matrix (#3011) * revert: enable all runtimes in python testing matrix * fix indentation for yml * chore: update to aws-sam-translator 1.37.0 (#3019) * chore: bump version to 1.26.0 (#3020) * chore: Improved --resolve-s3 option documentation and deployment without s3 error messages (#2983) * Improve documentation on --resolve-s3 option and improve s3 failure messages * Changed indentation for integration test on s3 error message * Fixed a typo in description * Improve spacing on help text for resolve-s3 option * Merge back to cdk develop branch (#3049) * feature: new SAM command supporting on CDK, sam init, sam package, sam deploy (#2994) * Cdk support package and deploy (#352) * Refactor project type click option * Refactor IAC helper * Update callbacks handling --cdk-app and --template * Add methods for stack in iac interface; Update CFN plugin to link image assets * Refactor option validations and update package cli interface * Update commands to include iac option validations * Fix iac validation * sam package for CDK * sam package & deploy for CDK * Update option validations to deal with guided deploy * Update test for guided deploy for CDK * Upgrade lambda builder * chore: Update aws_lambda_builders to 1.4.0 (#2903) * chore: Update aws_lambda_builders to 1.4.0 * Update integration tests for new maven behavior * Add integ test for PEP 600 tags * Update to update asset parameter after pacakage * Update iac cdk unit tests * Update iac cdk unit tests * resolve PR comments * resolve PR comments Co-authored-by: _sam <[email protected]> Co-authored-by: Mohamed Elasmar <[email protected]> * SAM CLI CDK init flow: (#344) * SAM CLI CDK init flow: interactive and no interactive * fix current test cases * black reformat * Allow clone from non-master branch * trigger tests * Resolve comments * Resolve comments, fix cdk runtime list, and improve docstring and error message * fix pylint * fix pylint * Update exception name for CDK project errors * Trigger appveyor * ci: Pin boto3-stubs to 1.17.90 due to a bug in 1.17.91 (#2942) * black reformat * Cdk support package and deploy fix (#2996) * Fix --resolve-s3 --s3-bucket validation under guided flow * Fix package resource assets * Add debug * Trigger test with debug * restart docker service in linux * revert - restart docker service in linux * Update appveyor.yml to log into ECR * Revert "Update appveyor.yml to log into ECR" This reverts commit e948298f1279c973fb8b596d39942afb18a32626. * Update appveyor.yml to log into Public ECR * Update appveyor.yml to explicitly specify server for logging in dockerhub * Disable python3.7, 3.6 to run integ test without pull limitation * fix rapid version regex * Update regex * fix integ test options * fix parsing the Lambda Function Image Uri * try fixing another integ test issue * resolve the resources assets * fix two log diff error * Fix recognizing assets in CFN project * Fix artifact_exporter unit test * Fix handling packageable resources in Metadata * Fix handling of Metadata resource in artifact exporter * Fix integ test - test_deploy_without_stack_name * Handling missing stack_name in iac_validator * Add more tests * Improve package regression log * Increase rerun number on two flaky tests test_all_containers_are_initialized_before_any_invoke/test_no_new_created_containers_after_lambda_function_invoke * Fix handling of multiple assets in one resource * Fix Handling of Metadata section * enable integration test for python 3.6 * enable integration test for python 3.7 * kick off tests Co-authored-by: Wing Fung Lau <[email protected]> Co-authored-by: _sam <[email protected]> Co-authored-by: Mohamed Elasmar <[email protected]> * chore: bump SAM CLI version * Merge cdk develop branch into beta release branch (#3047) * Cdk support package and deploy (#352) * Refactor project type click option * Refactor IAC helper * Update callbacks handling --cdk-app and --template * Add methods for stack in iac interface; Update CFN plugin to link image assets * Refactor option validations and update package cli interface * Update commands to include iac option validations * Fix iac validation * sam package for CDK * sam package & deploy for CDK * Update option validations to deal with guided deploy * Update test for guided deploy for CDK * Upgrade lambda builder * chore: Update aws_lambda_builders to 1.4.0 (#2903) * chore: Update aws_lambda_builders to 1.4.0 * Update integration tests for new maven behavior * Add integ test for PEP 600 tags * Update to update asset parameter after pacakage * Update iac cdk unit tests * Update iac cdk unit tests * resolve PR comments * resolve PR comments Co-authored-by: _sam <[email protected]> Co-authored-by: Mohamed Elasmar <[email protected]> * SAM CLI CDK init flow: (#344) * SAM CLI CDK init flow: interactive and no interactive * fix current test cases * black reformat * Allow clone from non-master branch * trigger tests * Resolve comments * Resolve comments, fix cdk runtime list, and improve docstring and error message * fix pylint * fix pylint * Update exception name for CDK project errors * Trigger appveyor * ci: Pin boto3-stubs to 1.17.90 due to a bug in 1.17.91 (#2942) * black reformat * Cdk support package and deploy fix (#2996) * Fix --resolve-s3 --s3-bucket validation under guided flow * Fix package resource assets * Add debug * Trigger test with debug * restart docker service in linux * revert - restart docker service in linux * Update appveyor.yml to log into ECR * Revert "Update appveyor.yml to log into ECR" This reverts commit e948298f1279c973fb8b596d39942afb18a32626. * Update appveyor.yml to log into Public ECR * Update appveyor.yml to explicitly specify server for logging in dockerhub * Disable python3.7, 3.6 to run integ test without pull limitation * fix rapid version regex * Update regex * fix integ test options * fix parsing the Lambda Function Image Uri * try fixing another integ test issue * resolve the resources assets * fix two log diff error * Fix recognizing assets in CFN project * Fix artifact_exporter unit test * Fix handling packageable resources in Metadata * Fix handling of Metadata resource in artifact exporter * Fix integ test - test_deploy_without_stack_name * Handling missing stack_name in iac_validator * Add more tests * Improve package regression log * Increase rerun number on two flaky tests test_all_containers_are_initialized_before_any_invoke/test_no_new_created_containers_after_lambda_function_invoke * Fix handling of multiple assets in one resource * Fix Handling of Metadata section * enable integration test for python 3.6 * enable integration test for python 3.7 * kick off tests * fix: interactive creating CDK project won't direct to the correct resource (#3044) Co-authored-by: Wing Fung Lau <[email protected]> Co-authored-by: _sam <[email protected]> Co-authored-by: Mohamed Elasmar <[email protected]> Co-authored-by: Wing Fung Lau <[email protected]> Co-authored-by: _sam <[email protected]> Co-authored-by: Mohamed Elasmar <[email protected]> * feat: Add SAM Pipeline commands (#3085) * sam pipeline bootstrap (#2811) * two-stages-pipeline plugin * typos * add docstring * make mypy happy * removing swap file * delete the two_stages_pipeline plugin as the pipeline-bootstrap command took over its responsibility * remove 'get_template_function_runtimes' function as the decision is made to not process the SAM template during pipeline init which was the only place we use the function * sam pipeline bootstrap command * move the pipelineconfig.toml file to .aws-sam * UX - rewriting Co-authored-by: Chris Rehn <[email protected]> * UX improvements * make black happy * apply review comments * UX - rewriting Co-authored-by: Chris Rehn <[email protected]> * refactor * Apply review comments * use python way of array elements assignments * Update samcli/lib/pipeline/bootstrap/stage.py Co-authored-by: _sam <[email protected]> * apply review comments * typo * read using utf-8 * create and user a safe version of the save_config method * apply review comments * rename _get_command_name to _get_command_names * don't save generated ARNs for now, will save during init * Revert "don't save generated ARNs for now, will save during init" This reverts commit d184e164022d9560131c62a826436edbc93da189. * Notify the user to rotate periodically rotate the IAM credentials * typo * Use AES instead of KMS for S3 SSE * rename Ecr to ECR and Iam to IAM * Grant lambda service explicit permissions to thhe ECR instead of relying on giving this permissions on ad-hoc while creating the container images Co-authored-by: Chris Rehn <[email protected]> Co-authored-by: _sam <[email protected]> * sam pipeline init command (#2831) * sam pipeline init command * apply review comments * apply review comments * display a message that we have successfully created the pipeline configuration file(s). * doc typo * Let 'sam pipeline init' prefills pipeline's infrastructure resources… (#2894) * Let 'sam pipeline init' prefills pipeline's infrastructure resources' values from 'sam pipeline bootstrap' results. * save bootstrapped sateg region * make black happy * exclude non-dict keys from samconfig.get_env_names method. * Rename the pipeline 'Stage' concept to 'Environment' (#2908) * Rename the pipeline 'Stage' concept to 'Environment' * typo * Rename --environment-name argument to --environment * Sam pipelines ux rename ecr repo to image repository (#2910) * Rename ecr-repo to image-repository * UT Fixes * typo * typo * feat: Support creating pipeline files directly into . without hooks (#2911) * feat: Support creating pipeline files directly into . without hooks * Integration test for pipeline init and pipeline bootstrap (#2841) * Expose Environment._get_stack_name for integ test to predict stack name * Add integ test for pipeline bootstrap * Add init integ test * small UX improvements: (#2914) * small UX improvements: 1. show a message when the user cancels a bootstrapping command. 2. Don't prompt for CI/CD provider or provider templates if there is only one choice. 3. Make PipelineFileAlreadyExistsError a UserError. 4. use the Colored class instead of fg='color' when prompting a colored message. 5. Fix a bug where we were not allowing empty response for not required questions. * Fix Integration Test: We now don't ask the user to select a provider's pipeline template if there is only one * Add docs for PipelineFileAlreadyExistsError * make black happy * Sam pipelines s3 security (#2975) * Deny non https requests for the artifacts S3 bucket * enable bucket serverside logging * add integration tests for artifacts bucket SSL-only requests and access logging * typo * Ensure the ArtifactsLoggingBucket denies non ssl requests (#2976) * Sam pipelines ux round 3 (#2979) * rename customer facing message 'CI/CD provider' to 'CI/CD system' * add a note about what 'Environment Name' is during the pipeline bootstrap guided context * Apply suggestions from code review typo Co-authored-by: Chris Rehn <[email protected]> Co-authored-by: Chris Rehn <[email protected]> * let pipeline IAM user assume only IAM roles tagged with Role=pipeline-execution-role (#2982) * Adding AWS_ prefix to displayed out. (#2993) Co-authored-by: Tarun Mall <[email protected]> * Add region to pipeline bootstrap interactive flow (#2997) * Ask AWS region in bootstrap interactive flow * Read default region from boto session first * Fix a unit test * Inform write to pipelineconfig.toml at the end of bootstrap (#3002) * Print info about pipelineconfig.toml after resources are bootstrapped * Update samcli/commands/pipeline/bootstrap/cli.py Co-authored-by: Chris Rehn <[email protected]> Co-authored-by: Chris Rehn <[email protected]> * List detected env names in pipeline init when prompt to input the env name (#3000) * Allow question.question can be resolved using key path * Pass the list of env names message (environment_names_message) into pipeline init interactive flow context * Update samcli/commands/pipeline/init/interactive_init_flow.py Co-authored-by: Chris Rehn <[email protected]> * Fix unit test (trigger pr builds) * Fix integ test * Fix integ test Co-authored-by: Chris Rehn <[email protected]> * Adding account id to bootstrap message. (#2998) * Adding account id to bootstrap message. * adding docstring * Addressing PR comments. * Adding unit tests. * Fixing unit tests. Co-authored-by: Tarun Mall <[email protected]> * Cfn creds fix (#3014) * Removing pipeline user creds from cfn output. This maintains same user exp. Co-authored-by: Tarun Mall <[email protected]> * Ux bootstrap revamp 20210706 (#3021) * Add intro paragraph to bootstrap * Add switch account prompt * Revamp stage definition prompt * Revamp existing resources prompt * Revamp security prompt * Allow answers to be changed later * Add exit message for bootstrap * Add exit message for bootstrap (1) * Add indentation to review values * Add "Below is the summary of the answers:" * Sweep pylint errors * Update unit tests * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/cli.py Co-authored-by: Chris Rehn <[email protected]> * Update unit tests * Add bold to other literals Co-authored-by: Chris Rehn <[email protected]> * Adding account condition for CFN execution role. (#3027) Co-authored-by: Tarun Mall <[email protected]> * pipeline UX revamp 20210707 (#3031) * Allow running bootstrap inside pipeline init * Select account credential source within bootstrap * Add bootstrap decorations within pipeline init * Removing ip range option from bootstrap. (#3036) * Removing ip range option from bootstrap. * Fixing unit test from UX PR. Co-authored-by: Tarun Mall <[email protected]> * Fix toml file incorrect read/write in init --bootstrap (#3037) * Temporarily removing account fix. (#3038) Co-authored-by: Tarun Mall <[email protected]> * Rename environment to stage (#3040) * Improve account source selection (#3042) * Fixing various cosmetics UX issues with pipeline workflow. (#3046) * Fixing credential to credentials * Forcing text color to yellow. * Adding new line after stage diagram. * Adding extra line after checking bootstrap message. * Renaming config -> configuration * account source -> credential source * Removing old message. * Fixing indentation in list. * Fixing bunch of indentation. * fixing f string Co-authored-by: Tarun Mall <[email protected]> * Auto skip questions if stage detected (#3045) * Autofill question if default value is presented * Allow to use index to select stage names (#3051) * Updating message when bootstrap stages are missing. (#3058) * Updating message when bootstrap stages are missing. * Fixing indendation Co-authored-by: Tarun Mall <[email protected]> * Fixing bootstrap integ tests. (#3061) * Fixing bootstrap integ tests. * Cleaning up some integ tests. * Using environment variables when running integ test on CI. * Using expression instead of full loop. * Adding instruction to use default profile on local. Co-authored-by: Tarun Mall <[email protected]> * Fix bootstrap test region (#3064) * Fix bootstrap region in integ test * Fix regions in non-interactive mode as well * Add more pipeline init integ test (#3065) * Fix existing pipeline init integ test * Add more pipeline init integ tests * Config file bug (#3066) * Validating config file after bootstrap stack creation. * Validating config file after bootstrap. Co-authored-by: Tarun Mall <[email protected]> * Fix pipeline init integ test because of pipelineconfig file exists (#3067) * Make stage name randomized to avoid race condition among multi canary runs (#3078) * Load number of stages from pipeline template (#3059) * Load number of stages from templates * Rename variable and add debug log * Add encoding to open() * Allow roles with Tag aws-sam-pipeline-codebuild-service-role to assume PipelineExecutionRole (#2950) * pipeline init UX: Ask to confirm when file exists (#3079) * Ask to confirm overriding if files already exist, or save to another directory * Add doc links (#3087) * Adding accidentally removed tests back. (#3088) Co-authored-by: Tarun Mall <[email protected]> Co-authored-by: elbayaaa <[email protected]> Co-authored-by: Chris Rehn <[email protected]> Co-authored-by: Ahmed Elbayaa <[email protected]> Co-authored-by: Tarun <[email protected]> Co-authored-by: Tarun Mall <[email protected]> * chore: bump aws-lambda-builder version to 1.5.0 (#3086) * chore: update to aws-sam-translator 1.38.0 (#3073) * ci: Update expected Jenkins file in pipeline integ test (#3090) * chore: Refine pipeline help text and update unit test (#3091) * Update --bucket help text * Update --stage help text * Update help text * Update help text * Update help text * Update help text * Update help text * Update jenkins generated files * Update some intro texts * Remove trialing spaces * Clearing pipeline integ test buckets with versioned objects. (#3094) * Clearing pipeline integ test buckets with versioned objects. * Fixing black formatting. Co-authored-by: Tarun Mall <[email protected]> * Fixing bug in bucket cleanup. (#3096) Co-authored-by: Tarun Mall <[email protected]> * Deleting bucket (#3097) Co-authored-by: Tarun Mall <[email protected]> * Revert "temp: disable testing against python 3.8, and enabled 3.7 (#3009)" (#3098) This reverts commit fe832185be09acb199b2a09ad73bf59e1553d131. Co-authored-by: Tarun Mall <[email protected]> * chore: bump SAM CLI version to 1.27.0 (#3101) * Add pipeline to pyinstaller (#3103) * Adding pipeline to pyinstaller. Co-authored-by: Tarun Mall <[email protected]> * Including stage resource yaml in pip. (#3106) * Including stage resource yaml in pip. * Bumping patch version Co-authored-by: Tarun Mall <[email protected]> * Add integ tests for package for cdk (#3077) * Add integ tests for package for cdk * Skip package integ test * Update CDK package integ tests * Black format and remove unsued imports * Remove unused imports * Fix unit tests in test_artifact_exporter * ci: Speed up unit test by caching the git clone (#3060) * ci: Speed up unit test by caching the git clone * Revert "Revert "temp: disable testing against python 3.8, and enabled 3.7"" (#3102) This reverts commit 1916bfa354b5d2612bd1bf9efd54a77e2bc66ff6. Revert "Revert "temp: disable testing against python 3.8, and enabled 3.7 (#3009)" (#3098)" (#3102) Co-authored-by: Tarun Mall <[email protected]> * Cdk support dev integ test deploy (#3111) * Refactor how stack_name is handled in guided flow * Add deploy integ tests for cdk support * Remove unused method * Update project default_stack to access internal member _stacks * Cdk support dev cdk python test utils (#3116) * Update package cdk test * Update cdk deploy tests * Run black reformat * More polishing in CdkPythonEnv * test: Integration test cases for CDK invoke (#3112) * Integration tests for CDK local invoke * Install requirement in setup * black reformat * fix test case on parameter overrides * test: Build Command CDK Integration Tests (#3108) * CDK Build integration tests * Add lambda handler for apigw test * Fix expected output string * Address pr comments * Update tests to include cdk download * Remove npm install * Update cdk pacakge test case * fix: fixing pipeline init integration test. (#3123) * fix: fixing pipeline init integration test so that it don't break every time we update our template. * black formatting. * cleaning up not needed file. Co-authored-by: Tarun Mall <[email protected]> * chore: upgrade pylint to 2.9.0 (#3119) * chore: fix pylint failures in python3.9 * chore: fix pylint failures in python3.9 * chore: bump pylint version to 2.9.0 * fix typo * Add disabling reasons on new rules * fix: integration test case related to recent fix on ruby (#3124) * fix: add dockerhub default login server, improve logs to check docker pull limitation (#3137) * fix: add sample payload for 'sam local generate-event stepfunctions error' (#3043) * add sample payload for 'sam local generate-event stepfunctions error' * add better default for error * fix: Error when trying to build CDK project with image assets. (#3136) * Fix image assets not building * Add unit tests for asset resolving * Remove unused import * chore: Use BUILD_TAG and JENKINS_URL to identify Jenkins env (#2805) * chore: Use BUILD_TAG instead of JENKINS_URL to identify Jenkins env * Keep JENKINS_URL * Request for Comments: Auto Create ECR Repos in Guided Deploy (#2675) * Added Auto Create ECR Design Doc * Updated Format * Addressed feedback * fix(bug): Pass boto_session to SAM Translator library (#2759) When validating the sam template, SAM CLI requires credentials to get the IAM Manged Policies. SAM Translator also requires the region in order to figure out the parition. Previously, SAM Translator assumed this to be on the Env but SAM CLI could get this information from a command line argument or a profile. This commit passes the boto_session into the SAM Translator lib (v1.35.0 or later), so that SAM Translator can figure out the partition from the information passed to SAM CLI. Co-authored-by: Jacob Fuss <[email protected]> Co-authored-by: Mathieu Grandis <[email protected]> * feat: add option --container-host and --container-host-interface to sam local commands (#2806) * chore: bump version to 1.23.0 (#2824) Co-authored-by: Xia Zhao <[email protected]> * refactor: Extract git-clone functionality out of InitTemplates class (#2821) * [Refactor] extract git-clone functionality out of InitTemplates class to its own class * apply review comments * typo * apply review comments * ignoring temp dirs used by dotnet (#2839) Co-authored-by: Slava Senchenko <[email protected]> * chore: Add GitHub actions to automate our issues workflow (#2521) * add github actions to automate our github issue workflow * reformat * update name format * update response message to be more precise * updated with the correct sam bot login name * updated with the correct token name * updated label name and bot name Co-authored-by: Mathieu Grandis <[email protected]> * Point numpy version to <1.20.3 (#2868) * Point numpy version to <1.19 to avoid PEP 317 failure * Update integ test python requirements which contain numpy * Fixing to numpy 1.20.2 * Revert "Fixing to numpy 1.20.2" This reverts commit a03f4d77e4b1588ecc3d0cbbe0f4c7c80ef60571. * Fixing numpy version to <1.20.3 * chore: Overhaul the development guide (#2827) * Validate default template.json (#2855) Issue: https://github.com/aws/aws-sam-cli/issues/2355 Added integration tests for `validate` command Co-authored-by: Slava Senchenko <[email protected]> Co-authored-by: _sam <[email protected]> * fix: package/deploy failure when Location/TemplateURL is virtual host S3 URL (#2785) * feat: Supports uncompression local layer zips in sam local (#2877) * refactor: refactor logs command library (#2862) * refactor logs command library * re-organize due to click usage * address comments * adding pylint disable for console consumer * make pylint happy with python 3.6 Co-authored-by: Mathieu Grandis <[email protected]> * sam init - Enable --app-template argument for Image package-type (#2864) * Enable --app-template argument for Image package-type while generating a new SAM project using 'sam init' * Fix the exception message * normalize pathes in UT to pass on windows * normalize project-template local path * fix: Ignore `.aws-sam` in sam build cache checksum (#2881) * feat: Allow dir_checksum() to accept a ignore_list * feat: Ignore .aws-sam when calculate cache md5 * fix: Fix crash when nested CFN stack has dict TemplateURL (unresolved intrinsics) (#2879) * fix: Fix crash when nested CFN stack has dict TemplateURL * Interactive flow question default answer from toml (#2850) * get questions' default answers from toml * make black happy * add more docs * rename question's attribute 'default_from_toml' to 'defaultFromToml' and rename 'valueof' to 'key' and add some docs * Add preload_value * Allow to pass toml file to interactive flow run() * Update related classes to utilize proload value context object * Update test * Add missing docstring * Remove samconfig change * Rename extra_context to context because it is required field now * Remove toml logics from this PR * Update comment Co-authored-by: Sam Liu <[email protected]> Co-authored-by: _sam <[email protected]> * Don't use layer.name in LayerBuildDefinition.__str__ (#2873) * Watchdog error (#2902) * chore: updating version of watchdog. Co-authored-by: Tarun Mall <[email protected]> * chore: Adds missing unit tests for LayerBuildDefinition in build_graph (#2883) * Adds missing unit tests for LayerBuildDefinition in build_graph * fix black formatting * fix: Build graph tests using assertTrue instead of assertEqual + added assertions (#2909) * samconfig debug level logging fixed; documentation updated (#2891) * samconfig debug level logging fixed; documentation updated * integration tests fix * help text improved Co-authored-by: Slava Senchenko <[email protected]> * chore: update to aws-sam-translator 1.36.0 (#2917) * Revert "samconfig debug level logging fixed; documentation updated (#2891)" (#2918) This reverts commit 2a13a69822660538c478118125eef50d0164995a. * chore: bump version to 1.24.0 (#2919) * fix: Windows default validate template integration test (#2924) * Enabled ability to provide tags as list in samconfig.toml file (#2912) * Enabled ability to provide tags as list in samconfig.toml file * Removed trailing white spaces and reformatted code * Added integration test for tags as list deploy command * Added integration test for tags as string from samconfig.toml Co-authored-by: Mohamed Elasmar <[email protected]> Co-authored-by: Wing Fung Lau <[email protected]> * fix: Add configparser to PyInstaller hiddenimports to resolve dependency issue from botocore (#2932) * Revert "Enabled ability to provide tags as list in samconfig.toml file (#2912)" (#2933) This reverts commit 104b5e5c528ef7e1ad0e83a5ba42316836a21e83. * chore: bump version to 1.24.1 (#2938) * chore: Update requests to 2.25.1 to remove the locking on urllib3 to 1.25 (#2929) * Updating tomlkit version as we need fix of the dataloss bug during copy() method use on Table object (#2939) * Updating tomlkit version as we need fix of the dataloss bug during copy() method use on Table object * Fixing types for tomlkit * Adding integration test for tomlkit not able to parse boolean issue. * Updating THIRD-PARTY-LICENSES file. * Parameterizing integ test filename Co-authored-by: Tarun Mall <[email protected]> Co-authored-by: _sam <[email protected]> * test: Fix the integration validate tests on Windows (#2940) * resolve pseudo region in build amd deploy comands (#2884) * resolve pseudo region from command argument or envvar if available * Revert "resolve pseudo region from command argument or envvar if available" This reverts commit abc0b2b62526f517dd633186861087fefb0f8b6e. * pass the aws-region to the BuildContext, DeployContext and Deploy command * Add integration tests * Make black happy * Temporary skip SAR build INTEGRATION TEST till we figure out the credeential issue * skip SAR tests when no credentials are available * Use the constant IntrinsicsSymbolTable.AWS_REGION instead of the string 'AWS::Region' * expand build SAR integration tests to four(all combinations of use-containr and us-east-2 region) * refactoring, merge stack_names and stack_names_with_regions together Co-authored-by: _sam <[email protected]> * fix: Catch more errors when building an image (#2857) * chore: fix canary/integration test issue (#2945) * feat: Allow tags as list input from samconfig.toml file (#2956) * Enabled ability to provide tags as list in samconfig.toml file * Removed trailing white spaces and reformatted code * Added integration test for tags as list deploy command * Added integration test for tags as string from samconfig.toml * Fixed Appveyer error by removing s3 info Co-authored-by: Mohamed Elasmar <[email protected]> Co-authored-by: Wing Fung Lau <[email protected]> * fix: Deploy integration tests for toml tags as a list (#2965) * chore: Increase awareness of same file warning during package (#2946) * chore: increase awareness of same file warning during package * fix formatting & grammar Co-authored-by: Mathieu Grandis <[email protected]> * fix: Allow the base64Encoded field in REST Api, skip validation of unknown fields and validate missing statusCode for Http Api (#2941) * fix API Gateway emulator: - skip validating the non allowed fields for Http Api Gateway, as it always skip the unknown fields - add base64Encoded as an allowed field for Rest Api gateway - base64 decoding will be always done for Http API gateway if the lambda response isBase64Encoded is true regardless the content-type - validate if statusCode is missing in case of Http API, and payload version 1.0 * - accept "true", "True", "false", "False" as valid isBase64Encoded values. - Validate on other isBase64Encoded Values - add more integration && unit test cases * fix lint && black issues * use smaller image to test Base64 response * fix: pass copy of environment variables for keeping cache valid (#2943) * fix: pass copy of environment variables for keeping cache valid * add integ tests * update docs * make black happy Co-authored-by: Qingchuan Ma <[email protected]> * fix: Skip build of Docker image if ImageUri is a valid ECR URL (#2934) (#2935) * Add condition to managed bucket policy (#2999) * chore: bump version to 1.25.0 (#3007) Co-authored-by: Sriram Madapusi Vasudevan <[email protected]> * temp: reduce python testing matrix (#3008) * temp: disable testing against python 3.8, and enabled 3.7 (#3009) * temp: disable testing against python 3.8, and enabled 3.7 * temp: disable testing against python 3.8, and enabled 3.7 & 3.6 * chore: update to aws-sam-translator 1.37.0 (#3019) * chore: bump version to 1.26.0 (#3020) * chore: Improved --resolve-s3 option documentation and deployment without s3 error messages (#2983) * Improve documentation on --resolve-s3 option and improve s3 failure messages * Changed indentation for integration test on s3 error message * Fixed a typo in description * Improve spacing on help text for resolve-s3 option * feat: Add SAM Pipeline commands (#3085) * sam pipeline bootstrap (#2811) * two-stages-pipeline plugin * typos * add docstring * make mypy happy * removing swap file * delete the two_stages_pipeline plugin as the pipeline-bootstrap command took over its responsibility * remove 'get_template_function_runtimes' function as the decision is made to not process the SAM template during pipeline init which was the only place we use the function * sam pipeline bootstrap command * move the pipelineconfig.toml file to .aws-sam * UX - rewriting Co-authored-by: Chris Rehn <[email protected]> * UX improvements * make black happy * apply review comments * UX - rewriting Co-authored-by: Chris Rehn <[email protected]> * refactor * Apply review comments * use python way of array elements assignments * Update samcli/lib/pipeline/bootstrap/stage.py Co-authored-by: _sam <[email protected]> * apply review comments * typo * read using utf-8 * create and user a safe version of the save_config method * apply review comments * rename _get_command_name to _get_command_names * don't save generated ARNs for now, will save during init * Revert "don't save generated ARNs for now, will save during init" This reverts commit d184e164022d9560131c62a826436edbc93da189. * Notify the user to rotate periodically rotate the IAM credentials * typo * Use AES instead of KMS for S3 SSE * rename Ecr to ECR and Iam to IAM * Grant lambda service explicit permissions to thhe ECR instead of relying on giving this permissions on ad-hoc while creating the container images Co-authored-by: Chris Rehn <[email protected]> Co-authored-by: _sam <[email protected]> * sam pipeline init command (#2831) * sam pipeline init command * apply review comments * apply review comments * display a message that we have successfully created the pipeline configuration file(s). * doc typo * Let 'sam pipeline init' prefills pipeline's infrastructure resources… (#2894) * Let 'sam pipeline init' prefills pipeline's infrastructure resources' values from 'sam pipeline bootstrap' results. * save bootstrapped sateg region * make black happy * exclude non-dict keys from samconfig.get_env_names method. * Rename the pipeline 'Stage' concept to 'Environment' (#2908) * Rename the pipeline 'Stage' concept to 'Environment' * typo * Rename --environment-name argument to --environment * Sam pipelines ux rename ecr repo to image repository (#2910) * Rename ecr-repo to image-repository * UT Fixes * typo * typo * feat: Support creating pipeline files directly into . without hooks (#2911) * feat: Support creating pipeline files directly into . without hooks * Integration test for pipeline init and pipeline bootstrap (#2841) * Expose Environment._get_stack_name for integ test to predict stack name * Add integ test for pipeline bootstrap * Add init integ test * small UX improvements: (#2914) * small UX improvements: 1. show a message when the user cancels a bootstrapping command. 2. Don't prompt for CI/CD provider or provider templates if there is only one choice. 3. Make PipelineFileAlreadyExistsError a UserError. 4. use the Colored class instead of fg='color' when prompting a colored message. 5. Fix a bug where we were not allowing empty response for not required questions. * Fix Integration Test: We now don't ask the user to select a provider's pipeline template if there is only one * Add docs for PipelineFileAlreadyExistsError * make black happy * Sam pipelines s3 security (#2975) * Deny non https requests for the artifacts S3 bucket * enable bucket serverside logging * add integration tests for artifacts bucket SSL-only requests and access logging * typo * Ensure the ArtifactsLoggingBucket denies non ssl requests (#2976) * Sam pipelines ux round 3 (#2979) * rename customer facing message 'CI/CD provider' to 'CI/CD system' * add a note about what 'Environment Name' is during the pipeline bootstrap guided context * Apply suggestions from code review typo Co-authored-by: Chris Rehn <[email protected]> Co-authored-by: Chris Rehn <[email protected]> * let pipeline IAM user assume only IAM roles tagged with Role=pipeline-execution-role (#2982) * Adding AWS_ prefix to displayed out. (#2993) Co-authored-by: Tarun Mall <[email protected]> * Add region to pipeline bootstrap interactive flow (#2997) * Ask AWS region in bootstrap interactive flow * Read default region from boto session first * Fix a unit test * Inform write to pipelineconfig.toml at the end of bootstrap (#3002) * Print info about pipelineconfig.toml after resources are bootstrapped * Update samcli/commands/pipeline/bootstrap/cli.py Co-authored-by: Chris Rehn <[email protected]> Co-authored-by: Chris Rehn <[email protected]> * List detected env names in pipeline init when prompt to input the env name (#3000) * Allow question.question can be resolved using key path * Pass the list of env names message (environment_names_message) into pipeline init interactive flow context * Update samcli/commands/pipeline/init/interactive_init_flow.py Co-authored-by: Chris Rehn <[email protected]> * Fix unit test (trigger pr builds) * Fix integ test * Fix integ test Co-authored-by: Chris Rehn <[email protected]> * Adding account id to bootstrap message. (#2998) * Adding account id to bootstrap message. * adding docstring * Addressing PR comments. * Adding unit tests. * Fixing unit tests. Co-authored-by: Tarun Mall <[email protected]> * Cfn creds fix (#3014) * Removing pipeline user creds from cfn output. This maintains same user exp. Co-authored-by: Tarun Mall <[email protected]> * Ux bootstrap revamp 20210706 (#3021) * Add intro paragraph to bootstrap * Add switch account prompt * Revamp stage definition prompt * Revamp existing resources prompt * Revamp security prompt * Allow answers to be changed later * Add exit message for bootstrap * Add exit message for bootstrap (1) * Add indentation to review values * Add "Below is the summary of the answers:" * Sweep pylint errors * Update unit tests * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <[email protected]> * Update samcli/commands/pipeline/bootstrap/cli.py Co-authored-by: Chris Rehn <[email protected]> * Update unit tests * Add bold to other literals Co-authored-by: Chris Rehn <[email protected]> * Adding account condition for CFN execution role. (#3027) Co-authored-by: Tarun Mall <[email protected]> * pipeline UX revamp 20210707 (#3031) * Allow running bootstrap inside pipeline init * Select account credential source within bootstrap * Add bootstrap decorations within pipeline init * Removing ip range option from bootstrap. (#3036) * Removing ip range option from bootstrap. * Fixing unit test from UX PR. Co-authored-by: Tarun Mall <[email protected]> * Fix toml file incorrect read/write in init --bootstrap (#3037) * Temporarily removing account fix. (#3038) Co-authored-by: Tarun Mall <[email protected]> * Rename environment to stage (#3040) * Improve account source selection (#3042) * Fixing various cosmetics UX issues with pipeline workflow. (#3046) * Fixing credential to credentials * Forcing text color to yellow. * Adding new line after stage diagram. * Adding extra line after checking bootstrap message. * Renaming config -> configuration * account source -> credential source * Removing old message. * Fixing indentation in list. * Fixing bunch of indentation. * fixing f string Co-authored-by: Tarun Mall <[email protected]> * Auto skip questions if stage detected (#3045) * Autofill question if default value is presented * Allow to use index to select stage names (#3051) * Updating message when bootstrap stages are missing. (#3058) * Updating message when bootstrap stages are missing. * Fixing indendation Co-authored-by: Tarun Mall <[email protected]> * Fixing bootstrap integ tests. (#3061) * Fixing bootstrap integ tests. * Cleaning up some integ tests. * Using environment variables when running integ test on CI. * Using expression instead of full loop. * Adding instruction to use default profile on local. Co-authored-by: Tarun Mall <[email protected]> * Fix bootstrap test region (#3064) * Fix bootstrap region in integ test * Fix regions in non-interactive mode as well * Add more pipeline init integ test (#3065) * Fix existing pipeline init integ test * Add more pipeline init integ tests * Config file bug (#3066) * Validating config file after bootstrap stack creation. * Validating config file after bootstrap. Co-authored-by: Tarun Mall <[email protected]> * Fix pipeline init integ test because of pipelineconfig file exists (#3067) * Make stage name randomized to avoid race condition among multi canary runs (#3078) * Load number of stages from pipeline template (#3059) * Load number of stages from templates * Rename variable and add debug log * Add encoding to open() * Allow roles with Tag aws-sam-pipeline-codebuild-service-role to assume PipelineExecutionRole (#2950) * pipeline init UX: Ask to confirm when file exists (#3079) * Ask to confirm overriding if files already exist, or save to another directory * Add doc links (#3087) * Adding accidentally removed tests back. (#3088) Co-authored-by: Tarun Mall <[email protected]> Co-authored-by: elbayaaa <[email protected]> Co-authored-by: Chris Rehn <[email protected]> Co-authored-by: Ahmed Elbayaa <[email protected]> Co-authored-by: Tarun <[email protected]> Co-authored-by: Tarun Mall <[email protected]> * chore: bump aws-lambda-builder version to 1.5.0 (#3086) * chore: update to aws-sam-translator 1.38.0 (#3073) * ci: Update expected Jenkins file in pipeline integ test (#3090) * chore: Refine pipeline help text and update unit test (#3091) * Update --bucket help text * Update --stage help text * Update help text * Update help text * Update help text * Update help text * Update help text * Update jenkins generated files * Update some intro texts * Remove trialing spaces * Clearing pipeline integ test buckets with versioned objects. (#3094) * Clearing pipeline integ test buckets with versioned objects. * Fixing black formatting. Co-authored-by: Tarun Mall <[email protected]> * Fixing bug in bucket cleanup. (#3096) Co-authored-by: Tarun Mall <[email protected]> * Deleting bucket (#3097) Co-authored-by: Tarun Mall <[email protected]> * chore: bump SAM CLI version to 1.27.0 (#3101) * Add pipeline to pyinstaller (#3103) * Adding pipeline to pyinstaller. Co-authored-by: Tarun Mall <[email protected]> * Including stage resource yaml in pip. (#3106) * Including stage resource yaml in pip. * Bumping patch version Co-authored-by: Tarun Mall <[email protected]> * ci: Speed up unit test by caching the git clone (#3060) * ci: Speed up unit test by caching the git clone * Revert "Revert "temp: disable testing against python 3.8, and enabled 3.7"" (#3102) This reverts commit 1916bfa354b5d2612bd1bf9efd54a77e2bc66ff6. Revert "Revert "temp: disable testing against python 3.8, and enabled 3.7 (#3009)" (#3098)" (#3102) Co-authored-by: Tarun Mall <[email protected]> * fix: fixing pipeline init integration test. (#3123) * fix: fixing pipeline init integration test so that it don't break every time we update our template. * black formatting. * cleaning up not needed file. Co-authored-by: Tarun Mall <[email protected]> * chore: upgrade pylint to 2.9.0 (#3119) * chore: fix pylint failures in python3.9 * chore: fix pylint failures in python3.9 * chore: bump pylint version to 2.9.0 * fix typo * Add disabling reasons on new rules * fix: integration test case related to recent fix on ruby (#3124) * fix: add dockerhub default login server, improve logs to check docker pull limitation (#3137) * fix: add sample payload for 'sam local generate-event stepfunctions error' (#3043) * add sample payload for 'sam local generate-event stepfunctions error' * add better default for error * fix conflicts * chore: removed unused code which was using pre-defined managed policy… (#3030) * chore: removed unused code which was using pre-defined managed policy list and used in a sam translator wrapper, but the code path is not used. * make black * feat(public-ecr): Download Emulation images (#3152) Co-authored-by: Jacob Fuss <[email protected]> Co-authored-by: Mathieu Grandis <[email protected]> * resolve PR comments * fix(integ): Use images that are in public ecr (#3162) Co-authored-by: Jacob Fuss <[email protected]> * Add ECR credentials for windows test (#3160) * Add ECR credentials for windows test * Remove the dockerhub env vars * fix(integ): Fix Invalid image tag errors (#3163) Co-authored-by: Jacob Fuss <[email protected]> * Install aws cli in the windows test jobs (#3164) * Add ECR credentials for windows test * Remove the dockerhub env vars * install aws cli in the windows test jobs * fix(integ): Add missing image to have deploy integ tests work (#3165) Co-authored-by: Jacob Fuss <[email protected]> * chore: Update dateparser to 1.0, update TestBuildCommand_PythonFunctions_Images test requirement (#3172) * chore: Update dateparser to 1.0 * Move public ECR limited test cases to Canary tests * Python39 support for samcli (#3173) * Python39 support for samcli (#354) * Python39 support for samcli * Updated reproducible-linux.txt and lambda_build_container.py for test purpose * Revert files after testing * updated integ test * updated appveyor * updated to appveyor * Update python3.9 appveyor config * update windows python3.9 executable path * update appveyor * fix lint and windows python appveyor script * bump version of lambda-builder to 1.6.0 Co-authored-by: jonife <[email protected]> * chore: bump SAM CLI version to 1.28.0 (#3174) * Sam delete develop merged (#3176) * feat: Delete methods for CF stacks and S3 files (#2981) * Added methods for cf and s3 files and init UI * Added unit tests for utils methods and s3_uploader * Removed s3_bucket and s3_prefix click options * Fixed lint errors and added few unit tests * Make black happy * Added LOG statements * Added and updated changes based on CR * Fixed the unit tests in artifact_exporter.py * Update HELP_TEXT in delete/command.py Co-authored-by: Chris Rehn <[email protected]> * Updated code based on Chris' comments * Small changes and fixes based on the comments * Removed region prompt * Update SAM context values for profile and region in delete_context.py * Added typing for get_cf_template_name method * Added stack_name prompt if the stack_name is not present in samconfig file * Replace [] with get() for stack-name in delete_context.py Co-authored-by: Chris Rehn <[email protected]> * Delete template artifacts (#3022) * Added methods for cf and s3 files and init UI * Added unit tests for utils methods and s3_uploader * Removed s3_bucket and s3_prefix click options * chore: Increase awareness of same file warning during package (#2946) * chore: increase awareness of same file warning during package * fix formatting & grammar Co-authored-by: Mathieu Grandis <[email protected]> * fix: Allow the base64Encoded field in REST Api, skip validation of unknown fields and validate missing statusCode for Http Api (#2941) * fix API Gateway emulator: - skip validating the non allowed fields for Http Api Gateway, as it always skip the unknown fields - add base64Encoded as an allowed field for Rest Api gateway - base64 decoding will be always done for Http API gateway if the lambda response isBase64Encoded is true regardless the content-type - validate if statusCode is missing in case of Http API, and payload version 1.0 * - accept "true", "True", "false", "False" as valid isBase64Encoded values. - Validate on other isBase64Encoded Values - add more integration && unit test cases * fix lint && black issues * use smaller image to test Base64 response * Fixed lint errors and added few unit tests * Make black happy * Added methods for deleting template artifacts * Wait method added for delete cf api * Added LOG statements * Added and updated changes based on CR * Fixed the unit tests in artifact_exporter.py * Update HELP_TEXT in delete/command.py Co-authored-by: Chris Rehn <[email protected]> * Updated code based on Chris' comments * Added condition for resources that have deletionpolicy specified * Small changes and fixes based on the comments * Removed region prompt * Added unit tests for ecr delete method and typing for methods * Reformatted delete_context and added option to skip user prompts * Removed return type from artifact_exporter for delete method * Added unit tests for artifact_exporter and delete_context * Added more unit tests for delete_context and artifact_exporter * Added more unit tests for delete_context and artifact_exporter * Added docs and comments for artifact_exporter and ecr_uploader * Added log statements in delete_context and some updates in unit tests * Changed force to no-prompts and updated ecr delete method error handling * Created a separate function for parsing ecr url in ecr_uploader * Reformatted Template class init to pass template_str and init template_dict * Changed how s3 url is obtained for resource_zip edge-case: aws:glue:job * Fixed edge case where resource artifact points to a path style url * run Make black * Made the parse s3 url funcs protected and defined a parent method and modified delete method for ResourceImageDict * Changed parse_ecr_url function name to parse_image_url Co-authored-by: Mehmet Nuri Deveci <[email protected]> Co-authored-by: Mathieu Grandis <[email protected]> Co-authored-by: Mohamed Elasmar <[email protected]> Co-authored-by: Chris Rehn <[email protected]> * Get s3 info cf template (#3050) * Added methods for cf and s3 files and init UI * Added unit tests for utils methods and s3_uploader * Removed s3_bucket and s3_prefix click options * chore: Increase awareness of same file warning during package (#2946) * chore: increase awareness of same file warning during package * fix formatting & grammar Co-authored-by: Mathieu Grandis <[email protected]> * fix: Allow th…

ssenchenko and others added 2 commits May 19, 2021 15:53

samconfig debug level logging fixed; documentation updated

bce682d

Merge branch 'develop' into 2747-samconfig

5e534fc

ssenchenko commented May 19, 2021

View reviewed changes

ssenchenko requested review from sriram-mv and aahung May 19, 2021 23:23

aahung requested a review from wchengru May 19, 2021 23:26

wchengru reviewed May 19, 2021

View reviewed changes

ssenchenko force-pushed the 2747-samconfig branch 3 times, most recently from 873adf5 to 90876e4 Compare May 20, 2021 01:08

integration tests fix

8486ab9

ssenchenko force-pushed the 2747-samconfig branch 2 times, most recently from 9a90cd8 to bfbba82 Compare May 21, 2021 17:08

Merge branch 'develop' into 2747-samconfig

2e72dcc

ssenchenko force-pushed the 2747-samconfig branch from bfbba82 to 2e72dcc Compare May 25, 2021 23:10

Merge branch 'develop' into 2747-samconfig

b00c06f

ssenchenko force-pushed the 2747-samconfig branch from 2bde3ab to b00c06f Compare May 25, 2021 23:57

Merge branch 'develop' into 2747-samconfig

bae2879

sriram-mv requested changes May 26, 2021

View reviewed changes

help text improved

2251c7d

aahung approved these changes May 26, 2021

View reviewed changes

mgrandis added the pr/internal label May 26, 2021

link change

e888c84

ssenchenko requested a review from sriram-mv May 26, 2021 22:54

sriram-mv approved these changes May 27, 2021

View reviewed changes

aahung approved these changes May 28, 2021

View reviewed changes

help messages from 3rd person

701cc97

Merge branch 'develop' into 2747-samconfig

6ce5e61

ssenchenko merged commit 2a13a69 into aws:develop May 28, 2021

ssenchenko deleted the 2747-samconfig branch May 28, 2021 17:58

mgrandis added a commit to mgrandis/aws-sam-cli that referenced this pull request Jun 1, 2021

Revert "samconfig debug level logging fixed; documentation updated (a…

7a3608c

…ws#2891)" This reverts commit 2a13a69.

qingchm pushed a commit that referenced this pull request Jun 1, 2021

Revert "samconfig debug level logging fixed; documentation updated (#…

5fecac5

…2891)" (#2918) This reverts commit 2a13a69.

moelasmar pushed a commit to moelasmar/aws-sam-cli that referenced this pull request Jul 1, 2021

Revert "samconfig debug level logging fixed; documentation updated (a…

b2d9d31

…ws#2891)" (aws#2918) This reverts commit 2a13a69.

moelasmar pushed a commit to moelasmar/aws-sam-cli that referenced this pull request Aug 5, 2021

Revert "samconfig debug level logging fixed; documentation updated (a…

0510186

…ws#2891)" (aws#2918) This reverts commit 2a13a69.

moelasmar pushed a commit to moelasmar/aws-sam-cli that referenced this pull request Aug 6, 2021

Revert "samconfig debug level logging fixed; documentation updated (a…

6beca00

…ws#2891)" (aws#2918) This reverts commit 2a13a69.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

samconfig debug level logging fixed; documentation updated #2891

samconfig debug level logging fixed; documentation updated #2891

ssenchenko commented May 19, 2021 •

edited

Loading

ssenchenko May 19, 2021

wchengru May 20, 2021

ssenchenko May 20, 2021

ssenchenko May 19, 2021

wchengru May 19, 2021

ssenchenko May 20, 2021

wchengru May 19, 2021

ssenchenko May 19, 2021

ssenchenko May 20, 2021

wchengru May 19, 2021

ssenchenko May 19, 2021

sriram-mv commented May 26, 2021

sriram-mv left a comment

sriram-mv May 26, 2021

ssenchenko May 26, 2021

sriram-mv May 26, 2021

ssenchenko May 26, 2021

sriram-mv May 26, 2021

ssenchenko May 26, 2021

sriram-mv May 26, 2021

ssenchenko May 26, 2021

aahung left a comment

aahung May 26, 2021

ssenchenko May 26, 2021

aahung May 26, 2021

ssenchenko May 26, 2021

sriram-mv May 27, 2021

ssenchenko May 28, 2021

aahung May 28, 2021

aahung May 28, 2021

	https://github.com/awslabs/serverless-application-model.
	https://github.com/aws/serverless-application-model.

		if process_stdout.startswith("Config file"):
		*_, process_stdout = process_stdout.partition("\n")

	"for build, local invocation etc. If template file is not specified explicitly SAM CLI expects it to be "
	"for build, local invocation, etc. If template file is not specified explicitly SAM CLI expects it to be "

	"Alternatively, if --config-file is explicitly specified, it can point to a custom samconfig.toml location."
	"Alternatively, if --config-file is specified, it can point to a custom samconfig.toml location."

samconfig debug level logging fixed; documentation updated #2891

samconfig debug level logging fixed; documentation updated #2891

Conversation

ssenchenko commented May 19, 2021 • edited Loading

Which issue(s) does this change fix?

Why is this change necessary?

How does it address the issue?

What side effects does this change have?

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sriram-mv commented May 26, 2021

sriram-mv left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aahung left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ssenchenko commented May 19, 2021 •

edited

Loading