Nested Application from the Local File System build does not work properly #1470

lucasam · 2019-10-17T15:11:53Z

Nested Application from the Local File System build does not work properly

Issue description

According the documentation
https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-template-nested-applications.html

The AWS::Serverless::Application can point to another template file (check Defining a Nested Application from the Local File System)

if you trigger the commands sam build; sam package, sam deploy the result of the nested application is a broken zip file.

Scenario

/ - javaproject
  | - template.yaml
  | - pom.xml
  | - src/main/java...
/ - template.yaml  (parent)

Parent template references javaproject/template.yaml as AWS::Serverless::Application`

Resources:
  AvatarApplication:
    Type: AWS::Serverless::Application
    Properties:
      Location: javaproject/template.yaml

The sequence sam build; sam package; sam deploy when triggered from parent directory completes. But the zipfile for the function under the nested application is wrongly built.

Let's check step by step using the sample code https://github.com/lucasam/sambuildissue2

Cloning sample code

git clone https://github.com/lucasam/sambuildissue2
cd sambuildissue2

sam build

lucasam$ .sam build --debug 
Using SAM Template at /Users/lucasam/Documents/src/sambuildissue2/template.yaml
Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
'build' command is called
No Parameters detected in the template
1 resources found in the template

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Package: sam package --s3-bucket <yourbucket>
    
Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 21, 'exitReason': 'success', 'exitCode': 0, 'requestId': '7178268e-8384-4ad2-a044-cfd938769ca7', 'installationId': '893c5aa8-d107-4b3b-8f82-e7cb5c9ce2fe', 'sessionId': '907647c9-698e-428a-a1ca-d86ca056821e', 'executionEnvironment': 'CLI', 'pyversion': '3.7.4', 'samcliVersion': '0.22.0'}}]}
HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)

sam package

lucasam$ sam package --output-template-file packaged.yaml  --s3-bucket $BUCKET 
Uploading to 41c665fe10f20b0417eb7f684d7a8072.template  791 / 791.0  (100.00%)
Successfully packaged artifacts and wrote output template to file packaged.yaml.
Execute the following command to deploy the packaged template
aws cloudformation deploy --template-file /Users/lucasam/Documents/src/sambuildissue2/packaged.yaml --stack-name <YOUR STACK NAME>

sam deploy

lucasam$ sam deploy  --template-file packaged.yaml  --stack-name sam-bug  --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND \
    --region us-east-1
Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - sam-bug

Everything looks fine so far. BUT

If you inspect the nested application you will find

f01898820a1c:x2 lucasam$ aws s3 cp s3://<BUCKET>/261b18ebec06a0f13ad2709228e60cca . 
download: s3://<BUCKET>/261b18ebec06a0f13ad2709228e60cca to ./261b18ebec06a0f13ad2709228e60cca
f01898820a1c:x2 lucasam$ unzip 261b18ebec06a0f13ad2709228e60cca
Archive:  261b18ebec06a0f13ad2709228e60cca
  inflating: template.yaml           
  inflating: sam-issue2.iml          
  inflating: pom.xml                 
  inflating: Readme.md               
  inflating: License.txt             
  inflating: src/main/java/com/test/FunctionOne.java

Issue

Observe that the zip file representing the nested application is just a zip of the Nested application folder.
Expected. SAM should interpret template.yaml of the nested application and build it properly.

Workaround

Build use sam build and sam package at the nested application. Then point at parent template the packaged.yaml instead of template.yaml

The text was updated successfully, but these errors were encountered:

jfuss · 2019-10-17T15:20:14Z

@lucasam Thanks for the issue. We currently do not support doing nested stack building/packaging/publishing of templates. This is very similar to #1443.

Marking as a feature request.

lucasam · 2019-10-17T15:46:39Z

@jfuss How can we update the documentation? Because there is a particular session that states exactly this scenario. And the cli completes successfully. It even generates a zip file with the 'unbuild' app.
If we claim this is not supported i have 2 suggestions.

Update the documentation instructing users to point to an 'already packaged' sam application -
Fail build when the parent application is pointing to a sam file that still needs transformation. The fact that sam cli take action and zip the folder is extremely misleading

jfuss · 2019-10-17T15:56:32Z

@lucasam Which part(s) of the docs leads you to this? Skimming through, I can't find where we have claimed this to work. You can also suggest updates by submitting the changes to https://github.com/awsdocs/aws-sam-developer-guide

marcschroeter · 2019-11-19T14:40:13Z

we really need support for nested stacks :-( Is there an idea if that will come?

henkesde · 2019-11-19T17:24:35Z

Hey guys,

@jfuss Do you already have some news about this request?
I think this is must have feature since I don't see any advantage of sam over cloudformation without this being supported.

UPDATE: I also created my own small shell script that just executes sam build --build-dir <BUILD_DIR> --template <TEMPLATE_PATH> before executing sam build on the top level of the application. This workaround is okay for me.

jeswanthamazon · 2020-01-20T19:11:25Z

+1 for this feature

inception-cto · 2020-02-10T02:20:04Z

Hi,

I have followed the workaround above for a python nested application. However, it is still does not work.

I have the following folder structure

.
├── MAKEFILE
├── README.md
├── __init__.py
├── apps
│   ├── __init__.py
│   ├── account
│   │   ├── __init__.py
│   │   ├── endpoints.py
│   │   ├── models.py
│   │   ├── requirements.txt
│   │   └── template.yaml
├── samconfig.toml
└── template.yaml

I followed the instructions above to build the nested app apps/account using sam build followed by sam package. The results was then a new packaged.yaml and the .aws-sam/ folder with the following structure

apps/account/.aws-sam/
└── build
    ├── GetAccountInfo
    │   ├── __init__.py
    │   ├── blinker
    │   ├── blinker-1.4.dist-info
    │   ├── boto3
    │   ├── boto3-1.11.13.dist-info
    │   ├── botocore
    │   ├── botocore-1.14.13.dist-info
    │   ├── dateutil
    │   ├── docutils
    │   ├── docutils-0.15.2.data
    │   ├── docutils-0.15.2.dist-info
    │   ├── dynamorm
    │   ├── dynamorm-0.10.0.dist-info
    │   ├── endpoints.py
    │   ├── jmespath
    │   ├── jmespath-0.9.4.data
    │   ├── jmespath-0.9.4.dist-info
    │   ├── marshmallow
    │   ├── marshmallow-3.4.0.dist-info
    │   ├── models.py
    │   ├── python_dateutil-2.8.1.dist-info
    │   ├── requirements.txt
    │   ├── s3transfer
    │   ├── s3transfer-0.3.3.dist-info
    │   ├── schema.yaml
    │   ├── six-1.14.0.dist-info
    │   ├── six.py
    │   ├── template.yaml
    │   ├── urllib3
    │   └── urllib3-1.25.8.dist-info
    ├── UpdateAccountInfo
    │   ├── __init__.py
    │   ├── blinker
    │   ├── blinker-1.4.dist-info
    │   ├── boto3
    │   ├── boto3-1.11.13.dist-info
    │   ├── botocore
    │   ├── botocore-1.14.13.dist-info
    │   ├── dateutil
    │   ├── docutils
    │   ├── docutils-0.15.2.data
    │   ├── docutils-0.15.2.dist-info
    │   ├── dynamorm
    │   ├── dynamorm-0.10.0.dist-info
    │   ├── endpoints.py
    │   ├── jmespath
    │   ├── jmespath-0.9.4.data
    │   ├── jmespath-0.9.4.dist-info
    │   ├── marshmallow
    │   ├── marshmallow-3.4.0.dist-info
    │   ├── models.py
    │   ├── python_dateutil-2.8.1.dist-info
    │   ├── requirements.txt
    │   ├── s3transfer
    │   ├── s3transfer-0.3.3.dist-info
    │   ├── schema.yaml
    │   ├── six-1.14.0.dist-info
    │   ├── six.py
    │   ├── template.yaml
    │   ├── urllib3
    │   └── urllib3-1.25.8.dist-info
    └── template.yaml

However, I get the error Unable to import module 'endpoints': No module named 'dynamorm'.

Am I missing something?

Thanks for the help!

ngander-amfam · 2020-02-26T21:24:01Z

+1 on fixing this issue.

jessedoyle · 2020-04-01T23:11:47Z

+1 - Nested applications are a common pattern!

codergeorgej · 2020-05-29T03:52:27Z

+1

eallais · 2020-06-05T13:20:58Z

+1

codergeorgej · 2020-06-08T00:51:17Z

@jfuss #1470 (comment),
any timeline on when we can get this feature?

divmgl · 2020-06-26T06:21:37Z

Without this feature I'm not sure how I'm going to prevent my template.yaml file from becoming 10,000+ lines without resorting to some kind of YAML or build tool.

liszi · 2020-07-16T10:27:31Z

It's a must have.
We need to use a bunch of nasty workarounds to fix this problem.
+1 to implement this feature.

winedrinker · 2020-07-16T11:23:16Z

+1

asifpy · 2020-07-29T21:55:54Z

+1

klembee · 2020-08-26T18:31:28Z

+1

dtelaroli · 2020-08-31T17:50:20Z

+1

mdkotafutami · 2020-09-03T03:00:00Z

+1

wdalmut · 2020-10-31T11:48:35Z

+1

klembee · 2020-10-31T13:22:55Z

To make nested applications, I first do sam build, sam package --s3-bucket mybucket --output-template-file packaged.yaml in the nested application folder. Then in the main template, I add:

`

NestedService:
    Type: AWS::CloudFormation::Stack
    DependsOn: MyAppSyncSchema
    Properties:
        TemplateURL: [path to packaged.yaml]
    Parameters: 
        [the parameters to pass]

`

under Resources.

aahung · 2021-03-03T19:09:02Z

Building resources in nested stack is enabled by default in #2662, to be released

aahung · 2021-03-04T19:37:51Z

released in v1.20.0

lvlattanzio · 2021-03-05T19:01:47Z

Nested stacks seemed to work fine for me before I upgraded my CLI to 1.20.0. Now I'm getting the following errors and there are no search results on the internet that I can find:

Nested application 'XXXXXXXXXXXXXXXX' has specified S3 location for Location which is unsupported. Skipping resources inside this nested application.

I am defining the nested application as follows:

  XXXXXXXXXXXXXXXX:
    Type: AWS::Serverless::Application
    Properties:
      Location: ./template-XXXXXXXXXXXXXXXX.yaml

Again, this was working before I updated my CLI. The nested stack contains a few standard resources and two AWS::Serverless::Function resources.

aahung · 2021-03-05T19:12:02Z

Nested stacks seemed to work fine for me before I upgraded my CLI to 1.20.0. Now I'm getting the following errors and there are no search results on the internet that I can find:

Nested application 'XXXXXXXXXXXXXXXX' has specified S3 location for Location which is unsupported. Skipping resources inside this nested application.

I am defining the nested application as follows:
  XXXXXXXXXXXXXXXX:
    Type: AWS::Serverless::Application
    Properties:
      Location: ./template-XXXXXXXXXXXXXXXX.yaml
Again, this was working before I updated my CLI. The nested stack contains a few standard resources and two AWS::Serverless::Function resources.

Hi @lvlattanzio, looks like this is generated from sam build, in previous sam-cli version building resources in nested stack is not supported, can you post the command you run and what's your workflow to build/deploy nested stacks? Since 1.20, you don't need to build them separately

lvlattanzio · 2021-03-05T21:14:01Z

@aahung I just run sam build && sam deploy, which uses the root template.yaml. This stack references the nested stack templates as shown above. Since this issue is closed I opened a separate issue:
#2689

My stacks updated and deployed appropriately so I think this error message might just be noise but I could be wrong.

aahung · 2021-03-05T21:15:22Z

@aahung I just run sam build && sam deploy, which uses the root template.yaml. This stack references the nested stack templates as shown above. Since this issue is closed I opened a separate issue:
#2689

My stacks updated and deployed appropriately so I think this error message might just be noise but I could be wrong.

thx, I will reply in the new issue

jfuss added the type/feature Feature request label Oct 17, 2019

daroot mentioned this issue Dec 30, 2019

Nested stacks are difficult to build and deploy for several reasons #1516

Closed

aahung added stage/waiting-for-release Fix has been merged to develop and is waiting for a release area/build sam build command area/nested-stacks labels Mar 3, 2021

aahung closed this as completed Mar 4, 2021

aahung removed the stage/waiting-for-release Fix has been merged to develop and is waiting for a release label Mar 4, 2021

cwilliamsonslyp mentioned this issue Aug 23, 2021

Nested stack makefiles are run but not deployed as expected #3196

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nested Application from the Local File System build does not work properly #1470

Nested Application from the Local File System build does not work properly #1470

lucasam commented Oct 17, 2019

jfuss commented Oct 17, 2019

lucasam commented Oct 17, 2019

jfuss commented Oct 17, 2019

marcschroeter commented Nov 19, 2019

henkesde commented Nov 19, 2019 •

edited

Loading

jeswanthamazon commented Jan 20, 2020

inception-cto commented Feb 10, 2020

ngander-amfam commented Feb 26, 2020

jessedoyle commented Apr 1, 2020

codergeorgej commented May 29, 2020

eallais commented Jun 5, 2020

codergeorgej commented Jun 8, 2020

divmgl commented Jun 26, 2020

liszi commented Jul 16, 2020

winedrinker commented Jul 16, 2020

asifpy commented Jul 29, 2020

klembee commented Aug 26, 2020

dtelaroli commented Aug 31, 2020

mdkotafutami commented Sep 3, 2020

wdalmut commented Oct 31, 2020

klembee commented Oct 31, 2020 •

edited

Loading

aahung commented Mar 3, 2021

aahung commented Mar 4, 2021

lvlattanzio commented Mar 5, 2021 •

edited

Loading

aahung commented Mar 5, 2021

lvlattanzio commented Mar 5, 2021

aahung commented Mar 5, 2021

Nested Application from the Local File System build does not work properly #1470

Nested Application from the Local File System build does not work properly #1470

Comments

lucasam commented Oct 17, 2019

Nested Application from the Local File System build does not work properly

Issue description

Scenario

Issue

Workaround

jfuss commented Oct 17, 2019

lucasam commented Oct 17, 2019

jfuss commented Oct 17, 2019

marcschroeter commented Nov 19, 2019

henkesde commented Nov 19, 2019 • edited Loading

jeswanthamazon commented Jan 20, 2020

inception-cto commented Feb 10, 2020

ngander-amfam commented Feb 26, 2020

jessedoyle commented Apr 1, 2020

codergeorgej commented May 29, 2020

eallais commented Jun 5, 2020

codergeorgej commented Jun 8, 2020

divmgl commented Jun 26, 2020

liszi commented Jul 16, 2020

winedrinker commented Jul 16, 2020

asifpy commented Jul 29, 2020

klembee commented Aug 26, 2020

dtelaroli commented Aug 31, 2020

mdkotafutami commented Sep 3, 2020

wdalmut commented Oct 31, 2020

klembee commented Oct 31, 2020 • edited Loading

aahung commented Mar 3, 2021

aahung commented Mar 4, 2021

lvlattanzio commented Mar 5, 2021 • edited Loading

aahung commented Mar 5, 2021

lvlattanzio commented Mar 5, 2021

aahung commented Mar 5, 2021

henkesde commented Nov 19, 2019 •

edited

Loading

klembee commented Oct 31, 2020 •

edited

Loading

lvlattanzio commented Mar 5, 2021 •

edited

Loading