Overlapping generateFiles[].targetPath in different modules do not raise an error and overwrite each other

[BillRamsey-minted]

Bug

generateFiles in a shared ModuleTemplate generates the files in the calling repository instead of the build directory and also if called twice, it over-writes each other.

Current Behavior

In my project, I include a source for the module template:

sources:
  - name: templates
    repositoryUrl: https://github.com/BillRamsey-minted/garden-template-example#main
    
---
kind: Module
type: container
name: bug-module
description: Container Module

---
kind: Module
type: templated
template: my-template
name: service-1
inputs:
  replicas: 1
  image: ${modules.bug-module.outputs.deployment-image-id}
  service-name: s1

---
kind: Module
type: templated
template: my-template
name: service-2
inputs:
  replicas: 1
  image: ${modules.bug-module.outputs.deployment-image-id}
  service-name: s2

The ModuleTemplate has a generateFiles:

    files: [".manifests/.manifests-template.yml"]
    generateFiles:
      - sourcePath: manifests-template.yml
        targetPath: ".manifests/.manifests-template.yml"

When I do a garden build that file shows up in the calling project in three places:

.manifests/.manifests-template.yml
.garden/build/s1/.manifests/.manifests-template.yml
.garden/build/s2/.manifests/.manifests-template.yml

However, both the build manifests in s1/ and s2/ show the same service name of s2.

Expected behavior

I would expect it not to create the file at the top level of the calling repo at all,
and the build/s1 and build/s2 to each be different manifests instead of both using the same service name.

Reproducible example

project.garden.txt

Workaround

I include the service name in the manifests:

    files: ['.manifests/.${inputs.service-name}-secrets.yml']
    generateFiles:
      - sourcePath: 'manifest-templates/secrets.yml'
        targetPath: '.manifests/.${inputs.service-name}-secrets.yml'

Suggested solution(s)

unknown

Your environment

EKS and docker desktop

garden version
0.12.63

[vvagaytsev]

@BillRamsey-minted thanks for reporting this!

It looks like the current behavior is formally correct, despite it looking confusing.
Let's take a closer look at the ModuleTemplate config:

    files: [".manifests/.manifests-template.yml"]
    generateFiles:
      - sourcePath: manifests-template.yml
        targetPath: ".manifests/.manifests-template.yml"

In the generateFiles section, we specify 2 items:

• sourcePath - relative to the path of the module. In our case it's relative to the path of the remote template module.
• targetPath - relative to the path of the module source directory (for remote modules this means the root of the module repository, otherwise the directory of the module configuration).

In the example above we have 2 modules defined in the same file. Thus, both templated modules have the same module source directory. This is the reason why the last resolved modules always rewrites the previous results.

I think we should implement some sanity checks and throw a configuration error if any module generateFiles.targetPaths overlap. It would be much clearer and better than unpredicted behavior.

Regarding multiple files:

.manifests/.manifests-template.yml
.garden/build/s1/.manifests/.manifests-template.yml
.garden/build/s2/.manifests/.manifests-template.yml

This is also expected. The first one (.manifests/.manifests-template.yml) is a result of the generateFiles.targetPath configuration. It appears when the modules are getting resolved, i.e. before any builds are triggered.

The next 2 files from the build directory are created while the build step. Each one is taken from the files configuration entry that points to the output of generateFiles.targetPath. so, we have the same contents in both files under different build directories because both are read from the same path. That, in turn, happens because the modules share the same config path.

So, if you define multiple modules in the same file and use generateFiles, then the workaround you suggested is actually a recommended naming pattern to avoid unexpected file rewrites.

However, it's a terrible UX that Garden silently rewrites the generated files and allows unpredicted behavior. I'll file a PR to raise a config error if there are any overlapping paths in the generated files of the different modules.