Add --exclude-processing option

[lliissoonngg]

Fixes #1911

This PR added a CLI option --exclude-processing and an equivelent config entry excludeProcessing, if it is set, the matching files will go with a different code path copyFileToOutDir.

To make the code reusable, the codes about file IO are offloaded fromfileToTemplate into readInFile, getOutfileHandler, and are called by copyFileToOutDir as well.

How to test:

Have a jpeg file in in directory.

case 1: (test cli argument --exclude-processing)

run:
./gomplate --exclude-processing '*.jpg' --input-dir=in --output-dir=out

the *.jpg exists in out directory

case 2: (test config file entry excludeProcessing)

have a .gomplate.yaml file in the directory:
`
inputDir: in/
outputDir: out/
excludeProcessing:

• '*.jpg'
  `

run:
./gomplate --input-dir=in --output-dir=out

the *.jpg exists in out directory

[hairyhenderson]

Hi @lliissoonngg, thanks for working on this.

Before I review this fully, a couple things:

1. some integration tests need to be written for this (see internal/tests/integration/)
2. I'm not sure about the name of this flag - "exclude-processing" isn't very clear - perhaps you have an idea on a better name? Or maybe @assayire from the original issue?

[hairyhenderson]

I think the previous text was fine - "ignore" is somewhat ambiguous in this context

[lliissoonngg]

@assayire gently bump on this, could you take a look of the parameter naming in this PR?

[assayire]

./gomplate --exclude-processing '*.jpg' --input-dir=in --output-dir=out

I am fine with the name.

Quick question: If I have exclude say jpg and png, would I do --exclude-processing '*.jpg,*.png'?

[lliissoonngg]

./gomplate --exclude-processing '*.jpg' --input-dir=in --output-dir=out

I am fine with the name.

Quick question: If I have exclude say jpg and png, would I do --exclude-processing '*.jpg,*.png'?

No, --excludes has higher priority, in this case jpg and png will not be copied. Let me know if this is good enough. I will update the doc after we all agree upon a working logic.

[assayire]

./gomplate --exclude-processing '*.jpg' --input-dir=in --output-dir=out

I am fine with the name.
Quick question: If I have exclude say jpg and png, would I do --exclude-processing '*.jpg,*.png'?

No, --excludes has higher priority, in this case jpg and png will not be copied. Let me know if this is good enough. I will update the doc after we all agree upon a working logic.

Oops .. i think i typed it wrong. I meant to ask that if i have exclude multiple files from processing, would this work - --exclude-processing '*.jpg,*.png,*.gif'?

[lliissoonngg]

./gomplate --exclude-processing '*.jpg' --input-dir=in --output-dir=out

I am fine with the name.
Quick question: If I have exclude say jpg and png, would I do --exclude-processing '*.jpg,*.png'?

No, --excludes has higher priority, in this case jpg and png will not be copied. Let me know if this is good enough. I will update the doc after we all agree upon a working logic.

Oops .. i think i typed it wrong. I meant to ask that if i have exclude multiple files from processing, would this work - --exclude-processing '*.jpg,*.png,*.gif'?

Yes, multiple patterns works (exactly the same as --excludes).

[hairyhenderson]

would this work - --exclude-processing '*.jpg,*.png,*.gif'?

If it works like --exclude then I think it would not be a comma-separated string, but instead multiple options, so:

--exclude-processing '*.jpg' --exclude-processing '*.png' --exclude-processing '*.gif'

For something so verbose as this it would make more sense to use a config file, which would look like:

excludeProcessing
  - '*.jpg'
  - '*.png'
  - '*.gif'

Hope that makes sense @assayire

[hairyhenderson]

@lliissoonngg thanks for the updates. Can you please rebase? I've just merged #1336 which makes a large number of changes that will affect this.

I'll make another review pass after that.

[tewfik-ghariani]

Would it be also possible to exclude the processing of certain files even if "input" and "output" are not specified ( regardless of the copying operation)?
For example

$ ls -1 helm-values/
template.yml.tmpl
values.yml

$ gomplate -f helm-values/ --exclude-processing  '*.yml'

[hairyhenderson]

@tewfik-ghariani that command doesn't really make sense - -f must refers to a file, not a directory. This option would only make sense when using --input-dir.

[tewfik-ghariani]

Ah right, @hairyhenderson thanks for the clarification ^^

[assayire]

would this work - --exclude-processing '*.jpg,*.png,*.gif'?

If it works like --exclude then I think it would not be a comma-separated string, but instead multiple options, so:

--exclude-processing '*.jpg' --exclude-processing '*.png' --exclude-processing '*.gif'

For something so verbose as this it would make more sense to use a config file, which would look like:

excludeProcessing
  - '*.jpg'
  - '*.png'
  - '*.gif'

Hope that makes sense @assayire

@hairyhenderson How would you specify it on command line to --exclude-processing? I mean the multiple files as you have specified in yaml.

[hairyhenderson]

@assayire like I said:

If it works like --exclude then I think it would not be a comma-separated string, but instead multiple options, so:

--exclude-processing '*.jpg' --exclude-processing '*.png' --exclude-processing '*.gif'

[assayire]

LGTM

[hairyhenderson]

Thanks again for this contribution @lliissoonngg! I'm going to rebase and merge this now, and will work (later) on replacing the xignore module.

[assayire]

Is this update released?

[mgbowman]

@assayire, it doesn't look like it. This was merged on Feb 6 and the v3.11.7 release was published on Jan 23, which is a pity because I could really use this for my current project. 😞

@hairyhenderson any chance we can get a v3.x release with this option or do we need to wait for v4.x?

[hairyhenderson]

This is a new feature and as such is going in the next release, which will be v4.

I could cut a v4.0.0-pre-3 pre-release if that helps.

(also in general please don't pose questions on closed issues, they're very easy to miss!)