[Change Proposal] Support Terraform related files in deploy/tf

[endorama]

While working at adding extended Terraform support to elastic-package Terraform Deployer (to enable GCP billing testing) package linting fails with this error:

Error: checking package failed: linting package failed: found 4 validation errors:
   1. item [.gitignore] is not allowed in folder [.../test/packages/parallel/gcp/data_stream/billing/_dev/deploy/tf]
   2. item [.terraform.lock.hcl] is not allowed in folder [.../test/packages/parallel/gcp/data_stream/billing/_dev/deploy/tf]
   3. item [billing-schema.json] is not allowed in folder [.../test/packages/parallel/gcp/data_stream/billing/_dev/deploy/tf]
   4. item [test-data.ndjson.tftpl] is not allowed in folder **[.../test/packages/parallel/gcp/data_stream/billing/_dev/deploy/tf]**

This change proposal aims to add support _dev/deploy/tf folder. for:

1. *.tftpl; tftpl is the recommended naming pattern to use for Terraform template files.
2. *.json; JSON is a standard storage format for accessory data. For the specific use case JSON is used to store GCP BigQuery Table schema.
3. .terraform.lock.hcl; this file is Terraform Dependency Lock Files. Terraform CLI itself suggest adding this file to version control for reproducible dependency installation.
4. .gitignore. It seems gitignore is not supported even at package level, and this makes everything a bit more complex. Adding support for this would also enabled ignoring the same files in related tools (elastic-package build for example).

[mtojek]

Let's split this into the MVP to unblock you and further improvements.

MVP:

1. *.tftpl - TF template files, approved.
2. *.json - fixtures and any other data, approved.
3. .terraform.lock.hcl - dependencies lock, approved.

Would you mind opening a PR to the package-spec to address these ideas?

Further improvement:

1. .gitignore - this is a nice finding! It shouldn't be part of the package-spec but should be ignored by the spec validator at all. Could you please open another issue to fix it?

cc @jsoriano

[jsoriano]

Further improvement:

1. `.gitignore` - this is a nice finding! It shouldn't be part of the package-spec but should be ignored by the spec validator at all. Could you please open another issue to fix it?

Maybe we can ignore all files starting by dots. Also when building packages.

[mtojek]

I'm afraid that when people discover it, they will start sneaking extra content in .meta, .data, .extra, etc. I remember that how the SEI team introduced the entire application in the Docker service :)

[endorama]

I opened #273 to ignore .gitignore during spec validation.

[endorama]

In order to leverage these changes I have to wait a release, may you share when you expect to cut one?

[jsoriano]

In order to leverage these changes I have to wait a release, may you share when you expect to cut one?

We have to merge #270, or similar, current code fails to validate some of the existing packages.