Merged in changes from upstream.

README.md

@@ -54,9 +54,6 @@ The following software packages, or their equivalents, are expected to be instal
 - [Docker](https://www.docker.com/) container runtime or a compatible tool, e.g. [Podman](https://podman.io/),
 - [asdf](https://asdf-vm.com/) version manager,
 - [GNU make](https://www.gnu.org/software/make/) 3.82 or later,
-- [GNU coreutils](https://www.gnu.org/software/coreutils/) and [GNU binutils](https://www.gnu.org/software/binutils/) may be required to build dependencies like Python, which may need to be compiled during installation. For macOS users, this has been scripted and automated by the `dotfiles` project; please see this [script](https://github.com/nhs-england-tools/dotfiles/blob/main/assets/20-install-base-packages.macos.sh) for details,
-- [Python](https://www.python.org/) required to run Git hooks,
-- [`jq`](https://jqlang.github.io/jq/) a lightweight and flexible command-line JSON processor.
 
 > [!NOTE]<br>
 > The version of GNU make available by default on macOS is earlier than 3.82. You will need to upgrade it or certain `make` tasks will fail. On macOS, you will need [Homebrew](https://brew.sh/) installed, then to install `make`, like so:
@@ -65,7 +62,16 @@ The following software packages, or their equivalents, are expected to be instal
 > brew install make
 > ```
 >
-> You will then see instructions to fix your `$PATH` variable to make the newly installed version available. If you are using [dotfiles](https://github.com/nhs-england-tools/dotfiles), this is all done for you.
+> You will then see instructions to fix your [`$PATH`](https://github.com/nhs-england-tools/dotfiles/blob/main/dot_path.tmpl) variable to make the newly installed version available. If you are using [dotfiles](https://github.com/nhs-england-tools/dotfiles), this is all done for you.
+
+- [GNU sed](https://www.gnu.org/software/sed/) and [GNU grep](https://www.gnu.org/software/grep/) are required for the scripted command-line output processing,
+- [GNU coreutils](https://www.gnu.org/software/coreutils/) and [GNU binutils](https://www.gnu.org/software/binutils/) may be required to build dependencies like Python, which may need to be compiled during installation,
+
+> [!NOTE]<br>
+> For macOS users, installation of the GNU toolchain has been scripted and automated as part of the `dotfiles` project. Please see this [script](https://github.com/nhs-england-tools/dotfiles/blob/main/assets/20-install-base-packages.macos.sh) for details.
+
+- [Python](https://www.python.org/) required to run Git hooks,
+- [`jq`](https://jqlang.github.io/jq/) a lightweight and flexible command-line JSON processor.
 
 ### Configuration
 
@@ -91,6 +97,18 @@ The [C4 model](https://c4model.com/) is a simple and intuitive way to create sof
 
 ![Repository Template](./docs/diagrams/Repository_Template_GitHub_Generic.png)
 
+The source for diagrams should be in Git for change control and review purposes. Recommendations are [draw.io](https://app.diagrams.net/) (example above in [docs](.docs/diagrams/) folder) and [Mermaids](https://github.com/mermaid-js/mermaid). Here is an example Mermaids sequence diagram:
+
+```mermaid
+sequenceDiagram
+    User->>+Service: GET /users?params=...
+    Service->>Service: auth request
+    Service->>Database: get all users
+    Database-->>Service: list of users
+    Service->>Service: filter users
+    Service-->>-User: list[User]
+```
+
 ### Modularity
 
 Most of the projects are built with customisability and extendability in mind. At a minimum, this can be achieved by implementing service level configuration options and settings. The intention of this section is to show how this can be used. If the system processes data, you could mention here for example how the input is prepared for testing - anonymised, synthetic or live data.

docs/developer-guides/Scripting_Terraform.md

@@ -74,35 +74,202 @@ Here are some key features built into this repository's Terraform module:
 
 ### Quick start
 
-Run the example:
+The Repository Template assumes that you will be constructing the bulk of your infrastructure in `infrastructure/modules` as generic deployment configuration, which you will then compose into environment-specific modules, each stored in their own directory under `infrastructure/environments`.  Let's create a simple deployable thing, and configure an S3 bucket.  We'll make the name of the bucket a variable, so that each environment can have its own.
+
+Open the file `infrastructure/modules/private_s3_bucket/main.tf`, and put this in it:
+
+```terraform
+# Define the provider
+provider "aws" {
+  region = "eu-west-2"
+}
+
+variable "bucket_name" {
+  description = "Name of the bucket, which can be different per environment"
+}
+
+resource "aws_s3_bucket" "my_bucket" {
+  bucket = var.bucket_name # Replace with your desired bucket name
+  acl    = "private"
+}
+```
+
+Note that the variable has been given no value.  This is intentional, and allows us to pass the bucket name in as a parameter from the environment.
+
+Now, we're going to define two deployment environments: `dev`, and `test`.  Run this:
+
+```bash
+mkdir -p infrastructure/environments/{dev,test}
+```
+
+It is important that the directory names match your environment names.
+
+Now, let's create the environment definition files.  Open `infrastructure/environments/dev/main.tf` and copy in:
+
+```terraform
+module "dev_environment" {
+  source = "../../modules/private_s3_bucket"
+  bucket_name = "nhse-ee-my-fancy-bucket"
+}
+```
+
+Some things to note:
+
+- The `source` path is relative to the directory that the `main.tf` file is in.  When `terraform` runs, it will `chdir` to that directory first, before doing anything else.
+- The `module` name, `"dev_environment"` here, can be anything.  Module names are only scoped to the file they're in, so you don't need to follow any particular convention here.
+- The `bucket_name` is going to end up as the bucket name in AWS.  It wants to be meaningful to you, and you need to pick your own.  The framework doesn't constrain your choice, but remember that AWS needs them to be globally unique and if you steal `"nhse-ee-my-fancy-bucket"` then I can't test these docs and then I will be sad.
+
+Let's create our `test` environment now.  Open `infrastructure/environments/test/main.tf` and copy in:
+
+```terraform
+module "test_environment" {
+  source = "../../modules/private_s3_bucket"
+  bucket_name = "nhse-ee-my-fancy-test-bucket"
+}
+```
+
+We have changed the bucket name here.  In this example, I am making no assumptions as to how your AWS accounts are set up.  If you intend for your development and test infrastructure to be in the same AWS account (perhaps by necessity, for organisational reasons) and you need to separate them by a naming convention, the framework can support that.
+
+Now we have our modules and our environments configured, we need to initialise each of them.  Run these two commands:
+
+```bash
+TF_ENV=dev make terraform-init
+TF_ENV=test make terraform-init
+```
+
+Each invocation will download the `terraform` dependencies we need.  The `TF_ENV` name we give to each invocation is the name of the environment, and must match the directory name we chose under `infrastructure/environments` so that `make` gives the right parameters to `terraform`.
+
+We are now ready to try deploying to AWS, from our local environment.
+
+I am going to assume that you have an `~/.aws/credentials` file set up with a separate profile for each environment that you want to use, called `my-test-environment` and `my-dev-environment`.  They might have the same credential values in them, in which case `terraform` will create the resources in the same account; or you might have them set up to deploy to different accounts.  Either would work.
+
+Run the following:
+
+```shell
+TF_ENV=dev AWS_PROFILE=my-dev-environment make terraform-plan
+```
+
+If all is working correctly (and you may need to do a round of `aws sso login` first), you should see this output:
+
+```text
+
+Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+  + create
+
+Terraform will perform the following actions:
+
+  # module.dev_environment.aws_s3_bucket.my_bucket will be created
+  + resource "aws_s3_bucket" "my_bucket" {
+      + acceleration_status         = (known after apply)
+      + acl                         = "private"
+      + arn                         = (known after apply)
+      + bucket                      = "my-dev-bucket"
+      + bucket_domain_name          = (known after apply)
+      + bucket_prefix               = (known after apply)
+      + bucket_regional_domain_name = (known after apply)
+      + force_destroy               = false
+      + hosted_zone_id              = (known after apply)
+      + id                          = (known after apply)
+      + object_lock_enabled         = (known after apply)
+      + policy                      = (known after apply)
+      + region                      = (known after apply)
+      + request_payer               = (known after apply)
+      + tags_all                    = (known after apply)
+      + website_domain              = (known after apply)
+      + website_endpoint            = (known after apply)
+    }
+
+Plan: 1 to add, 0 to change, 0 to destroy.
+
+─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
+
+Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.
+
+```
+
+No errors found, so we can now create the bucket:
 
 ```shell
-# AWS console access setup
-export AWS_ACCESS_KEY_ID="..."
-export AWS_SECRET_ACCESS_KEY="..."
-export AWS_SESSION_TOKEN="..."
+ $ TF_ENV=dev AWS_PROFILE=my-dev-environment make terraform-apply
+
+Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+  + create
+
+Terraform will perform the following actions:
+
+  # module.dev_environment.aws_s3_bucket.my_bucket will be created
+  + resource "aws_s3_bucket" "my_bucket" {
+      + acceleration_status         = (known after apply)
+      + acl                         = (known after apply)
+      + arn                         = (known after apply)
+      + bucket                      = "nhse-ee-my-dev-bucket"
+      + bucket_domain_name          = (known after apply)
+      + bucket_prefix               = (known after apply)
+      + bucket_regional_domain_name = (known after apply)
+      + force_destroy               = false
+      + hosted_zone_id              = (known after apply)
+      + id                          = (known after apply)
+      + object_lock_enabled         = (known after apply)
+      + policy                      = (known after apply)
+      + region                      = (known after apply)
+      + request_payer               = (known after apply)
+      + tags_all                    = (known after apply)
+      + website_domain              = (known after apply)
+      + website_endpoint            = (known after apply)
+    }
+
+Plan: 1 to add, 0 to change, 0 to destroy.
+
+Do you want to perform these actions?
+  Terraform will perform the actions described above.
+  Only 'yes' will be accepted to approve.
+
+  Enter a value: yes
+
+module.dev_environment.aws_s3_bucket.my_bucket: Creating...
+module.dev_environment.aws_s3_bucket.my_bucket: Creation complete after 1s [id=nhse-ee-my-dev-bucket]
+
+Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
+
 ```
 
+You will notice here that I needed to confirm the action to `terraform` manually.  If you don't want to do that, you can pass the `-auto-approve` option to `terraform` like this:
+
 ```shell
-$ make terraform-example-provision-aws-infrastructure
+TF_ENV=dev AWS_PROFILE=my-dev-environment make terraform-apply opts="-auto-approve"
+```
 
-Initializing the backend..
-...
-Plan: 5 to add, 0 to change, 0 to destroy.
-Saved the plan to: terraform.tfplan
-To perform exactly these actions, run the following command to apply:
-    terraform apply "terraform.tfplan"
+If you check the contents of your AWS account, you should see your new bucket:
+
+```shell
+ $ aws s3 ls --profile my-dev-environment
 ...
-Apply complete! Resources: 5 added, 0 changed, 0 destroyed.
+2024-03-01 16:33:55 nhse-ee-my-dev-bucket
+```
 
-$ make terraform-example-destroy-aws-infrastructure
+Now I don't want to leave that there, so I will run the corresponding `destroy` command to get rid of it:
 
-...
-Plan: 0 to add, 0 to change, 5 to destroy.
-...
-Apply complete! Resources: 0 added, 0 changed, 5 destroyed.
+```shell
+ $ TF_ENV=dev AWS_PROFILE=my-dev-environment make terraform-destroy opts="-auto-approve"
+module.dev_environment.aws_s3_bucket.my_bucket: Refreshing state... [id=nhse-ee-my-dev-bucket]
+
+Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+  - destroy
+
+Terraform will perform the following actions:
+
+  # module.dev_environment.aws_s3_bucket.my_bucket will be destroyed
+  ...(more terraform output not shown because it's boring, but the end result is the bucket going away)
 ```
 
+To create your `test` environment, you run the same commands with `test` where previously you had `dev`:
+
+```shell
+TF_ENV=test AWS_PROFILE=my-test-environment make terraform-apply opts="-auto-approve"
+```
+
+To use the same `terraform` files in a GitHub action, see the docs [here](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services).
+
 ### Your stack implementation
 
 Always follow [best practices for using Terraform](https://cloud.google.com/docs/terraform/best-practices-for-terraform) while providing infrastructure as code (IaC) for your service.

scripts/config/vale/styles/Vocab/words/accept.txt

@@ -13,6 +13,7 @@ Trufflehog
 bot
 idempotence
 onboarding
+sed
 toolchain
 [A-Z]+s
 GitHub

scripts/githooks/check-markdown-format.sh

@@ -38,7 +38,7 @@ function main() {
   check=${check:-working-tree-changes}
   case $check in
     "all")
-      files="$(find ./ -type f -name "*.md")"
+      files="$(git ls-files "*.md")"
       ;;
     "staged-changes")
       files="$(git diff --diff-filter=ACMRT --name-only --cached "*.md")"