From 8ee445a020d1a866839efedf5a9612ddb8c69db8 Mon Sep 17 00:00:00 2001 From: penghaoh Date: Tue, 24 Aug 2021 11:29:50 -0700 Subject: [PATCH 1/5] docs: add task def override --- site/content/docs/developing/overrides.en.md | 50 +++++++++++++++++++ .../docs/include/common-svc-fields.en.md | 15 +++++- 2 files changed, 63 insertions(+), 2 deletions(-) create mode 100644 site/content/docs/developing/overrides.en.md diff --git a/site/content/docs/developing/overrides.en.md b/site/content/docs/developing/overrides.en.md new file mode 100644 index 00000000000..41f6a3641ac --- /dev/null +++ b/site/content/docs/developing/overrides.en.md @@ -0,0 +1,50 @@ +# Overrides +If there are fields that are not configurable in the [manifest](../manifest/overview.en.md), users can bypass some of them by applying override rules to the CloudFormation template Copilot generates out of the manifest. + +## How to specify override rules? +For each override rule, users need to construct the **path** and **value** of the CloudFormation resource field they want to override. + +``` yaml +: + - path: + value: +``` + +## Override Behaviors + +- Copilot recursively creates `map` fields if they don't exist in the **path**. For example: if `B` and `C` don't exist, `A.B.C` will create `B` and `C` + +- Use `-` as index to append a new member to a `list` field. The field will be initiated with this new member if it doesn't exist + +- **value** must be scalar value like `bool`, `string`, or `int`, and the new value replaces the old value if exists + +## Examples + +### Task Definition Override +!!! Attention + Users are not allowed to modify the following fields in the task definition. + + * Family + * ContainerDefinitions[].Name (name of any existing container) + +Below is an example of adding [`Ulimits`](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-containerdefinitions-ulimit.html) to the main container. + +``` yaml +taskdef_overrides: + - path: "ContainerDefinitions[0].Ulimits[-].Name" + value: "cpu" + - path: "ContainerDefinitions[0].Ulimits[0].SoftLimit" + value: 1024 + - path: "ContainerDefinitions[0].Ulimits[0].HardLimit" + value: 2048 +``` + +Below is exposing an extra UDP port for the main container. + +``` yaml +taskdef_overrides: + - path: "ContainerDefinitions[0].PortMappings[-].ContainerPort" + value: 2056 + - path: "ContainerDefinitions[0].PortMappings[1].Protocol" + value: "udp" +``` diff --git a/site/content/docs/include/common-svc-fields.en.md b/site/content/docs/include/common-svc-fields.en.md index 84942170848..abb73cd94f7 100644 --- a/site/content/docs/include/common-svc-fields.en.md +++ b/site/content/docs/include/common-svc-fields.en.md @@ -204,10 +204,10 @@ Required. The ID of the filesystem you would like to mount. Optional. Defaults to `/`. Specify the location in the EFS filesystem you would like to use as the root of your volume. Must be fewer than 255 characters and must consist only of the characters `a-zA-Z0-9.-_/`. If using an access point, `root_dir` must be either empty or `/` and `auth.iam` must be `true`. volume.efs.`uid` Uint32 -Optional. Must be specified with `gid`. Mutually exclusive with `root_dir`, `auth`, and `id`. The POSIX UID to use for the dedicated access point created for the managed EFS filesystem. +Optional. Must be specified with `gid`. Mutually exclusive with `root_dir`, `auth`, and `id`. The POSIX UID to use for the dedicated access point created for the managed EFS filesystem. volume.efs.`gid` Uint32 -Optional. Must be specified with `uid`. Mutually exclusive with `root_dir`, `auth`, and `id`. The POSIX GID to use for the dedicated access point created for the managed EFS filesystem. +Optional. Must be specified with `uid`. Mutually exclusive with `root_dir`, `auth`, and `id`. The POSIX GID to use for the dedicated access point created for the managed EFS filesystem. volume.efs.`auth` Map Specify advanced authorization configuration for EFS. @@ -240,5 +240,16 @@ Optional. The full config file path in your custom Fluent Bit image.
+`taskdef_overrides` Array +The `taskdef_overrides` section allows users to apply overriding rules to their ECS Task Definitions (see examples [here](../developing/overrides.en.md#task-definition-override)). + +taskdef_overrides.`path` String +Required. Path to the Task Definition field to override. + +taskdef_overrides.`value` Any +Required. Value of the Task Definition field to override. + +
+ `environments` Map The environment section lets you override any value in your manifest based on the environment you're in. In the example manifest above, we're overriding the count parameter so that we can run 2 copies of our service in our 'prod' environment, and 2 copies using Fargate Spot capacity in our 'staging' environment. From 443a9a4192771818d92466d80ccbdd5e374d0a14 Mon Sep 17 00:00:00 2001 From: penghaoh Date: Wed, 25 Aug 2021 15:57:36 -0700 Subject: [PATCH 2/5] Address feedback --- mkdocs.yml | 1 + site/content/docs/developing/overrides.en.md | 50 ----------------- .../docs/developing/taskdef-overrides.en.md | 55 +++++++++++++++++++ .../docs/include/common-svc-fields.en.md | 2 +- 4 files changed, 57 insertions(+), 51 deletions(-) delete mode 100644 site/content/docs/developing/overrides.en.md create mode 100644 site/content/docs/developing/taskdef-overrides.en.md diff --git a/mkdocs.yml b/mkdocs.yml index 52eb709b0b6..893ad563e1f 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -47,6 +47,7 @@ nav: - Custom Environment Resources: docs/developing/custom-environment-resources.en.md - Secrets: docs/developing/secrets.en.md - Service Discovery: docs/developing/service-discovery.en.md + - Task Definition Overrides: docs/developing/taskdef-overrides.en.md - Additional AWS Resources: docs/developing/additional-aws-resources.en.md - Sidecars: docs/developing/sidecars.en.md - Storage: docs/developing/storage.en.md diff --git a/site/content/docs/developing/overrides.en.md b/site/content/docs/developing/overrides.en.md deleted file mode 100644 index 41f6a3641ac..00000000000 --- a/site/content/docs/developing/overrides.en.md +++ /dev/null @@ -1,50 +0,0 @@ -# Overrides -If there are fields that are not configurable in the [manifest](../manifest/overview.en.md), users can bypass some of them by applying override rules to the CloudFormation template Copilot generates out of the manifest. - -## How to specify override rules? -For each override rule, users need to construct the **path** and **value** of the CloudFormation resource field they want to override. - -``` yaml -: - - path: - value: -``` - -## Override Behaviors - -- Copilot recursively creates `map` fields if they don't exist in the **path**. For example: if `B` and `C` don't exist, `A.B.C` will create `B` and `C` - -- Use `-` as index to append a new member to a `list` field. The field will be initiated with this new member if it doesn't exist - -- **value** must be scalar value like `bool`, `string`, or `int`, and the new value replaces the old value if exists - -## Examples - -### Task Definition Override -!!! Attention - Users are not allowed to modify the following fields in the task definition. - - * Family - * ContainerDefinitions[].Name (name of any existing container) - -Below is an example of adding [`Ulimits`](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-containerdefinitions-ulimit.html) to the main container. - -``` yaml -taskdef_overrides: - - path: "ContainerDefinitions[0].Ulimits[-].Name" - value: "cpu" - - path: "ContainerDefinitions[0].Ulimits[0].SoftLimit" - value: 1024 - - path: "ContainerDefinitions[0].Ulimits[0].HardLimit" - value: 2048 -``` - -Below is exposing an extra UDP port for the main container. - -``` yaml -taskdef_overrides: - - path: "ContainerDefinitions[0].PortMappings[-].ContainerPort" - value: 2056 - - path: "ContainerDefinitions[0].PortMappings[1].Protocol" - value: "udp" -``` diff --git a/site/content/docs/developing/taskdef-overrides.en.md b/site/content/docs/developing/taskdef-overrides.en.md new file mode 100644 index 00000000000..e1f8a84efd7 --- /dev/null +++ b/site/content/docs/developing/taskdef-overrides.en.md @@ -0,0 +1,55 @@ +# Task Definition Overrides +If there are fields that are not configurable in the [manifest](../manifest/overview.en.md), users can bypass some of the [ECS Task Definition setting](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-taskdefinition.html) by applying override rules to the CloudFormation template Copilot generates out of the manifest. + +## How to specify override rules? +For each override rule, users need to construct the **path** and **value** of the CloudFormation resource field they want to override. + +``` yaml +taskdef_overrides: + - path: + value: +``` + +## Override Behaviors + +- Use `-` as index to append a new member to a `list` field + +- When applying the override rule, Copilot inserts or updates the fields along the path. More specifically, Copilot recursively inserts fields if they don't exist in the **path**. For example: if `B` and `C` don't exist, `A.B[-].C` will create `B` and `C` + +!!! Attention + Users are not allowed to modify the following fields in the task definition. + + * [Family](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-taskdefinition.html#cfn-ecs-taskdefinition-family) + * [ContainerDefinitions[].Name](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-containerdefinitions.html#cfn-ecs-taskdefinition-containerdefinition-name) + +## Examples + +**Add [`Ulimits`](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-containerdefinitions-ulimit.html) to the main container** + +``` yaml +taskdef_overrides: + - path: ContainerDefinitions[0].Ulimits[-] + value: + Name: "cpu" + SoftLimit: 1024 + HardLimit: 2048 +``` + +**Expose an extra UDP port** + +``` yaml +taskdef_overrides: + - path: "ContainerDefinitions[0].PortMappings[-].ContainerPort" + value: 2056 + // PortMappings[1] gets the port mapping added by the previous rule, since by default Copilot creates a port mapping. + - path: "ContainerDefinitions[0].PortMappings[1].Protocol" + value: "udp" +``` + +**Give read-only access to the root file system** + +``` yaml +taskdef_overrides: + - path: "ContainerDefinitions[0].ReadonlyRootFilesystem" + value: true +``` diff --git a/site/content/docs/include/common-svc-fields.en.md b/site/content/docs/include/common-svc-fields.en.md index abb73cd94f7..50887eff5ad 100644 --- a/site/content/docs/include/common-svc-fields.en.md +++ b/site/content/docs/include/common-svc-fields.en.md @@ -241,7 +241,7 @@ Optional. The full config file path in your custom Fluent Bit image.
`taskdef_overrides` Array -The `taskdef_overrides` section allows users to apply overriding rules to their ECS Task Definitions (see examples [here](../developing/overrides.en.md#task-definition-override)). +The `taskdef_overrides` section allows users to apply overriding rules to their ECS Task Definitions (see examples [here](../developing/taskdef-overrides.en.md#examples)). taskdef_overrides.`path` String Required. Path to the Task Definition field to override. From 61886e504c848108a658661e3e1fdd4797218a18 Mon Sep 17 00:00:00 2001 From: penghaoh Date: Wed, 25 Aug 2021 18:20:51 -0700 Subject: [PATCH 3/5] Address feedback --- .../docs/developing/taskdef-overrides.en.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/site/content/docs/developing/taskdef-overrides.en.md b/site/content/docs/developing/taskdef-overrides.en.md index e1f8a84efd7..4a8384a3136 100644 --- a/site/content/docs/developing/taskdef-overrides.en.md +++ b/site/content/docs/developing/taskdef-overrides.en.md @@ -1,8 +1,10 @@ # Task Definition Overrides -If there are fields that are not configurable in the [manifest](../manifest/overview.en.md), users can bypass some of the [ECS Task Definition setting](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-taskdefinition.html) by applying override rules to the CloudFormation template Copilot generates out of the manifest. +Copilot generates CloudFormation templates using configuration specified in the [manifest](../manifest/overview.en.md). However, there are fields that are not configurable in the manifest. For example, You might want to configure the [`Ulimits`](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-containerdefinitions.html#cfn-ecs-taskdefinition-containerdefinition-ulimits) for your workload container, but it is not exposed in our manifest. + +You can configure [ECS Task Definition settings](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-taskdefinition.html) by specifying override rules, which will be applied to the CloudFormation template that Copilot generates out of the manifest. ## How to specify override rules? -For each override rule, users need to construct the **path** and **value** of the CloudFormation resource field they want to override. +For each override rule, you need to specify a **path** of the CloudFormation resource field you want to override, and a **value** of that field. ``` yaml taskdef_overrides: @@ -14,17 +16,17 @@ taskdef_overrides: - Use `-` as index to append a new member to a `list` field -- When applying the override rule, Copilot inserts or updates the fields along the path. More specifically, Copilot recursively inserts fields if they don't exist in the **path**. For example: if `B` and `C` don't exist, `A.B[-].C` will create `B` and `C` +- When applying an override rule, Copilot inserts or updates the fields along the path. More specifically, Copilot recursively inserts fields if they don't exist in the CloudFormation template. For example: if a rule has the path `A.B[-].C` (`B` and `C` don't exist), Copilot will insert the field `B` and `C`. A concrete example can be found [below](#add-ulimits-to-the-main-container). !!! Attention - Users are not allowed to modify the following fields in the task definition. + The following fields in the task definition are not allowed to be modified. * [Family](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-taskdefinition.html#cfn-ecs-taskdefinition-family) * [ContainerDefinitions[].Name](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-containerdefinitions.html#cfn-ecs-taskdefinition-containerdefinition-name) ## Examples -**Add [`Ulimits`](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-containerdefinitions-ulimit.html) to the main container** +### Add `Ulimits` to the main container ``` yaml taskdef_overrides: @@ -35,7 +37,7 @@ taskdef_overrides: HardLimit: 2048 ``` -**Expose an extra UDP port** +### Expose an extra UDP port ``` yaml taskdef_overrides: @@ -46,7 +48,7 @@ taskdef_overrides: value: "udp" ``` -**Give read-only access to the root file system** +### Give read-only access to the root file system ``` yaml taskdef_overrides: From 145aa773b3338b095705d6179095a3b75b7eafbc Mon Sep 17 00:00:00 2001 From: penghaoh Date: Thu, 26 Aug 2021 09:55:39 -0700 Subject: [PATCH 4/5] Address feedback --- mkdocs.yml | 2 +- .../docs/developing/taskdef-overrides.en.md | 30 +++++++++++++++---- 2 files changed, 25 insertions(+), 7 deletions(-) diff --git a/mkdocs.yml b/mkdocs.yml index 893ad563e1f..d2ffd8ccda4 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -47,10 +47,10 @@ nav: - Custom Environment Resources: docs/developing/custom-environment-resources.en.md - Secrets: docs/developing/secrets.en.md - Service Discovery: docs/developing/service-discovery.en.md - - Task Definition Overrides: docs/developing/taskdef-overrides.en.md - Additional AWS Resources: docs/developing/additional-aws-resources.en.md - Sidecars: docs/developing/sidecars.en.md - Storage: docs/developing/storage.en.md + - Task Definition Overrides: docs/developing/taskdef-overrides.en.md - Commands: - Getting Started: - init: docs/commands/init.en.md diff --git a/site/content/docs/developing/taskdef-overrides.en.md b/site/content/docs/developing/taskdef-overrides.en.md index 4a8384a3136..ddee08b9005 100644 --- a/site/content/docs/developing/taskdef-overrides.en.md +++ b/site/content/docs/developing/taskdef-overrides.en.md @@ -1,22 +1,36 @@ # Task Definition Overrides + +!!! Attention + :warning: Task Definition overrides is an advanced use case. Overriding a field might cause the task not able to launch. Please use with caution! + Copilot generates CloudFormation templates using configuration specified in the [manifest](../manifest/overview.en.md). However, there are fields that are not configurable in the manifest. For example, You might want to configure the [`Ulimits`](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-containerdefinitions.html#cfn-ecs-taskdefinition-containerdefinition-ulimits) for your workload container, but it is not exposed in our manifest. -You can configure [ECS Task Definition settings](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-taskdefinition.html) by specifying override rules, which will be applied to the CloudFormation template that Copilot generates out of the manifest. +You can configure additional [ECS Task Definition settings](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-taskdefinition.html) by specifying `taskdef_overrides` rules, which will be applied to the CloudFormation template that Copilot generates out of the manifest. ## How to specify override rules? For each override rule, you need to specify a **path** of the CloudFormation resource field you want to override, and a **value** of that field. +The following is an example valid `taskdef_overrides` field that can be applied to a manifest file: + ``` yaml taskdef_overrides: - - path: - value: +- path: ContainerDefinitions[0].Cpu + value: 512 +- path: ContainerDefinitions[0].Memory + value: 1024 ``` -## Override Behaviors +Each rule is applied sequentially to the CloudFormation template. The resulting CloudFormation template becomes the target of the next rule. Evaluation continues until all rules are successfully applied or an error is encountered. + +## Path Evaluation -- Use `-` as index to append a new member to a `list` field +- The `path` field is a `'.'` character separated path to a target [Task Definition field under `Properties` in CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-taskdefinition.html). -- When applying an override rule, Copilot inserts or updates the fields along the path. More specifically, Copilot recursively inserts fields if they don't exist in the CloudFormation template. For example: if a rule has the path `A.B[-].C` (`B` and `C` don't exist), Copilot will insert the field `B` and `C`. A concrete example can be found [below](#add-ulimits-to-the-main-container). +- Copilot recursively inserts fields if they don't exist in the CloudFormation template. For example: if a rule has the path `A.B[-].C` (`B` and `C` don't exist), Copilot will insert the field `B` and `C`. A concrete example can be found [below](#add-ulimits-to-the-main-container). + +- If the target path specifies an member that already exists, that member's value is replaced. + +- To append a new member to a `list` field such as `Ulimits` you can use the special character `-`: `Ulimits[-]`. !!! Attention The following fields in the task definition are not allowed to be modified. @@ -24,6 +38,10 @@ taskdef_overrides: * [Family](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-taskdefinition.html#cfn-ecs-taskdefinition-family) * [ContainerDefinitions[].Name](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-containerdefinitions.html#cfn-ecs-taskdefinition-containerdefinition-name) +## Testing + +In order to ensure that your override rules behave as expected, we recommend running `copilot svc package` or `copilot job package` to preview the generated CloudFormation template. + ## Examples ### Add `Ulimits` to the main container From fbe204979bead0a4aa7e3ccb3ab130394b5c8cd9 Mon Sep 17 00:00:00 2001 From: penghaoh Date: Thu, 26 Aug 2021 12:04:19 -0700 Subject: [PATCH 5/5] Fix nits --- site/content/docs/developing/taskdef-overrides.en.md | 2 +- site/content/docs/include/common-svc-fields.en.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/site/content/docs/developing/taskdef-overrides.en.md b/site/content/docs/developing/taskdef-overrides.en.md index ddee08b9005..8e0fd3c8ce7 100644 --- a/site/content/docs/developing/taskdef-overrides.en.md +++ b/site/content/docs/developing/taskdef-overrides.en.md @@ -28,7 +28,7 @@ Each rule is applied sequentially to the CloudFormation template. The resulting - Copilot recursively inserts fields if they don't exist in the CloudFormation template. For example: if a rule has the path `A.B[-].C` (`B` and `C` don't exist), Copilot will insert the field `B` and `C`. A concrete example can be found [below](#add-ulimits-to-the-main-container). -- If the target path specifies an member that already exists, that member's value is replaced. +- If the target path specifies a member that already exists, that member's value is replaced. - To append a new member to a `list` field such as `Ulimits` you can use the special character `-`: `Ulimits[-]`. diff --git a/site/content/docs/include/common-svc-fields.en.md b/site/content/docs/include/common-svc-fields.en.md index 50887eff5ad..357ee1f530f 100644 --- a/site/content/docs/include/common-svc-fields.en.md +++ b/site/content/docs/include/common-svc-fields.en.md @@ -240,7 +240,7 @@ Optional. The full config file path in your custom Fluent Bit image.
-`taskdef_overrides` Array +`taskdef_overrides` Array of Rules The `taskdef_overrides` section allows users to apply overriding rules to their ECS Task Definitions (see examples [here](../developing/taskdef-overrides.en.md#examples)). taskdef_overrides.`path` String