[Feature] Sidecar support

[iamhopaul123]

Overview

Currently adding sidecar container in ecs-preview needs users to run app package then modify the generated CloudFormation template, then push their image manually, which is not an easy task. As a result, we consider adding sidecar patterns for ecs-preview to provide a pleasant experience of adding common sidecar containers.

Type of Sidecars

A sidecar container is a second container added to the task definition. We can generally categorize sidecar containers into three types: explicit sidecar, implicit sidecar (hidecar), and generic sidecar.

Explicit Sidecar

Explicit sidecar is a type of sidecar container which requires users to customized/configured image and explicitly performs as a sidecar container. To take NGINX as an example, users need to configure nginx.conf file and build their own NGINX image. Most of users are aware of the fact that they have NGINX as a sidecar image for their service.

Implicit Sidecar (Hidecar)

Implicit sidecar, on the contrary, doesn’t require users to have their own sidecar image and most of users do not even know the existence of the sidecar container. For example, currently Firelens as a sidecar container help users to route their app log. However, the images used by the sidecar container are basically the same so that users do not need to configure the image itself. AppMesh can be another example since most of cases the sidecar containers consume the same envoy image.

Generic Sidecar

Generic sidecars are sidecar containers that are not supported as a pattern. For example the sidecar requires adding/modifying other resources and IAM policies. According to our tenet Make difficult simple and make impossible possible, we support generic sidecars by allowing them to add addons stack for adding additional resources or policies.

Design Proposal

For explicit sidecar containers, we’ll have a “sidecars” field in the manifest and users can configure their explicit sidecar containers. And for implicit sidecar containers it should be case by case. For example, we’ll have a “firelensLog” field for configuring the log for their application, in which users can configure Firelens if they’d like to use fluentbit to route their log elsewhere. For generic sidecars, we’ll introduce how to add them as sidecars inecs-preview.

Explicit Sidecar

We’ll take X-ray as an example for adding explicit sidecars. To add an X-ray sidecar container to their application, users need to modify their manifest after running app init.

Manifest

sidecars:
  xray:
    port: 2000/udp
    image: 123456789012.dkr.ecr.us-east-2.amazonaws.com/xray-daemon    # Image URL for sidecar container. Local images and images in the Docker Hub registry and ECR repository are supported.
    # credentialParameter: # ARN of the secret containing the private repository credentials.

After configuring the manifest, users can run app deploy to deploy their application with sidecar container.

Accommodate with Current Commands

app show
app show requires no changes.

app logs
app logs will includes the log of the sidecar container.

app status
app status requires no changes.

Implicit Sidecars

Since for implicit sidecars the setting will be case by case. Here we will start with Firelens.

Manifest

logging:                              # Enable Firelens to route your logs (to CloudWatch by default)
  destination:                        # Configure where to send logs to.
    Name: cloudwatch
    # includePattern: ^[a-z][aeiou].*$
    # excludePattern: ^.*[aeiou]$
  enableMetadata: false               # Enable ECS log metadata or not.
  # secretOptions:                      # The secrets to pass to the log configuration from AWS Systems Manager (SSM) Parameter Store.
    # LOG_TOKEN: LOG_TOKEN              # The key is the name of the environment variable, the value is the name of the SSM parameter. 
  # configFile: ./extra.conf            # Path to the local config file for Firelens. Specified to override other configurations.
  # permissionFile: ./permissions.json  # Path to the optional permission file if "configFile" is set.

By default, we'll set the destination to be CloudWatch, and users can modify the destination to somewhere else. However, permission file might be required. See more Firelens example in Appendix.

Accommodate with Current Commands

app show
app show requires no changes.

app logs
app logs will show log of the main container and Firelens sidecar container if destination is set to "cloudwatch". However, it will only show the log of the sidecar container if the main app logs are routed to elsewhere (since it will be difficult for us to get those logs after they are routed to their destinations).

app status
app status requires no changes.

Generic Sidecars

For generic sidecars that we don't have a pattern yet, users can add those sidecars with addon stack. However, since addon stack can only add new resources/policies, if their sidecar containers require any main CFN changes, then they need to modify the CFN template generated by app package. In the future we might have a flag in app deploy that allows them to deploy their modified CFN template generated by app package.

Appendix

Send to multiple destinations

logging:                              # Enable Firelens to route your logs (to CloudWatch by default)
  destination:                        # Configure where to send logs to.  
    # includePattern: ^[a-z][aeiou].*$
    # excludePattern: ^.*[aeiou]$
  enableMetadata: false               # Enable ECS log metadata or not.
  # secretOptions:                      # The secrets to pass to the log configuration from AWS Systems Manager (SSM) Parameter Store.
    # LOG_TOKEN: LOG_TOKEN              # The key is the name of the environment variable, the value is the name of the SSM parameter. 
  configFile: ./extra.conf            # Path to the local config file for Firelens. Specified to override other configurations.
  permissionFile: ./permissions.json  # Path to the optional permission file if "configFile" is set.

extra.conf

[OUTPUT]
    Name   firehose
    Match  *
    region us-west-2
    delivery_stream stream-one

[OUTPUT]
    Name   firehose
    Match  *
    region us-west-2
    delivery_stream stream-two

permissions.json

{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Action": [
            "firehose:PutRecordBatch",
        ],
        "Resource": "*"
    }]
}

Send logs to datadog

logging:                              # Enable Firelens to route your logs (to CloudWatch by default)
  destination:                        # Configure where to send logs to.  
    Name: datadog,
    # includePattern: ^[a-z][aeiou].*$
    # excludePattern: ^.*[aeiou]$
    Host: http-intake.logs.datadoghq.com
    TLS: on
    dd_service: my-httpd-service
    dd_source: httpd
    dd_tags: project:example
    provider: ecs
  enableMetadata: false               # Enable ECS log metadata or not.
  # secretOptions:                      # The secrets to pass to the log configuration from AWS Systems Manager (SSM) Parameter Store.
    # LOG_TOKEN: LOG_TOKEN              # The key is the name of the environment variable, the value is the name of the SSM parameter. 
  # configFile: ./extra.conf            # Path to the local config file for Firelens. Specified to override other configurations.
  # permissionFile: ./permissions.json  # Path to the optional permission file if "configFile" is set.

[iamhopaul123]

Milestone

• Update manifest pkg to parse config for explicit sidecar (Update manifest pkg to expose config for explicit sidecar #949)
• Update manifest pkg to parse config for Firelens (Update manifest pkg to expose config for firelens as hidecar #958)
• Add target group registration config in “http” field allowing users to register different container
• Update CFN template and deploy/stack pkg to accommodate manifest changes
• Fix app logs for Firelens
• Add e2e test
• Add doc and wiki for this feature and also add examples