aws_ecs_task_definition needs updating even if nothing changes (look at "portMappings")

[LinguineCode]

Terraform Version

Terraform v0.11.3

• provider.aws v1.8.0
• provider.template v1.0.0

Affected Resource(s)

Please list the resources as a list, for example:

• aws_ecs_task_definition

Terraform Configuration Files

resource "aws_ecs_task_definition" "main" {
  container_definitions = <<EOF
[
    {
        "name": "thename",
        "image": "theimage",
        "cpu": 10,
        "memory": 250,
        "links": [],
        "portMappings": [
            {
                "containerPort": 8888
            }
        ],
        "essential": true,
        "entryPoint": [],
        "command": [],
        "environment" : [
          { "name" : "PORT", "value" : "8888" }
        ],
        "mountPoints": [],
        "volumesFrom": []
    }
]
EOF
  family                = "${basename("${path.cwd}")}"
  network_mode          = "awsvpc"

  volume {
    name      = "shared"
    host_path = "/mnt/efs1/shared"
  }

}

Debug Output

I will paste upon request. Too lazy to edit out sensitive data.

Panic Output

no panic output

Expected Behavior

aws_ecs_task_definition did not change, so it needs no update

Actual Behavior

aws_ecs_task_definition needed an update

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply

Important Factoids

I am 99% certain the portMappings is to blame. If I delete settings in portMappings area, the task does not need an update.

References

Are there any other GitHub issues (open or closed) or Pull Requests that should be linked here?
#2339

[LinguineCode]

NEW FINDING! network_mode = bridge does not cause this problem. You have to set:

network_mode = host, or, network_mode = awsvpc

[LinguineCode]

Hey everyone, let me know if you need me to do any troubleshooting or answer any questions for you

[tomelliff]

I've been seeing the same thing and it's finally annoyed me enough to take a look at it. However while I have a few other things that cause a diff (duplicated environment variables due to hacking things for variable environment configuration in a module) I also see the same issue as @seanscottking around portMappings.

Taking a look at the code though there's already code to suppress the diff when protocol = "tcp" and hostPort = 0 and even has a passing test so I'm not too sure why this is showing a diff.

[pgavlin]

We've been hitting this as well. It is absolutely the portMappings that are to blame.

As per the AWS docs, the hostPort for a portMapping must either be 0 or equal to the containerPort when the network mode is awsvpc. As it so happens, ECS happily makes this transformation atop the incoming container definition.

To be more concrete, the following TF config:

provider "aws" {
  region = "us-west-2"
}

resource "aws_ecs_task_definition" "jaeger" {
  family                = "jaeger"
  container_definitions = "[{\"name\": \"jaeger-collector\", \"image\": \"jaegertracing/jaeger-collector\", \"essential\": true, \"portMappings\": [{\"containerPort\": 9411}]}]"

  memory = 1024
  cpu = 256

  network_mode = "awsvpc"
}

Will generate the following state:

{
    "version": 3,
    "terraform_version": "0.11.1",
    "serial": 3,
    "lineage": "3bac4887-fc94-4393-a75e-bbd11e663c00",
    "modules": [
        {
            "path": [
                "root"
            ],
            "outputs": {},
            "resources": {
                "aws_ecs_task_definition.jaeger": {
                    "type": "aws_ecs_task_definition",
                    "depends_on": [],
                    "primary": {
                        "id": "jaeger",
                        "attributes": {
                            "arn": "arn:aws:ecs:us-west-2:153052954103:task-definition/jaeger:1",
                            "container_definitions": "[{\"cpu\":0,\"environment\":[],\"essential\":true,\"image\":\"jaegertracing/jaeger-collector\",\"mountPoints\":[],\"name\":\"jaeger-collector\",\"portMappings\":[{\"containerPort\":9411,\"hostPort\":9411,\"protocol\":\"tcp\"}],\"volumesFrom\":[]}]",
                            "cpu": "256",
                            "execution_role_arn": "",
                            "family": "jaeger",
                            "id": "jaeger",
                            "memory": "1024",
                            "network_mode": "awsvpc",
                            "placement_constraints.#": "0",
                            "requires_compatibilities.#": "0",
                            "revision": "1",
                            "task_role_arn": "",
                            "volume.#": "0"
                        },
                        "meta": {
                            "schema_version": "1"
                        },
                        "tainted": false
                    },
                    "deposed": [],
                    "provider": "provider.aws"
                }
            },
            "depends_on": []
        }
    ]
}

Note in particular the addition of the hostPort property in the sole container definition. This addition will then cause successive calls to Diff to decide that the container definition has changed even if it has not:

2018-04-24T15:37:13.250-0700 [DEBUG] plugin.terraform-provider-aws_v1.15.0_x4: 2018/04/24 15:37:13 [DEBUG] Canonical definitions are not equal.
2018-04-24T15:37:13.250-0700 [DEBUG] plugin.terraform-provider-aws_v1.15.0_x4: First: [{"essential":true,"image":"jaegertracing/jaeger-collector","name":"jaeger-collector","portMappings":[{"containerPort":9411,"hostPort":9411}]}]
2018-04-24T15:37:13.250-0700 [DEBUG] plugin.terraform-provider-aws_v1.15.0_x4: Second: [{"essential":true,"image":"jaegertracing/jaeger-collector","name":"jaeger-collector","portMappings":[{"containerPort":9411}]}]

I believe that a reasonable fix inline with the docs would be to ignore differences between hostPort for awsvpc if the diff is between 0 or unset and containerPort.

Thoughts?

[bflad]

The fix for this has been merged into master and will release with version 1.36.0 of the AWS provider, likely tomorrow. 👍

[viatcheslavmogilevsky]

The provider also keeps updating aws_ecs_task_definition when healthCheck config of container definition isn't complete.

Terraform v0.11.7
provider.aws v1.25.0

For example, this task definition keeps updating:

...
      "healthCheck": {
          "command": [
            "CMD-SHELL",
            "curl -s -f -X GET 'localhost:8080/check' || exit 1"
          ],
          "startPeriod": 100,
      }
...

So I have to include whole config to fix the issue:

...
      "healthCheck": {
          "command": [
            "CMD-SHELL",
            "curl -s -f -X GET 'localhost:8080/check' || exit 1"
          ],
          "interval":30,
          "retries":3,
          "startPeriod":100,
          "timeout":5
      }
...

Update: this behaviour could be happen in the latest version at the moment: v1.35.0

[bflad]

@viatcheslavmogilevsky can you please open a separate issue or double check that there is not an existing service/ecs issue already reported for that? Thanks.

The port mappings fix has been released in version 1.36.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!