unexpected behavior of defaultTags

[filip-zyzniewski]

Hello!

• Vote on this issue by adding a 👍 reaction
• To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already)

Issue details

The AWS provider ignores changes to defaultTags after a resource has been created unless other attributes are being changed

Steps to reproduce

1. Specify some defaultTags and create a bucket. The bucket has the tags as expected:

~/tmp/wtf/defaultTags$ cat index.ts
import * as aws from "@pulumi/aws";

const bucket = new aws.s3.Bucket("bucket-with-default-tags", {bucket: "bucket-with-default-tags"});
~/tmp/wtf/defaultTags$ cat Pulumi.dev.yaml
encryptionsalt: v1:ggyom6IhdG8=:v1:sJqljlbglc8JJgpe:jYzdmym/+StNPwPbn+7uIpAIL+fKLA==
config:
  aws:region: eu-west-1
  aws:defaultTags:
    tags:
      Tag1: value1
      Tag2: value2
~/tmp/wtf/defaultTags$ pulumi up --diff
Previewing update (dev):
  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:dev::defaultTags::pulumi:pulumi:Stack::defaultTags-dev]
    + aws:s3/bucket:Bucket: (create)
        [urn=urn:pulumi:dev::defaultTags::aws:s3/bucket:Bucket::bucket-with-default-tags]
        [provider=urn:pulumi:dev::defaultTags::pulumi:providers:aws::default_4_23_0::04da6b54-80e4-46f7-96ec-b56ff0331ba9]
        acl         : "private"
        bucket      : "bucket-with-default-tags"
        forceDestroy: false

import * as aws from "@pulumi/aws";
Resources:
    + 1 to create
    1 unchanged
Do you want to perform this update? yes
Updating (dev):
  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:dev::defaultTags::pulumi:pulumi:Stack::defaultTags-dev]
    + aws:s3/bucket:Bucket: (create)
        [urn=urn:pulumi:dev::defaultTags::aws:s3/bucket:Bucket::bucket-with-default-tags]
        [provider=urn:pulumi:dev::defaultTags::pulumi:providers:aws::default_4_23_0::8510e26a-009d-460f-a5e1-4f5e95a6af22]
        acl         : "private"
        bucket      : "bucket-with-default-tags"
        forceDestroy: false
Resources:
    + 1 created
    1 unchanged

Duration: 11s
~/tmp/wtf/defaultTags$ aws s3api get-bucket-tagging --bucket bucket-with-default-tags
{
    "TagSet": [
        {
            "Key": "Tag1",
            "Value": "value1"
        },
        {
            "Key": "Tag2",
            "Value": "value2"
        }
    ]
}

2. Add some new tags to the defaultTags and attempt an update, which shows no diff and results in no changes:

~/tmp/wtf/defaultTags$ echo '      Tag3: value3' >> Pulumi.dev.yaml
~/tmp/wtf/defaultTags$ echo '      Tag4: value4' >> Pulumi.dev.yaml
~/tmp/wtf/defaultTags$ cat Pulumi.dev.yaml
encryptionsalt: v1:ggyom6IhdG8=:v1:sJqljlbglc8JJgpe:jYzdmym/+StNPwPbn+7uIpAIL+fKLA==
config:
  aws:region: eu-west-1
  aws:defaultTags:
    tags:
      Tag1: value1
      Tag2: value2
      Tag3: value3
      Tag4: value4
~/tmp/wtf/defaultTags$ pulumi up --diff
Previewing update (dev):
  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:dev::defaultTags::pulumi:pulumi:Stack::defaultTags-dev]
Resources:
    2 unchanged
Do you want to perform this update? yes
Updating (dev):
  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:dev::defaultTags::pulumi:pulumi:Stack::defaultTags-dev]
Resources:
    2 unchanged

Duration: 3s
~/tmp/wtf/defaultTags$ aws s3api get-bucket-tagging --bucket bucket-with-default-tags
{
    "TagSet": [
        {
            "Key": "Tag1",
            "Value": "value1"
        },
        {
            "Key": "Tag2",
            "Value": "value2"
        }
    ]
}

3. Change another property of the bucket and observe how the new tags are also added to the bucket (but are not being shown in the preview):

~/tmp/wtf/defaultTags$ vim index.ts
~/tmp/wtf/defaultTags$ cat index.ts
import * as aws from "@pulumi/aws";

const bucket = new aws.s3.Bucket("bucket-with-default-tags", {
    bucket: "bucket-with-default-tags",
    acl: "authenticated-read",
});
~/tmp/wtf/defaultTags$ pulumi up --diff
Previewing update (dev):
  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:dev::defaultTags::pulumi:pulumi:Stack::defaultTags-dev]
    ~ aws:s3/bucket:Bucket: (update)
        [id=bucket-with-default-tags]
        [urn=urn:pulumi:dev::defaultTags::aws:s3/bucket:Bucket::bucket-with-default-tags]
        [provider=urn:pulumi:dev::defaultTags::pulumi:providers:aws::default_4_23_0::8510e26a-009d-460f-a5e1-4f5e95a6af22]
      ~ acl: "private" => "authenticated-read"
Resources:
    ~ 1 to update
    1 unchanged
Do you want to perform this update? yes
Updating (dev):
  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:dev::defaultTags::pulumi:pulumi:Stack::defaultTags-dev]
    ~ aws:s3/bucket:Bucket: (update)
        [id=bucket-with-default-tags]
        [urn=urn:pulumi:dev::defaultTags::aws:s3/bucket:Bucket::bucket-with-default-tags]
        [provider=urn:pulumi:dev::defaultTags::pulumi:providers:aws::default_4_23_0::8510e26a-009d-460f-a5e1-4f5e95a6af22]
      ~ acl: "private" => "authenticated-read"
        --outputs:--
      ~ tagsAll                          : {
            Tag1: "value1"
            Tag2: "value2"
          + Tag3: "value3"
          + Tag4: "value4"
        }
Resources:
    ~ 1 updated
    1 unchanged

Duration: 15s
~/tmp/wtf/defaultTags$ aws s3api get-bucket-tagging --bucket bucket-with-default-tags
{
    "TagSet": [
        {
            "Key": "Tag1",
            "Value": "value1"
        },
        {
            "Key": "Tag4",
            "Value": "value4"
        },
        {
            "Key": "Tag3",
            "Value": "value3"
        },
        {
            "Key": "Tag2",
            "Value": "value2"
        }
    ]
}
~/tmp/wtf/defaultTags$

Expected:

• I expected the Tag3 and Tag4 to be added to the bucket in step 2.
• I also expected the preview in step 3 to show the addition of Tag3 and Tag4 given that it has added them in the update.

Actual:

• Tag3 and Tag4 were not added in step 2
• Addition of Tag3 and Tag4 was not shown in preview of step 3

Inspiration for this issue report: https://stackoverflow.com/a/68829181/13846505

[filip-zyzniewski]

The terraform provider behaves as expected:

~/tmp/wtf/defaultTags/terraform$ ls
main.tf
~/tmp/wtf/defaultTags/terraform$ cat main.tf
provider "aws" {
  region = "eu-west-1"
  default_tags {
    tags = {
      Tag1 = "Value1"
      Tag2 = "Value2"
    }
  }
}

resource "aws_s3_bucket" "bucket-with-default-tags-terraform" {
  bucket = "bucket-with-default-tags-terraform"
}
~/tmp/wtf/defaultTags/terraform$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/aws v3.62.0...
- Installed hashicorp/aws v3.62.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository

provider "aws" {
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
~/tmp/wtf/defaultTags/terraform$ terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_s3_bucket.bucket-with-default-tags-terraform will be created
  + resource "aws_s3_bucket" "bucket-with-default-tags-terraform" {
      + acceleration_status         = (known after apply)
      + acl                         = "private"
      + arn                         = (known after apply)
      + bucket                      = "bucket-with-default-tags-terraform"
      + bucket_domain_name          = (known after apply)
      + bucket_regional_domain_name = (known after apply)
      + force_destroy               = false
      + hosted_zone_id              = (known after apply)
      + id                          = (known after apply)
      + region                      = (known after apply)
      + request_payer               = (known after apply)
      + tags_all                    = {
          + "Tag1" = "Value1"
          + "Tag2" = "Value2"
        }
      + website_domain              = (known after apply)
      + website_endpoint            = (known after apply)

      + versioning {
          + enabled    = (known after apply)
          + mfa_delete = (known after apply)
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_s3_bucket.bucket-with-default-tags-terraform: Creating...
aws_s3_bucket.bucket-with-default-tags-terraform: Creation complete after 7s [id=bucket-with-default-tags-terraform]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
~/tmp/wtf/defaultTags/terraform$ aws s3api get-bucket-tagging --bucket bucket-with-default-tags-terraform
{
    "TagSet": [
        {
            "Key": "Tag1",
            "Value": "Value1"
        },
        {
            "Key": "Tag2",
            "Value": "Value2"
        }
    ]
}
~/tmp/wtf/defaultTags/terraform$ vim main.tf
~/tmp/wtf/defaultTags/terraform$ cat main.tf
provider "aws" {
  region = "eu-west-1"
  default_tags {
    tags = {
      Tag1 = "Value1"
      Tag2 = "Value2"
      Tag3 = "Value3"
      Tag4 = "Value4"
    }
  }
}

resource "aws_s3_bucket" "bucket-with-default-tags-terraform" {
  bucket = "bucket-with-default-tags-terraform"
}
~/tmp/wtf/defaultTags/terraform$ terraform apply
aws_s3_bucket.bucket-with-default-tags-terraform: Refreshing state... [id=bucket-with-default-tags-terraform]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # aws_s3_bucket.bucket-with-default-tags-terraform has been changed
  ~ resource "aws_s3_bucket" "bucket-with-default-tags-terraform" {
        id                          = "bucket-with-default-tags-terraform"
      + tags                        = {}
        # (10 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to these changes.

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_s3_bucket.bucket-with-default-tags-terraform will be updated in-place
  ~ resource "aws_s3_bucket" "bucket-with-default-tags-terraform" {
        id                          = "bucket-with-default-tags-terraform"
        tags                        = {}
      ~ tags_all                    = {
          + "Tag3" = "Value3"
          + "Tag4" = "Value4"
            # (2 unchanged elements hidden)
        }
        # (9 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_s3_bucket.bucket-with-default-tags-terraform: Modifying... [id=bucket-with-default-tags-terraform]
aws_s3_bucket.bucket-with-default-tags-terraform: Modifications complete after 7s [id=bucket-with-default-tags-terraform]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
~/tmp/wtf/defaultTags/terraform$ aws s3api get-bucket-tagging --bucket bucket-with-default-tags-terraform
{
    "TagSet": [
        {
            "Key": "Tag1",
            "Value": "Value1"
        },
        {
            "Key": "Tag4",
            "Value": "Value4"
        },
        {
            "Key": "Tag3",
            "Value": "Value3"
        },
        {
            "Key": "Tag2",
            "Value": "Value2"
        }
    ]
}
~/tmp/wtf/defaultTags/terraform$

[leezen]

Thanks for reporting this. I tried to see if this happened with explicit providers (i.e. declaring a provider and using that) and it does as well. Only the provider is updated, but nothing flows to the underlying resources. I suspect this is related to being able to actually DiffConfig/CheckConfig (see pulumi/pulumi-terraform-bridge#244)

[filip-taxamo]

@leezen any updates on this by any chance :)?

[mwcampbell]

Anyone know of a workaround for this? I already have a stack with hundreds of resources, and I want to add a common tag to all of them. I was hoping I could use defaultTags to do this. Is there another way that doesn't require manually tagging them all?

[filip-zyzniewski]

@mwcampbell you can use something like https://www.pulumi.com/blog/automatically-enforcing-aws-resource-tagging-policies/#automatically-applying-tags .

[adam-kiss-sg]

Any update on this? Or workaround? Almost a year later this still seems to be an issue, defaultTags don't seem to be appearing in any diff at all, and it also doesn't trigger the update of resources.

@filip-zyzniewski your alternative sounds promising, but it includes maintaining a long list of resources which are taggable (https://github.com/joeduffy/aws-tags-example/blob/master/autotag-py/taggable.py). Or is there any source of those resources? (eg: somewhere in the pulumi sdk?)

[filip-zyzniewski]

@filip-zyzniewski your alternative sounds promising, but it includes maintaining a long list of resources which are taggable (https://github.com/joeduffy/aws-tags-example/blob/master/autotag-py/taggable.py).

The way we are approaching this now is:

import pulumi
import pulumi.resource as res
import typeguard


def is_taggable(resource):
    internal_init = getattr(type(resource), "_internal_init", None)
    if not internal_init:
        return False
    tags_type = internal_init.__annotations__.get("tags", None)
    if not tags_type:
        return False
    try:
        typeguard.check_type("tags", {"": ""}, tags_type)
    except TypeError:
        return False
    return True


def register_auto_tags(**auto_tags):
    def add_tags(
        args: res.ResourceTransformationArgs,
    ) -> res.ResourceTransformationResult:
        props = args.props
        if is_taggable(args.resource):
            props = dict(props)
            props["tags"] = auto_tags | (props.get("tags") or {})
        return res.ResourceTransformationResult(props=props, opts=args.opts)

    pulumi.runtime.register_stack_transformation(add_tags)

[tmehlinger]

Also running into this. A fix would be much appreciated.

[NWCalvank]

This behaviour makes defaultTags effectively useless.

Is there a clear root cause of the issue? If the Terraform provider performs as-expected then this is a Pulumi-specific issue, correct?

If anyone internal to Pulumi could point the community in the right direction here, I'm sure there might be someone (possibly me) who would be willing to help with pushing a fix.

[AaronFriel]

I'm pleased to say that the next release of the AWS provider v5 will contain a fix for this issue. Thanks all for your patience and understanding.

[jdonohoo]

Thank you for jumping on this, I was one of the enterprise customers who called out how bad this issue was impacting my organization.

[estiller]

@AaronFriel, I updated @pulumi/aws to the latest version, and indeed defaultTags were updated for most resources. However, this still doesn't work for resources created with @pulumi/awsx. Will you publish a new version of that package as well?

Is @pulumi/awsx pinned to a specific older version of @pulumi/aws?
https://github.com/pulumi/pulumi-awsx/blob/master/awsx/package.json#L24

To clarify, I have resources created with both @pulumi/aws and @pulumi/awsx in the same TypeScript stack. The resources created with @pulumi/aws picked up the new default tags as expected, but those created via @pulumi/awsx were not updated and kept the old, bad behaviour.

[AaronFriel]

@mikhailshilkov can you answer the question above?

[estiller]

@AaronFriel, @mikhailshilkov - should I open a new issue on this under the pulumi-awsx repository?
#1655 (comment)