Refactor Resource Tagging Code to use keyvaluetags Package

[bflad]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Since its inception, the Terraform AWS Provider has supported resource tagging on various Terraform resources. While in theory most resource tagging looks similar to practitioners (a simple key-value mapping), in practice there are underlying differences between how each AWS service API implements tagging support. Another challenge within this space is that the AWS Go SDK implementation itself can differ (e.g. whether a key-value map or struct containing a Key and Value field) and that each AWS service has its own Go types in the latter implementation.

The internal keyvaluetags package was introduced into this codebase to provide a consistent interface for handling AWS resource key-value tags. Using this package simplifies the process of adding support for resource tagging and will allow for much easier implementation of provider-wide enhancements for tagging support, such as the ability to specify default tags and tags to ignore for all resources under a provider (e.g. #7926). To sustainably support those enhancements, a refactoring of the existing resources to use the new package needs to occur.

The existing tagging code across AWS services in the Terraform AWS Provider varies on a few dimensions:

• Whether the service uses map[string]*string versus an AWS Go SDK specific type (e.g. []*eks.Tag)
• Whether the resource supports tagging on creation or a separate API call after creation
• Whether the resource supports tags as part of reading the resource or requires a separate API call
• Whether the tagging logic is directly within resource functions, in separate functions in the resource file, or separate functions in a service-based aws/tagsXXX.go file

Each of these will slightly vary the refactoring instructions, however in principle the highest level of the refactoring process is:

• Per data source, switching the implementation in the Read function to using the keyvaluetags package
• Per resource, switching the implementations in the Create, Read, and Update functions to using the keyvaluetags package
• Per service, removing any aws/tagsXXX.go and associated aws/tagsXXX_test.go files

To prevent lengthy review processes and overlapping contributions, this refactoring should preferably be performed per resource for AWS services that support tagging across more than (roughly) three datasources/resources or can be done at once for an entire AWS service with a smaller implementation. Any functionality missing from the keyvaluetags library should preferably be submitted separately.

Below is additional technical information to help guide this refactoring process and the refactoring checklist.

Refactoring Process

In the all cases, the data source or resource Go file (e.g. aws/resource_aws_eks_cluster.go) will need a new import:

"github.com/terraform-providers/terraform-provider-aws/aws/internal/keyvaluetags"

Data Source or Resource Read Function

If the tags are returned in the API response for retrieving the data source or resource, you may see code like the following:

if err := d.Set("tags", tagsToMap(vpc.Tags)); err != nil {
	return fmt.Errorf("error setting tags: %s", err)
}

This can be replaced similar to the below (where Ec2 should be replaced with the relevant service name):

if err := d.Set("tags", keyvaluetags.Ec2KeyValueTags(vpc.Tags).IgnoreAws().Map()); err != nil {
	return fmt.Errorf("error setting tags: %s", err)
}

Otherwise, if the tags are returned by making a separate API call, you may see code like the following:

// Service using a helper function to list tags and call d.Set("tags", ...)
if err := getTagsECR(conn, d); err != nil {
	return fmt.Errorf("error getting ECR repository tags: %s", err)
}

// Service not using a helper function
tagsResp, err := conn.ListTagsForResource(&sfn.ListTagsForResourceInput{
	ResourceArn: aws.String(d.Id()),
})

if err := d.Set("tags", tagsToMapSfn(tagsResp.Tags)); err != nil {
	return fmt.Errorf("error setting tags: %s", err)
}

Both of these can be replaced similar to the below (where Ecr should be replaced with the relevant service name and arn generally containing the resource ARN or ID used to fetch the tags):

tags, err := keyvaluetags.EcrListTags(conn, arn)

if err != nil {
	return fmt.Errorf("error listing tags for ECR Repository (%s): %s", arn, err)
}

if err := d.Set("tags", tags.IgnoreAws().Map()); err != nil {
	return fmt.Errorf("error setting tags: %s", err)
}

Resource Create Function

If the resource supports tagging on creation, you may see code like the following in the input:

Tags: tagsFromMapECR(d.Get("tags").(map[string]interface{})),

This can be replaced similar to the below (where Ecr should be replaced with the relevant service name):

Tags: keyvaluetags.New(d.Get("tags").(map[string]interface{})).IgnoreAws().EcrTags(),

Otherwise, if the resource requires a separate API call to tag a resource after recreation, it can be replaced similar to the below (where Ec2 should be replaced with the relevant service name and d.Id() replaced with the appropriate resource ARN or ID to update the tags):

if v := d.Get("tags").(map[string]interface{}); len(v) > 0 {
	if err := keyvaluetags.Ec2UpdateTags(conn, d.Id(), nil, v); err != nil {
		return fmt.Errorf("error adding tags: %s", err)
	}
}

Resource Update Function

You may see code like the following (with or without the if d.HasChange("tags") conditional):

// Service using a helper function to fully update tags
if d.HasChange("tags") {
	if err := setTagsECR(conn, d); err != nil {
		return fmt.Errorf("error setting ECR repository tags: %s", err)
	}
}

// Service using a helper function only to diff tags
if d.HasChange("tags") {
	oldTagsRaw, newTagsRaw := d.GetChange("tags")
	oldTagsMap := oldTagsRaw.(map[string]interface{})
	newTagsMap := newTagsRaw.(map[string]interface{})
	createTags, removeTags := diffTagsSfn(tagsFromMapSfn(oldTagsMap), tagsFromMapSfn(newTagsMap))

	if len(removeTags) > 0 {
		removeTagKeys := make([]*string, len(removeTags))
		for i, removeTag := range removeTags {
			removeTagKeys[i] = removeTag.Key
		}

		input := &sfn.UntagResourceInput{
			ResourceArn: aws.String(d.Id()),
			TagKeys:     removeTagKeys,
		}

		log.Printf("[DEBUG] Untagging State Function: %s", input)
		if _, err := conn.UntagResource(input); err != nil {
			return fmt.Errorf("error untagging State Function (%s): %s", d.Id(), err)
		}
	}

	if len(createTags) > 0 {
		input := &sfn.TagResourceInput{
			ResourceArn: aws.String(d.Id()),
			Tags:        createTags,
		}

		log.Printf("[DEBUG] Tagging State Function: %s", input)
		if _, err := conn.TagResource(input); err != nil {
			return fmt.Errorf("error tagging State Function (%s): %s", d.Id(), err)
		}
	}
}

Both of these can be replaced similar to the below (where Ecr should be replaced with the relevant service name, arn generally containing the resource ARN or ID used to update the tags, and adding the if d.HasChange("tags") { if missing in the update function):

if d.HasChange("tags") {
	o, n := d.GetChange("tags")

	if err := keyvaluetags.EcrUpdateTags(conn, arn, o, n); err != nil {
		return fmt.Errorf("error updating ECR Repository (%s) tags: %s", arn, err)
	}
}

Removing Service Tagging Files

Once the codebase is no longer using any functions from them, any relevant aws/tags{SERVICE}.go and aws/tags{SERVICE}_test.go files can be removed. If these unused functions are left in the codebase during a pull request, the pull request testing will flag these. You can run make lint locally to check for unused functions before submission, if you desire.

Additional References

If you would like to contribute, but are stuck in the details, these references may also help:

• The Contributing Guide section on Adding Resource Tagging Support includes specific code examples on how to accomplish Create, Read, and Update functionality with the keyvaluetags package.
• Example data source refactor where tags are included in API response
• Example data source refactor where tags require separate API call
• Example resource refactor

Definition of Done

• aws/data_source_aws_acmpca_certificate_authority.go
• aws/data_source_aws_ami.go
• aws/data_source_aws_cloudformation_stack.go
• aws/data_source_aws_customer_gateway.go
• aws/data_source_aws_db_cluster_snapshot.go
• aws/data_source_aws_db_instance.go
• aws/data_source_aws_dynamodb_table.go
• aws/data_source_aws_ebs_snapshot.go
• aws/data_source_aws_ebs_volume.go
• aws/data_source_aws_ec2_transit_gateway.go
• aws/data_source_aws_ec2_transit_gateway_dx_gateway_attachment.go
• aws/data_source_aws_ec2_transit_gateway_route_table.go
• aws/data_source_aws_ec2_transit_gateway_vpc_attachment.go
• aws/data_source_aws_ec2_transit_gateway_vpn_attachment.go
• aws/data_source_aws_ecr_repository.go
• aws/data_source_aws_efs_file_system.go
• aws/data_source_aws_eip.go
• aws/data_source_aws_eks_cluster.go
• aws/data_source_aws_elasticache_cluster.go
• aws/data_source_aws_elasticsearch_domain.go
• aws/data_source_aws_elb.go
• aws/data_source_aws_instance.go
• aws/data_source_aws_internet_gateway.go
• aws/data_source_aws_kinesis_stream.go
• aws/data_source_aws_lambda_function.go
• aws/data_source_aws_launch_template.go
• aws/data_source_aws_lb.go
• aws/data_source_aws_lb_target_group.go
• aws/data_source_aws_mq_broker.go
• aws/data_source_aws_msk_cluster.go
• aws/data_source_aws_nat_gateway.go
• aws/data_source_aws_network_acls.go
• aws/data_source_aws_network_interface.go
• aws/data_source_aws_network_interfaces.go
• aws/data_source_aws_ram_resource_share.go
• aws/data_source_aws_rds_cluster.go
• aws/data_source_aws_redshift_cluster.go
• aws/data_source_aws_route53_resolver_rule.go
• aws/data_source_aws_route53_zone.go
• aws/data_source_aws_route_table.go
• aws/data_source_aws_route_tables.go
• aws/data_source_aws_s3_bucket_object.go
• aws/data_source_aws_secretsmanager_secret.go
• aws/data_source_aws_security_group.go
• aws/data_source_aws_security_groups.go
• aws/data_source_aws_subnet.go
• aws/data_source_aws_subnet_ids.go
• aws/data_source_aws_vpc.go
• aws/data_source_aws_vpc_dhcp_options.go
• aws/data_source_aws_vpc_endpoint.go
• aws/data_source_aws_vpc_endpoint_service.go
• aws/data_source_aws_vpc_peering_connection.go
• aws/data_source_aws_vpcs.go
• aws/data_source_aws_vpn_gateway.go
• aws/resource_aws_acm_certificate.go
• aws/resource_aws_acmpca_certificate_authority.go
• aws/resource_aws_ami.go
• aws/resource_aws_ami_copy.go
• aws/resource_aws_ami_from_instance.go
• aws/resource_aws_api_gateway_api_key.go
• aws/resource_aws_api_gateway_client_certificate.go
• aws/resource_aws_api_gateway_domain_name.go
• aws/resource_aws_api_gateway_rest_api.go
• aws/resource_aws_api_gateway_stage.go
• aws/resource_aws_api_gateway_usage_plan.go
• aws/resource_aws_api_gateway_vpc_link.go
• aws/resource_aws_appmesh_mesh.go
• aws/resource_aws_appmesh_route.go
• aws/resource_aws_appmesh_virtual_node.go
• aws/resource_aws_appmesh_virtual_router.go
• aws/resource_aws_appmesh_virtual_service.go
• aws/resource_aws_appsync_graphql_api.go
• aws/resource_aws_athena_workgroup.go
• aws/resource_aws_autoscaling_group.go (skip for now)
• aws/resource_aws_backup_plan.go
• aws/resource_aws_backup_vault.go
• aws/resource_aws_batch_compute_environment.go
• aws/resource_aws_cloudformation_stack.go
• aws/resource_aws_cloudformation_stack_set.go
• aws/resource_aws_cloudfront_distribution.go
• aws/resource_aws_cloudhsm2_cluster.go
• aws/resource_aws_cloudtrail.go
• aws/resource_aws_cloudwatch_event_rule.go
• aws/resource_aws_cloudwatch_log_group.go
• aws/resource_aws_cloudwatch_metric_alarm.go
• aws/resource_aws_codebuild_project.go
• aws/resource_aws_codecommit_repository.go
• aws/resource_aws_codepipeline.go
• aws/resource_aws_codepipeline_webhook.go
• aws/resource_aws_cognito_identity_pool.go
• aws/resource_aws_cognito_user_pool.go
• aws/resource_aws_config_aggregate_authorization.go
• aws/resource_aws_config_config_rule.go
• aws/resource_aws_config_configuration_aggregator.go
• aws/resource_aws_customer_gateway.go
• aws/resource_aws_datapipeline_pipeline.go
• aws/resource_aws_datasync_agent.go
• aws/resource_aws_datasync_location_efs.go
• aws/resource_aws_datasync_location_nfs.go
• aws/resource_aws_datasync_location_s3.go
• aws/resource_aws_datasync_task.go
• aws/resource_aws_dax_cluster.go
• aws/resource_aws_db_cluster_snapshot.go
• aws/resource_aws_db_event_subscription.go
• aws/resource_aws_db_instance.go
• aws/resource_aws_db_option_group.go
• aws/resource_aws_db_parameter_group.go
• aws/resource_aws_db_security_group.go
• aws/resource_aws_db_snapshot.go
• aws/resource_aws_db_subnet_group.go
• aws/resource_aws_default_network_acl.go
• aws/resource_aws_default_route_table.go
• aws/resource_aws_directory_service_directory.go
• aws/resource_aws_dms_endpoint.go
• aws/resource_aws_dms_replication_instance.go
• aws/resource_aws_dms_replication_subnet_group.go
• aws/resource_aws_dms_replication_task.go
• aws/resource_aws_docdb_cluster.go
• aws/resource_aws_docdb_cluster_instance.go
• aws/resource_aws_docdb_cluster_parameter_group.go
• aws/resource_aws_docdb_subnet_group.go
• aws/resource_aws_dx_connection.go
• aws/resource_aws_dx_hosted_private_virtual_interface_accepter.go
• aws/resource_aws_dx_hosted_public_virtual_interface_accepter.go
• aws/resource_aws_dx_lag.go
• aws/resource_aws_dx_private_virtual_interface.go
• aws/resource_aws_dx_public_virtual_interface.go
• aws/resource_aws_dx_transit_virtual_interface.go
• aws/resource_aws_dynamodb_table.go
• aws/resource_aws_ebs_snapshot.go
• aws/resource_aws_ebs_snapshot_copy.go
• aws/resource_aws_ebs_volume.go
• aws/resource_aws_ec2_capacity_reservation.go
• aws/resource_aws_ec2_client_vpn_endpoint.go
• aws/resource_aws_ec2_fleet.go
• aws/resource_aws_ec2_transit_gateway.go
• aws/resource_aws_ec2_transit_gateway_route_table.go
• aws/resource_aws_ec2_transit_gateway_vpc_attachment.go
• aws/resource_aws_ec2_transit_gateway_vpc_attachment_accepter.go
• aws/resource_aws_ecr_repository.go
• aws/resource_aws_ecs_cluster.go
• aws/resource_aws_ecs_service.go
• aws/resource_aws_ecs_task_definition.go
• aws/resource_aws_efs_file_system.go
• aws/resource_aws_eip.go
• aws/resource_aws_eks_cluster.go
• aws/resource_aws_elastic_beanstalk_application.go
• aws/resource_aws_elastic_beanstalk_application_version.go
• aws/resource_aws_elastic_beanstalk_environment.go
• aws/resource_aws_elasticache_cluster.go
• aws/resource_aws_elasticache_replication_group.go
• aws/resource_aws_elasticsearch_domain.go
• aws/resource_aws_elb.go
• aws/resource_aws_emr_cluster.go
• aws/resource_aws_fsx_lustre_file_system.go
• aws/resource_aws_fsx_windows_file_system.go
• aws/resource_aws_glacier_vault.go
• aws/resource_aws_glue_crawler.go
• aws/resource_aws_glue_job.go
• aws/resource_aws_glue_trigger.go
• aws/resource_aws_iam_role.go
• aws/resource_aws_iam_user.go
• aws/resource_aws_inspector_resource_group.go
• aws/resource_aws_instance.go
• aws/resource_aws_internet_gateway.go
• aws/resource_aws_kinesis_analytics_application.go
• aws/resource_aws_kinesis_firehose_delivery_stream.go
• aws/resource_aws_kinesis_stream.go
• aws/resource_aws_kms_external_key.go
• aws/resource_aws_kms_key.go
• aws/resource_aws_lambda_function.go
• aws/resource_aws_launch_template.go
• aws/resource_aws_lb.go
• aws/resource_aws_lb_target_group.go
• aws/resource_aws_licensemanager_license_configuration.go
• aws/resource_aws_lightsail_instance.go
• aws/resource_aws_media_package_channel.go
• aws/resource_aws_media_store_container.go
• aws/resource_aws_mq_broker.go
• aws/resource_aws_mq_configuration.go
• aws/resource_aws_msk_cluster.go
• aws/resource_aws_nat_gateway.go
• aws/resource_aws_neptune_cluster.go
• aws/resource_aws_neptune_cluster_instance.go
• aws/resource_aws_neptune_cluster_parameter_group.go
• aws/resource_aws_neptune_event_subscription.go
• aws/resource_aws_neptune_parameter_group.go
• aws/resource_aws_neptune_subnet_group.go
• aws/resource_aws_network_acl.go
• aws/resource_aws_network_interface.go
• aws/resource_aws_opsworks_stack.go
• aws/resource_aws_organizations_account.go
• aws/resource_aws_pinpoint_app.go
• aws/resource_aws_ram_resource_share.go
• aws/resource_aws_rds_cluster.go
• aws/resource_aws_rds_cluster_instance.go
• aws/resource_aws_rds_cluster_parameter_group.go
• aws/resource_aws_redshift_cluster.go
• aws/resource_aws_redshift_event_subscription.go
• aws/resource_aws_redshift_parameter_group.go
• aws/resource_aws_redshift_snapshot_copy_grant.go
• aws/resource_aws_redshift_snapshot_schedule.go
• aws/resource_aws_redshift_subnet_group.go
• aws/resource_aws_route53_health_check.go
• aws/resource_aws_route53_resolver_endpoint.go
• aws/resource_aws_route53_resolver_rule.go
• aws/resource_aws_route53_zone.go
• aws/resource_aws_route_table.go
• aws/resource_aws_s3_bucket.go
• aws/resource_aws_s3_bucket_metric.go
• aws/resource_aws_s3_bucket_object.go
• aws/resource_aws_sagemaker_endpoint.go
• aws/resource_aws_sagemaker_endpoint_configuration.go
• aws/resource_aws_sagemaker_model.go
• aws/resource_aws_sagemaker_notebook_instance.go
• aws/resource_aws_secretsmanager_secret.go
• aws/resource_aws_security_group.go
• aws/resource_aws_servicecatalog_portfolio.go
• aws/resource_aws_sfn_activity.go
• aws/resource_aws_sfn_state_machine.go
• aws/resource_aws_sns_topic.go
• aws/resource_aws_spot_fleet_request.go
• aws/resource_aws_spot_instance_request.go
• aws/resource_aws_sqs_queue.go
• aws/resource_aws_ssm_activation.go
• aws/resource_aws_ssm_document.go
• aws/resource_aws_ssm_maintenance_window.go
• aws/resource_aws_ssm_parameter.go
• aws/resource_aws_ssm_patch_baseline.go
• aws/resource_aws_storagegateway_cached_iscsi_volume.go
• aws/resource_aws_storagegateway_gateway.go
• aws/resource_aws_storagegateway_nfs_file_share.go
• aws/resource_aws_storagegateway_smb_file_share.go
• aws/resource_aws_subnet.go
• aws/resource_aws_swf_domain.go
• aws/resource_aws_transfer_server.go
• aws/resource_aws_transfer_user.go
• aws/resource_aws_vpc.go
• aws/resource_aws_vpc_dhcp_options.go
• aws/resource_aws_vpc_endpoint.go
• aws/resource_aws_vpc_endpoint_service.go
• aws/resource_aws_vpc_peering_connection.go
• aws/resource_aws_vpc_peering_connection_accepter.go
• aws/resource_aws_vpn_connection.go
• aws/resource_aws_vpn_gateway.go
• aws/resource_aws_waf_rate_based_rule.go
• aws/resource_aws_waf_rule.go
• aws/resource_aws_waf_rule_group.go
• aws/resource_aws_waf_web_acl.go
• aws/resource_aws_wafregional_rate_based_rule.go
• aws/resource_aws_wafregional_rule.go
• aws/resource_aws_wafregional_rule_group.go
• aws/resource_aws_wafregional_web_acl.go

[ewbankkit]

@bflad Should we mark the S3 data sources and resources as skip for now as S3 doesn't have keyvaluetags implementation because of its unique Put not Add and Delete implementation?

[ghost]

This has been released in version 2.53.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!