azurerm_cdn_frontdoor_route_disable_link_to_default_domain not working as expected - execution order issue?

[slime-uk]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.1.9

AzureRM Provider Version

3.25.0

Affected Resource(s)/Data Source(s)

azurerm_cdn_frontdoor_route_disable_link_to_default_domain

Terraform Configuration Files

resource "azurerm_resource_group" "example-rg" {
        name                            = "my-azfd-rg"
        location                        = "centralus"
}

# NOTE: Not included here but must also spin up a private link service per region using for_each on a list of regions (var.afd_regions): azurerm_private_link_service.azfd-pl-svc["region"]

resource "azurerm_cdn_frontdoor_profile" "example-afd-profile" {
	name 				= "${var.prefix}myazurefd"
	resource_group_name 		= azurerm_resource_group.example-rg.name
	sku_name 			= "Premium_AzureFrontDoor"
	response_timeout_seconds	= 240
	
	depends_on = [
		azurerm_private_link_service.azfd-pl-svc, 
	]
}

resource "azurerm_cdn_frontdoor_endpoint" "example-afd-endpoint" {
	name 				= "${var.prefix}myazurefdendpoint"
	cdn_frontdoor_profile_id 	= azurerm_cdn_frontdoor_profile.example-afd-profile.id
	enabled 			= true
}

# We need multiple origin groups - 1 per region in map var
resource "azurerm_cdn_frontdoor_origin_group" "example-afd-origingrps" {

	# Loop through all regions
	for_each = var.afd_regions

	name                     = "afd-og-${each.key}"
  	cdn_frontdoor_profile_id = azurerm_cdn_frontdoor_profile.example-afd-profile.id
  	
	session_affinity_enabled 					= false
  	restore_traffic_time_to_healed_or_new_endpoint_in_minutes 	= 10

  	load_balancing {
    	   additional_latency_in_milliseconds = 50
    	   sample_size                        = 4
    	   successful_samples_required        = 3
  	}
}

# We need multiple origins - well 1 per origin group defined above
resource "azurerm_cdn_frontdoor_origin" "example-afd-origins" {

	# Loop through all regions
	for_each = var.afd_regions

	name                          = "afd-origin-${each.key}"
	cdn_frontdoor_origin_group_id = azurerm_cdn_frontdoor_origin_group.example-afd-origingrp[each.key].id

  	enabled                        = true
  	certificate_name_check_enabled = true
  	host_name                      = trimsuffix("${azurerm_private_dns_a_record.istio-int-lb[each.key].fqdn}", ".")
  	origin_host_header             = trimsuffix("${azurerm_private_dns_a_record.istio-int-lb[each.key].fqdn}", ".")
	http_port        	       = 80
    https_port 		               = 443
  	priority                       = 5
  	weight                         = 1

  	private_link {
        	request_message        = "Private link service from AzFD TF deployment"
        	location               = each.value.region
    	        private_link_target_id = azurerm_private_link_service.azfd-pl-svc[each.key].id
  	}

	# This says that we also need a depends_on here too: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/cdn_frontdoor_origin
	depends_on = [
		azurerm_private_link_service.azfd-pl-svc, 
	]
}

# AzFD needs a route to all regions in var map
resource "azurerm_cdn_frontdoor_route" "example-afd-origingrp-routes" {
	
	# Loop through all regions
	for_each = var.afd_regions
	
	name                          	= "afd-route-${each.key}"
  	cdn_frontdoor_endpoint_id     	= azurerm_cdn_frontdoor_endpoint.example-afd-endpoint.id
  	cdn_frontdoor_origin_group_id 	= azurerm_cdn_frontdoor_origin_group.example-afd-origingrp[each.key].id
  	cdn_frontdoor_origin_ids      	= [azurerm_cdn_frontdoor_origin.example-afd-origins[each.key].id]
  	cdn_frontdoor_rule_set_ids    	= [azurerm_cdn_frontdoor_rule_set.example-afd-rule-set.id]
  	enabled                       	= true

  	forwarding_protocol    		= "MatchRequest"
  	https_redirect_enabled 		= false
  	patterns_to_match      		= var.map_afd_routes_patterns[each.key]
  	supported_protocols    		= ["Http", "Https"]
}

# Loop through all required custom domains - NOTE we do not associate to an AzFD route here but later on
resource "azurerm_cdn_frontdoor_custom_domain" "example-afd-customdomains" {
	
	# Loop around all provided domain names
	for_each = var.afd_domain_names
	
	name 	                 = replace("${each.value}.${azurerm_dns_zone.public-dnszone.name}", ".", "-")
  	cdn_frontdoor_profile_id = azurerm_cdn_frontdoor_profile.example-afd-profile.id
  	# Seems to be a bug in the later TF AzureRM providers - see: https://github.com/hashicorp/terraform-provider-azurerm/issues/18039 and https://github.com/hashicorp/terraform-provider-azurerm/issues/18554 and 
	#  so we try and workaround it?
	#dns_zone_id             = azurerm_dns_zone.public-dnszone.id
	dns_zone_id              = replace(azurerm_dns_zone.public-dnszone.id, "dnszones", "dnsZones")
  	host_name 	         = "${each.value}.${azurerm_dns_zone.public-dnszone.name}"

        # This is optional and under AzAPI we did not do this yet but later with a PATCH operation - may be simpler to leave as is!!
  	#associate_with_cdn_frontdoor_route_id = ??

  	tls {
    	   certificate_type    = "ManagedCertificate"
    	   minimum_tls_version = "TLS12"
  	}
}

# We need _dnsauth and CNAME DNS recs for the custom domains
resource "azurerm_dns_txt_record" "example-dnszone-customdoms-verification-txt-recs" {
	
	# Loop around all provided domain names
	for_each = var.afd_domain_names

  	name                = "_dnsauth.${each.value}"
  	zone_name           = azurerm_dns_zone.public-dnszone.name
  	resource_group_name = azurerm_resource_group.example-rg.name
  	ttl                 = 3600 #1 hr

	record {
		value 			= azurerm_cdn_frontdoor_custom_domain.example-afd-customdomain[each.value].validation_token
	}
}

resource "azurerm_dns_cname_record" "example-dnszone-customdoms-cname-recs" {
	
	# Loop around all provided domain names
	for_each = var.afd_domain_names

  	name                = each.value
  	zone_name           = azurerm_dns_zone.public-dnszone.name
  	resource_group_name = azurerm_resource_group.example-rg.name
  	ttl                 = 3600 #1 hr
  	record				= azurerm_cdn_frontdoor_endpoint.example-afd-endpoint.host_name

  	tags 				= var.tags-core

	depends_on = [
		azurerm_cdn_frontdoor_route.example-afd-origingrp-routes, 
		azurerm_cdn_frontdoor_security_policy.example-afd-customdomain-waf-sp, 
	]
}

resource "azurerm_cdn_frontdoor_route_disable_link_to_default_domain" "example-cd-assoc" {
	
	# Loop through all existing routes where we need the custom domain added - use new map
	for_each = var.map_afd_routes_doms

	# This is the route to now disable link to default domain
	cdn_frontdoor_route_id          = azurerm_cdn_frontdoor_route.example-afd-origingrp-routes[each.key].id
  	# And this is the list of orgs to link this route to - similar to the AzAPI patch operation we used to do
	cdn_frontdoor_custom_domain_ids = [
		for domname in each.value : "${azurerm_cdn_frontdoor_custom_domain.example-afd-customdomain[domname].id}, "
	]

	depends_on = [
		azurerm_cdn_frontdoor_route.example-afd-origingrp-routes, 
		azurerm_dns_txt_record.public-dnszone-customdoms-verification-txt-recs, 
                azurerm_cdn_frontdoor_custom_domain.example-afd-customdomain, 
	]
}

# Note: Also this example is missing a WAF security policy and firewall policy, AzFD ruleset and rules but they all work fine

Debug Output/Panic Output

TFC Error:

Error: it is invalid to disable the 'LinkToDefaultDomain' for the CDN Front Door Route (Name: route-XXX-centralus) since the route does not have any CDN Front Door Custom Domains associated with it
with azurerm_cdn_frontdoor_route_disable_link_to_default_domain.example["centralus"]
on XXX.tf line 422, in resource "azurerm_cdn_frontdoor_route_disable_link_to_default_domain" "example":
resource "azurerm_cdn_frontdoor_route_disable_link_to_default_domain" "example" {

Full Debug Log gist: https://gist.github.com/slime-uk/d767be6546f8bb9cefb85044555578c5

Expected Behaviour

It's not 100% obvious to me what this TF resource "azurerm_cdn_frontdoor_route_disable_link_to_default_domain" does but based on the fact that while we were waiting for TF to support all the newer CDN Front Door resources, we were using native TF and mostly the Azure API (AzAPI provider). In that we were creating the AzFD profile, endpoint, routes, custom domains and then patching the route (using azapi_update_resource) to only then turn off link to default domain but also add in links to new custom domains. This worked well.

Based on that I was thinking/assuming that "azurerm_cdn_frontdoor_route_disable_link_to_default_domain" does the same thing? This is certainly what I was expecting...

So, we create the AzFD endpoint, routes (linked to default AzFD endpoint), custom domains and then try and use this new resource "azurerm_cdn_frontdoor_route_disable_link_to_default_domain" to do two things:
For_each route:

1. Enable AzFD route link to new custom domain
2. Disable AzFD route link to default domain

Actual Behaviour

However, it seems Azure objects and says you can't switch off link to default domain with no custom domains defined on the route. I'd agree but isn't this resource doing this both for us? Is it simply doing the two things in an invalid order? Should it be associating the custom domain to the route first and then (if successful), disabling link to default domain?

It seems to be doing it in this (invalid?)P order:

For_each route:

1. Disable the link to default domain
2. Enable link to new custom domain (never gets here as Azure errors on step 1)

Steps to Reproduce

You will need some additional variables defined:

A string for the env prefix:
var.prefix = "sbx" for example, or "dev".

A set(string) of AzFD routes/regions to create as we want a single AzFD with multiple routes to AKS clusters in different regions
var.afd_regions = ["centralus", "westeurope"]

A set(string) of AzFD custom domains to create:
var.afd_domain_names = ["domain1", "domain2"]

A map(set(string)) of AzFD patterns to match for each route:
var.map_afd_routes_patterns = {
"centralus" = ["/example-cus/"]
"westeurope" = ["/example-we/"]
}

And a corresponding map of sets which maps required regions to AzFD routes per custom domains
var.map_afd_routes_doms = {
"centralus" = ["domain1", "domain2"]
"westeurope" = ["domain1"]
}

Create a PLS per region, PEs, istio load balancers and then then AzFD as per configuration.

Then TF plan (successful) and then apply.

Important Factoids

No response

References

No response

[WodansSon]

@slime-uk, thank you for opening this issue. This resource is very difficult to shoehorn into Terraform because the API and Terraform are diametrically opposed to how they work. I understand your frustration with the azurerm_cdn_frontdoor_route_disable_link_to_default_domain resource... it was my first shot at trying to fix this issue, and it was bad... lol... sorry. The reason I created the azurerm_cdn_frontdoor_route_disable_link_to_default_domain resource was to work around the issue of the Route being required to have an associated "domain" for it's creation by the API. Which meant that the Route either had to be associated via the LinkToDefaultDomain field or by a pre-existing Custom Domain . The issue here is the old chicken or the egg story, and to get the creation/destruction order correct you had to expose a cyclic reference between the Custom Domain resource and the Route resource which the Terraform Core runtime would detect. To make the problem even more difficult, you can not delete the Custom Domain, via the API, if the Custom Domain is associated with one or more Routes. The fact that the API has this as a requirement for destruction of the Custom Domain is already at odds with the way Terraform works. So that is where the azurerm_cdn_frontdoor_route_disable_link_to_default_domain resource comes from. The, more or less, hack to get this to work in Terraform, was to always create the Route with LinkToDefaultDomain as true and expose a virtual association resource that controlled that value in the Route which could then be ordered by a dependency within the configuration file. It was confusing and not very intuitive, I understand that, but it did work, but was not a very well thought out solution to the issue... my bad. I have spent a lot of time thinking about this issue and came up with a totally new solution that I think is way better and more closely mirrors the experience in the portal. In my new PR I have now moved the link_to_default_domain and the cdn_frontdoor_custom_domain_ids fields back into the route resource. Totally deprecated the azurerm_cdn_frontdoor_route_disable_link_to_default_domain resource and then exposed a new azurerm_cdn_frontdoor_custom_domain_association resource that semi-manages the Routes cdn_frontdoor_custom_domain_ids field. It is still sorta wonky, but given the way the API works and how Terraform creates its dependency graphs I feel this is the best happy medium we could come up with at this point. That said, I am talking with the service team about making the Custom Domain automatically remove itself from the Routes association upon deletion of the Custom Domain which is the root cause of why all of this came about. Sorry if this is TL;DR, but I thought you should know why this implementation was so weird. 🙂

[slime-uk]

@WodansSon Thank you for the detailed reply and explanation which I do sort of follow! Also, many thanks for all the hard work getting the new AzFD resources into TF AzureRM provider! Many many thanks.

FYI - For now, I have removed the "azurerm_cdn_frontdoor_route_disable_link_to_default_domain" resource from our TF configuration and have restored our AzAPI implementation which after all the above code then "patches" the existing (TF created now) AzFD routes and does the new associations to the custom domains and disables the links to default domain - this is working very well.

resource "azapi_update_resource" "example-afd-origingrp-routes-cd-assoc" {
	
        # Loop through all existing routes where we need the custom domain added - use new map
	for_each = var.map_afd_routes_doms
    
        type 		= "Microsoft.Cdn/profiles/afdEndpoints/routes@2021-06-01"
	resource_id 	= azurerm_cdn_frontdoor_route.example-afd-origingrp-routes[each.key].id
	
  	body = jsonencode({
    	   properties = {
		customDomains = [
			for domname in each.value : {
				id = azurerm_cdn_frontdoor_custom_domain.example-afd-customdomains[domname].id
			}
		]
		linkToDefaultDomain = "Disabled"
           }
  	})

 	depends_on = [
		azurerm_cdn_frontdoor_route.example-afd-origingrp-routes, 
		azurerm_dns_txt_record.public-dnszone-customdoms-verification-txt-recs,
		azurerm_cdn_frontdoor_customdomain.example-afd-customdomains, 
  	]
}

Again, many thanks!

[github-actions]

This functionality has been released in v3.27.0 of the Terraform Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[slime-uk]

Hi @WodansSon

After seeing and reading your latest PR, and v3.27.0 provider change log, I removed our last remaining AzAPI resource and added in new resource "azurerm_cdn_frontdoor_custom_domain_association" "example-afd-origingrp-routes-cd-assoc" to manage the custom domain to route associations.

I uplifted to 3.27.0 and changed my map of routes to custom domains, to be custom domains to AzFD routes:

var.map_afd_doms_routes = {
"domain1" = ["centralus", "westeurope"]
"domain2" = ["centralus"]
}

and also added this new resource block code:

resource "azurerm_cdn_frontdoor_custom_domain_association" "example-afd-origingrp-routes-cd-assoc"

   # Loop around NEW map (map-afd_doms_routes) of all domains and then for each, loop around all required routes for that custom domain
   for_each = var.map_afd_doms_routes 

   cdn_frontdoor_custom_domain_id = azurerm_cdn_frontdoor_route.example-afd-origingrp-routes[each.key].id
   cdn_frontdoor_route_ids = [
      for route_index in each.value : azurerm_cdn_frontdoor_route.example-afd-origingrp-routes[route_index].id
   ]
}

On TFC plan, I am getting this crash in 3.27.0 provider:

Stack trace from the terraform-provider-azurerm_v3.27.0_x5 plugin:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x46407c5]

goroutine 6728 [running]:
github.com/hashicorp/terraform-provider-azurerm/internal/services/cdn.flattenRuleSetResourceArray(0xc001faf878)
github.com/hashicorp/terraform-provider-azurerm/internal/services/cdn/cdn_frontdoor_route_resource.go:617 +0xc5
github.com/hashicorp/terraform-provider-azurerm/internal/services/cdn.resourceCdnFrontDoorRouteRead(0xc003281a00, {0x590f740?, 0xc000591c00?})
github.com/hashicorp/terraform-provider-azurerm/internal/services/cdn/cdn_frontdoor_route_resource.go:375 +0xab4
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0x6a838e8?, {0x6a838e8?, 0xc00328b4a0?}, 0xd?, {0x590f740?, 0xc000591c00?})
github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:712 +0x178
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc000981a40, {0x6a838e8, 0xc00328b4a0}, 0xc00036f860, {0x590f740, 0xc000591c00})
github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:1015 +0x585
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0xc00050bb60, {0x6a838e8?, 0xc00328b3b0?}, 0xc001afc680)
github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/grpc_provider.go:613 +0x497
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ReadResource(0xc001580be0, {0x6a838e8?, 0xc00328ae70?}, 0xc0029ff080)
github.com/hashicorp/[email protected]/tfprotov5/tf5server/server.go:747 +0x41e
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler({0x5f0d6c0?, 0xc001580be0}, {0x6a838e8, 0xc00328ae70}, 0xc0029ff020, 0x0)
github.com/hashicorp/[email protected]/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:349 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0000eaa80, {0x6a95540, 0xc00035e1a0}, 0xc00328d440, 0xc0016bed20, 0xab7fa10, 0x0)
google.golang.org/[email protected]/server.go:1283 +0xcfd
google.golang.org/grpc.(*Server).handleStream(0xc0000eaa80, {0x6a95540, 0xc00035e1a0}, 0xc00328d440, 0x0)
google.golang.org/[email protected]/server.go:1620 +0xa1b
google.golang.org/grpc.(*Server).serveStreams.func1.2()
google.golang.org/[email protected]/server.go:922 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
google.golang.org/[email protected]/server.go:920 +0x28a

Error: The terraform-provider-azurerm_v3.27.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

The errors reported after that in TFC are:

Error: Plugin did not respond
with azurerm_cdn_frontdoor_route.exampl-afd-origingrp-routes["centralus"]
on XXX.tf line 133, in resource "azurerm_cdn_frontdoor_route" "example-afd-origingrp-routes":
resource "azurerm_cdn_frontdoor_route" "example-afd-origingrp-routes" {
The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The plugin logs may contain more details.

When I create the AzFD route, I do not associate to any custom domains at that time, but I now say link to default = false.

snippet of route resource block:

   # 14/10/22 - Add in v 3.27.0 of the provider if it will allow and we will now associate with new resource later (resource "azurerm_cdn_frontdoor_custom_domain_association")
	link_to_default_domain          = false
	# And this is new but optional in 3.27.0 - so try to setup routes with no associations - not even to default domain, then setup associations later using new resource in 3.27.0
	#cdn_frontdoor_custom_domain_ids = ??

Any ideas? Is it because I've been trying other ways to solve this? Shall I tear the AzFD down completely and re-plan? Any thoughts? Thanks!

[slime-uk]

An update - tearing AzFD down and re-planning did indeed work nicely - however the provider 3.27.0 is not working as expected.

So, I now associate the routes to custom domains and also now using the new resource azurerm_cdn_frontdoor_custom_domain_association also associate the custom domains to the routes and although the plan and apply works, when I later remove a custom domain from our configuration, terraform destroys the new resource azurerm_cdn_frontdoor_custom_domain_association OK first (and looking in the portal using FD Manager, the association appears gone) but Terraform does not do anything to the route (so it must still be associated to the custom domain), but then immediately tries to delete the custom domain and Azure objects "Error: waiting for the deletion of Front Door Custom Domain: (XXX / Profile Name "YYY" etc.): Code="BadRequest" Message=ErrorMessage Host with Id: XX for tenant Id: YYY is still referenced by partners: ZZZZ"

FYI - I have also tried to only define the AzFD routes and link to default domains and NOT associate with any custom domains but then later use new resource azurerm_cdn_frontdoor_custom_domain_association - that fails saying we can't do that when the routes are not currently associated with the custom domains so "please remove the CDN FrontDoor Route from your 'cdn_frontdoor_custom_domain_association' configuration block". I also tried adding back in routes, linked to default domain AND also also associating to custom domains but also using new resource as well - again the creation plan/apply works but not the deletion of a custom domain as again it's still associated to a route.

Ideally we'd like to (our use case) create the AzFD profile, origins, origin groups, routes (linked only to default domain if we must but prefer not appreciate the Azure portal doesn't allow this - routes must be associated to at least 1 domain), then create custom domains as and when and part of that process would also need to associate to existing routes as we define. Ideally we'd like to be able to also disable link_to_default_domain as and when too. We'd like to then also be able to remove a custom domain should we need to of course without error or needing to do 2 Terraform plans and applies in stages - which is the only workaround we have right now).

[slime-uk]

Update - the deletion worked but it seems ONLY if when I create the routes I do also set link_to_default_domain = true in the route configuration as well as associate to all custom domains at that time, and also use the new resource azurerm_cdn_frontdoor_custom_domain_association to also set the associations.

This is so close - my only ask (please!) is why must we have a route associated to default domain for a custom domain to be deleted later successfully in one pass?

[slime-uk]

Opened a new issue as requested! #18844

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.