azurerm_network_security_group_association - prevent deadlock between association and network interface creation

[nitzanm]

I believe we have found a bug in the Azure Resource Manager provider for Terraform, having to do with the azurerm_subnet_network_security_group_association resource and its interaction with azurerm_network_interface.

The issue is that the azurerm_subnet_network_security_group_association resource, when being created, locks the following resources, in the following order:

1. Network security group
2. Virtual network
3. Subnet

However, the azurerm_network_interface resource, when being created, locks the following resources, in the following order:

1. Network security group
2. Subnet
3. Virtual network

You will notice that both lock the virtual network and subnet, but in the opposite order. This means that if both resources happen to be created at the same time in two threads, the following could happen:

1. The association successfully locks the virtual network.
2. The network interface successfully locks the subnet.
3. The association now tries to lock the subnet, but it can't, because the network interface already locked it.
4. The network interface now tries to lock the virtual network, but it can't, because the association already locked it.

In this situation, the two resources will wait indefinitely for each other until Terraform is terminated - a deadlock. Our trigger for finding this bug was that Terraform would be trying to create interfaces and associations forever. Of course, this only happens intermittently, because it depends on the exact order above happening. For example (resource names redacted):

azurerm_network_interface.interface1: Still creating... [58m30s elapsed]
azurerm_network_interface.interface2: Still creating... [58m30s elapsed]
azurerm_network_interface.interface3: Still creating... [58m30s elapsed]
azurerm_subnet_network_security_group_association.association1: Still creating... [58m20s elapsed]

To fix this, we simply need to lock the three resources in the same order across both resources. As a workaround, we are currently having all network interfaces depend on our NSG association, since that will ensure both are never created at the same time.

As an aside, is there any way to review all locks across the provider, to ensure this situation doesn't happen elsewhere? Perhaps there could be a policy to always lock resources in alphabetical order so that in the future this doesn't happen again?

[katbyte]

Fantastic spot @nitzanm!

Thank you very much for this fix, LGTM and I'll review all other lock orders in the provider now.

[ghost]

This has been released in version 1.36.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 1.36.0"
}
# ... other configuration ...

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!