Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

azurerm remote state backend fails to initialize properly when snapshot=true and lock=false #27771

Open
dz-sourced opened this issue Feb 14, 2021 · 0 comments · May be fixed by #27797
Open

azurerm remote state backend fails to initialize properly when snapshot=true and lock=false #27771

dz-sourced opened this issue Feb 14, 2021 · 0 comments · May be fixed by #27797

Comments

@dz-sourced
Copy link

Under certain circumstances azurerm remote state provider fails to create an initial empty state and crashes during
subsequent operations. When using azurerm remote state provider with the snapshot=true option and running terraform apply
command with -lock=false option remote state doesn't create an initial empty state file in the Storage Account and fails
trying to snapshot a blob that doesn't exist. While we don't usually use remote state without locking, some tools, notably Terratest
sets -lock=false explicitly.

Terraform Version

Terraform v0.14.4

Terraform Configuration Files

terraform {
  backend "azurerm" {
    resource_group_name  = "test-azurerm-storage-rg"
    storage_account_name = "sometestaccount"
    container_name       = "tfstate"
    key                  = "terraform.tfstate"
    snapshot             = true
  }
}

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "rg" {
  name     = "test-rg"
  location = "canadacentral"
}

Expected Behavior

Terraform runs an apply command and saves state file to the Azure Blob Storage Account

Actual Behavior

Terraform fails to snapshot a file that doesn't exist and fails:

Failed to save state: error snapshotting Blob "terraform.tfstate" (Container "tfstate" / Account "sometestaccount"):
blobs.Client#Snapshot: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service retur
ned an error. Status=404 Code="BlobNotFound" Message="The specified blob does not exist.\nRequestId:xxx-yyy-zzz

Steps to Reproduce

  1. Create a TF configuration as shown above. Actual resources doesn't really matter. Make sure backend option snapshot is set to true
  2. Run terraform init
  3. Run terraform apply -lock=false

Additional Context

I was able to track down this issue to this line

if name != backend.DefaultStateName {

For some reason in the azure remote state implementation a new empty blob is only created in the Backend.StateMgr() when
state name does not equal the default name. There is another place where an empty blob is being created:
in the RemoteClient.Lock() function
properties, err := c.giovanniBlobClient.GetProperties(ctx, c.accountName, c.containerName, c.keyName, blobs.GetPropertiesInput{})
but this code is never executed when you set lock=false
Finally when a RemoteClient.Put() is called a snapshot is attempted, but because of the issues described above, no
blob is found

While this sounds like a rare combination of options we did run into this problem when using Terratest in combination with Terragrunt. I also think azurerm remote state code can benefit from a bit of a cleanup to eliminate multiple places where
a new empty state is created. For example in the S3 backend implementation a new state is created in the Backend.StateMgr function

I'm currently working on a PR to fix this issue in the azurerm backend by forcing a new blob to be created in the Backend.StateMgr function regardless of the state name and also potentially removing new blob creation code from the RemoteClient.Lock() function to avoid confusion. Let me know if this is not the right direction, e.g. if there is not guarantee
Backend.StateMgr is called before any other state operation. It doesn't seem to be the case, but I didn't spend enough time
with this codebase yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants