You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Under certain circumstances azurerm remote state provider fails to create an initial empty state and crashes during
subsequent operations. When using azurerm remote state provider with the snapshot=true option and running terraform apply
command with -lock=false option remote state doesn't create an initial empty state file in the Storage Account and fails
trying to snapshot a blob that doesn't exist. While we don't usually use remote state without locking, some tools, notably Terratest
sets -lock=false explicitly.
Terraform runs an apply command and saves state file to the Azure Blob Storage Account
Actual Behavior
Terraform fails to snapshot a file that doesn't exist and fails:
Failed to save state: error snapshotting Blob "terraform.tfstate" (Container "tfstate" / Account "sometestaccount"):
blobs.Client#Snapshot: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service retur
ned an error. Status=404 Code="BlobNotFound" Message="The specified blob does not exist.\nRequestId:xxx-yyy-zzz
Steps to Reproduce
Create a TF configuration as shown above. Actual resources doesn't really matter. Make sure backend option snapshot is set to true
For some reason in the azure remote state implementation a new empty blob is only created in the Backend.StateMgr() when
state name does not equal the default name. There is another place where an empty blob is being created:
in the RemoteClient.Lock() function
but this code is never executed when you set lock=false
Finally when a RemoteClient.Put() is called a snapshot is attempted, but because of the issues described above, no
blob is found
While this sounds like a rare combination of options we did run into this problem when using Terratest in combination with Terragrunt. I also think azurerm remote state code can benefit from a bit of a cleanup to eliminate multiple places where
a new empty state is created. For example in the S3 backend implementation a new state is created in the Backend.StateMgr function
I'm currently working on a PR to fix this issue in the azurerm backend by forcing a new blob to be created in the Backend.StateMgr function regardless of the state name and also potentially removing new blob creation code from the RemoteClient.Lock() function to avoid confusion. Let me know if this is not the right direction, e.g. if there is not guarantee Backend.StateMgr is called before any other state operation. It doesn't seem to be the case, but I didn't spend enough time
with this codebase yet.
The text was updated successfully, but these errors were encountered:
Under certain circumstances
azurerm
remote state provider fails to create an initial empty state and crashes duringsubsequent operations. When using
azurerm
remote state provider with thesnapshot=true
option and runningterraform apply
command with
-lock=false
option remote state doesn't create an initial empty state file in the Storage Account and failstrying to snapshot a blob that doesn't exist. While we don't usually use remote state without locking, some tools, notably Terratest
sets
-lock=false
explicitly.Terraform Version
Terraform Configuration Files
Expected Behavior
Terraform runs an
apply
command and saves state file to the Azure Blob Storage AccountActual Behavior
Terraform fails to snapshot a file that doesn't exist and fails:
Steps to Reproduce
snapshot
is set totrue
terraform init
terraform apply -lock=false
Additional Context
I was able to track down this issue to this line
terraform/backend/remote-state/azure/backend_state.go
Line 100 in a351053
For some reason in the azure remote state implementation a new empty blob is only created in the
Backend.StateMgr()
whenstate name does not equal the default name. There is another place where an empty blob is being created:
in the
RemoteClient.Lock()
functionterraform/backend/remote-state/azure/client.go
Line 151 in a351053
lock=false
Finally when a
RemoteClient.Put()
is called a snapshot is attempted, but because of the issues described above, noblob is found
terraform/backend/remote-state/azure/client.go
Line 75 in a351053
While this sounds like a rare combination of options we did run into this problem when using Terratest in combination with Terragrunt. I also think
azurerm
remote state code can benefit from a bit of a cleanup to eliminate multiple places wherea new empty state is created. For example in the S3 backend implementation a new state is created in the
Backend.StateMgr
functionterraform/backend/remote-state/s3/backend_state.go
Line 156 in a351053
I'm currently working on a PR to fix this issue in the
azurerm
backend by forcing a new blob to be created in theBackend.StateMgr
function regardless of the state name and also potentially removing new blob creation code from theRemoteClient.Lock()
function to avoid confusion. Let me know if this is not the right direction, e.g. if there is not guaranteeBackend.StateMgr
is called before any other state operation. It doesn't seem to be the case, but I didn't spend enough timewith this codebase yet.
The text was updated successfully, but these errors were encountered: