Support for Azure Storage Account Data protection

[davelosert]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Azure Storage Accounts now support some Data-Protection configurations, for example versioning or soft deletion for blob-storages.

Would be great if this could be configured with Terraform as well.

New or Affected Resource(s)

• azurerm_storage_account

Potential Terraform Configuration

resource "azurerm_storage_account" "example" {
  name                     = "examplestoracc"
  resource_group_name      = azurerm_resource_group.example.name
  location                 = azurerm_resource_group.example.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
  data_protection = {
    turn_on_versioning         = true
    turn_on_soft_deletion      = true
    keep_deleted_blobs_in_days = 7
    ...
  }
}

References

• https://docs.microsoft.com/en-us/azure/storage/blobs/versioning-overview?tabs=powershell
• https://docs.microsoft.com/en-us/azure/storage/blobs/soft-delete-blob-overview
• https://docs.microsoft.com/en-us/azure/storage/blobs/soft-delete-container-overview

[fluffy-cakes]

Upvoting, this has been available for a while now: https://azure.microsoft.com/en-gb/updates/azure-blob-versioning-is-now-general-available/

Current work around (not hugely ideal)

resource "azurerm_template_deployment" "asdf" {
    name                     = "asdf"
    resource_group_name      = azurerm_resource_group.asdf.name
    deployment_mode          = "Incremental"
    parameters               = {
        "storageAccount"     = azurerm_storage_account.asdf.name
    }

    template_body = <<DEPLOY
        {
            "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
            "contentVersion": "1.0.0.0",
            "parameters": {
                "storageAccount": {
                    "type": "string",
                    "metadata": {
                        "description": "Storage Account Name"}
                }
            },
            "variables": {},
            "resources": [
                {
                    "type": "Microsoft.Storage/storageAccounts/blobServices",
                    "apiVersion": "2019-06-01",
                    "name": "[concat(parameters('storageAccount'), '/default')]",
                    "properties": {
                        "IsVersioningEnabled": true
                    }
                }
            ]
        }
    DEPLOY
}

[StefanSchoof]

The turn_on_soft_deletion you can do today with a delete_retention_policy:

resource "azurerm_storage_account" "storage_account_attachments" {
  ...
  blob_properties {
    delete_retention_policy {
      days = 365
    }
  }
}

[kpakur]

In addition to #8268 (comment) I'm also adding template example to include changeFeed, restorePolicy and containerDeleteRetentionPolicy

resource "azurerm_storage_account" "asdfstorage" {
  name                     = oooo
  resource_group_name      = azurerm_resource_group.asdf.name
  ...
  blob_properties {
    delete_retention_policy {
      days = 365
    }
  }
}

resource "azurerm_template_deployment" "asdf" {
    name                     = "asdf"
    resource_group_name      = azurerm_resource_group.asdf.name
    deployment_mode          = "Incremental"
    parameters               = {
        "storageAccount"     = azurerm_storage_account.asdfstorage.name
    }
    template_body = <<DEPLOY 
        {
            "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
            "contentVersion": "1.0.0.0",
            "parameters": {
                "storageAccount": {
                    "type": "string",
                    "metadata": {
                        "description": "Storage Account Name"}
                }
            },
            "variables": {},
            "resources": [
                {
                    "type": "Microsoft.Storage/storageAccounts/blobServices",
                    "apiVersion": "2019-06-01",
                    "name": "[concat(parameters('storageAccount'), '/default')]",
                    "properties": {
                        "IsVersioningEnabled": true,
                        "ChangeFeed": {
                            "enabled": true
                        },
                        "RestorePolicy": {
                            "enabled": true,
                            "days": 364
                        },
                        "ContainerDeleteRetentionPolicy": {
                            "enabled": true,
                            "days": 7
                        }
                    }
                }
            ]
        }
    DEPLOY
}

[iayongwa]

The only shortcoming I've experienced so far with @fluffy-cakes work around is that it only works for Standard account tier for storage account. I just tried it in my pipeline with Premium and it fails.

[Eric-Jckson]

With the new feature for blob versioning this would be a valuable feature for the storage account.

https://docs.microsoft.com/en-us/azure/storage/blobs/versioning-overview

[sean-t-shen]

Yes, these need to be supported. Upvoting...
Until then, we will have to use ARM template to configure the features not yet covered.

[ekUSA]

Upvoting, this has been available for a while now: https://azure.microsoft.com/en-gb/updates/azure-blob-versioning-is-now-general-available/

Current work around (not hugely ideal)

resource "azurerm_template_deployment" "asdf" {
    name                     = "asdf"
    resource_group_name      = azurerm_resource_group.asdf.name
    deployment_mode          = "Incremental"
    parameters               = {
        "storageAccount"     = azurerm_storage_account.asdf.name
    }

    template_body = <<DEPLOY
        {
            "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
            "contentVersion": "1.0.0.0",
            "parameters": {
                "storageAccount": {
                    "type": "string",
                    "metadata": {
                        "description": "Storage Account Name"}
                }
            },
            "variables": {},
            "resources": [
                {
                    "type": "Microsoft.Storage/storageAccounts/blobServices",
                    "apiVersion": "2019-06-01",
                    "name": "[concat(parameters('storageAccount'), '/default')]",
                    "properties": {
                        "IsVersioningEnabled": true
                    }
                }
            ]
        }
    DEPLOY
}

Does anyone know why when I try to apply this workaround, I run into this error? thanks

The Terraform configuration must be valid before initialization so that
Terraform can determine which modules and providers need to be installed.

Error: Invalid expression

on main.tf line 90, in resource "azurerm_template_deployment" "qacdnstg_dataProtection":
90: template_body = <<DEPLOY

Expected the start of an expression, but found an invalid expression token.

Error: Argument or block definition required

on main.tf line 124:
124: DEPLOY

An argument or block definition is required here. To set an argument, use the
equals sign "=" to introduce the argument value.

[sean-t-shen]

I tried to reproduce your error, but I was not able to. I was able to deploy exactly (well, almost) the same as you did. I was successful.

Here is my Terraform and provider version -->
terraform v0.13.6

• provider registry.terraform.io/hashicorp/azurerm v2.44.0

Here is my code (the only change I changed was the storage account name, because "asdf" is taken up already (probably by you yourself )

resource "azurerm_resource_group" "asdf" {
name = "asdf"
location = "East US"
}

resource "azurerm_storage_account" "asdf" {
name = "asdfseantshen"
resource_group_name = azurerm_resource_group.asdf.name
location = azurerm_resource_group.asdf.location
account_tier = "Standard"
account_replication_type = "LRS"
account_kind = "StorageV2"
}

resource "azurerm_template_deployment" "asdf" {
name = "asdf"
resource_group_name = azurerm_resource_group.asdf.name
deployment_mode = "Incremental"
parameters = {
"storageAccount" = azurerm_storage_account.asdf.name
}

template_body = <<DEPLOY
    {
        "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
        "contentVersion": "1.0.0.0",
        "parameters": {
            "storageAccount": {
                "type": "string",
                "metadata": {
                    "description": "Storage Account Name"}
            }
        },
        "variables": {},
        "resources": [
            {
                "type": "Microsoft.Storage/storageAccounts/blobServices",
                "apiVersion": "2019-06-01",
                "name": "[concat(parameters('storageAccount'), '/default')]",
                "properties": {
                    "IsVersioningEnabled": true
                }
            }
        ]
    }
DEPLOY

}

A few observations:

1. In your original code, you had specified the deployment name as "asdf", but in your error, it is complaining about another deployment "qacdnstg_dataProtection". I suppose you just changed your sample code to show case the issue, while in your original code the Terraform reference to the deployment is really qacdnstg_dataProtection. I hope that is the case. If not, then you need to look into the code around the other deployment.

2. I think there may be something wrong in your main.tf?
   Here is my local version of the Terraform main.tf code on the terraform and provider block. In reality, I also have a backend block since I am using Terraform cloud. But if you are local, maybe you want to check and compare with the below.

terraform {
required_providers {
azurerm = {
source = "hashicorp/azurerm"
version = "=2.44.0"
}
}
}

provider "azurerm" {
features {}
}

3. And once you make sure you are ok, do an "terraform init" to make everything ok, and afterwards, do a "terraform --version" and share the output in the post here.

Thanks
Sean

[MattiasAng]

Created a pull reuqest for versioning.

ChangeFeed API seems to be either broken or documentation is not up to date. I am unable to set a retentionInDays parameter.
https://docs.microsoft.com/en-us/rest/api/storagerp/blobservices/setserviceproperties#changefeed

[ekUSA]

template_body = <<DEPLOY

thanks for trying to reproduce it @sean-t-shen ,

turns out I was missing the = before <<

[sean-t-shen]

Good for you to find where the problem is.

[ghost]

This has been released in version 2.57.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 2.57.0"
}
# ... other configuration ...

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.