Add support for Azure Blob Storage Lifecycle

[aerialls]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

You can now have lifecycles directly in your Azure Storage accounts. The documentation is available here.

The lifecycle management policy lets you:

• Transition blobs to a cooler storage tier (hot to cool, hot to archive, or cool to archive) to optimize for performance and cost
• Delete blobs at the end of their lifecycles
• Define rules to be run once per day at the storage account level
• Apply rules to containers or a subset of blobs (using prefixes as filters)

New or Affected Resource(s)

• azurerm_storage_account

[stuartleeks]

I've been looking over how similar things have been implemented in various providers and have two thoughts on this.

Approach 1 - JSON
Encode rules as JSON in the management policy:

resource "azurerm_storage_account" "example" {
  name                     = "myaccount"
  resource_group_name      = "myresourcegroup"
  location                 = "westeurope"
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_storage_management_policy" "example" {
  storage_account_name ="${azurerm_storage_account.example.name}"

  rules = jsonencode([
    {
      "name": "ruleFoo",
      "enabled": true,
      "type": "Lifecycle",
      "definition": {
        "filters": {
          "blobTypes": [ "blockBlob" ],
          "prefixMatch": [ "container1/foo" ]
        },
        "actions": {
          "baseBlob": {
            "tierToCool": { "daysAfterModificationGreaterThan": 30 },
            "tierToArchive": { "daysAfterModificationGreaterThan": 90 },
            "delete": { "daysAfterModificationGreaterThan": 2555 }
          },
          "snapshot": {
            "delete": { "daysAfterCreationGreaterThan": 90 }
          }
        }
      }
    }])
  }
}

Approach 2 - HCL
Use arguments and blocks:

resource "azurerm_storage_account" "example" {
  name                     = "myaccount"
  resource_group_name      = "myresourcegroup"
  location                 = "westeurope"
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_storage_management_policy" "example" {
  storage_account_name ="${azurerm_storage_account.example.name}"

  rule {
    name = "rule1"
    enabled = true
    type = "Lifecycle"
    definition {
      filters {
        prefix_match = ["container1/wibble"]
        blob_types = ["blockBlob"]
      }
      actions = {
        base_blob {
          tier_to_cool {
            days_after_modification_greater_than = 30
          }
          tier_to_archive { 
            days_after_modification_greater_than = 90
          }
          delete {
              days_after_modification_greater_than = 2555
          }
        snapshot {
          delete {
            days_after_creation_greater_than = 90
          }
        }
      }
    }
  }
}

Comments welcome!

[stuartleeks]

@katbyte, @tombuildsstuff I'd welcome your thoughts on this

[tombuildsstuff]

chatting in Slack: in general we use the HCL approach unless there's a good reason not too (e.g. the ARM Template Resource) - since this allows us to apply additional validation to fields as necessary

[stuartleeks]

After further discussion in slack:

resource "azurerm_storage_account" "example" {
  name                     = "myaccount"
  resource_group_name      = "myresourcegroup"
  location                 = "westeurope"
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_storage_management_policy" "example" {
  storage_account_id ="${azurerm_storage_account.example.id}"

  rule {
    name = "rule1"
    enabled = true
    type = "Lifecycle"
    filters {
      prefix_match = ["container1/wibble"]
      blob_types = ["blockBlob"]
    }
    actions {
      base_blob {
        tier_to_cool_after_days_modification_greater_than = 30
        tier_to_archive_after_days_modification_greater_than = 90
        delete_after_days_modification_greater_than = 2555
      }
      snapshot {
        delete_after_days_modification_greater_than = 90
    }
  }
}

[stuartleeks]

For reference, the Azure API docs are here

[stuartleeks]

I've encountered an issue with the Go SDK for Azure which I'm working through.

@tombuildsstuff do you want me to submit a separate PR for updating to the new Go SDK when it is released as you did when I bumped the storage api version previously?

[tombuildsstuff]

@stuartleeks as that's a breaking change to the Swagger it'll need to go out in the next major version of the SDK (presumably v32.0.0) which generally happens at the end of the month; but that sounds great if you don't mind 👍

[obeleh]

I've just imported my existing rules. When I do a terraform plan I get:

Error: invalid value for rule.0.name (A rule name can contain any combination of alpha numeric characters.)

I had hyphens in my rule names.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!