Terraform wants to change azurerm_monitor_diagnostic_setting log category settings #5673

Tommy-Ten · 2020-02-11T10:00:00Z

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.12.20

provider.azurerm v1.41.0

Affected Resource(s)

azurerm_monitor_diagnostic_setting

Terraform Configuration Files

resource "azurerm_monitor_diagnostic_setting" "export_activity_logs" {
  name                           = var.actLogName
  target_resource_id             = data.azurerm_subscription.current.id
  storage_account_id             = data.azurerm_storage_account.logging_activity_logs_archive_storage_account.id
  eventhub_authorization_rule_id = azurerm_eventhub_namespace_authorization_rule.eventhub_auth_rule.id

  log {
    category = "Administrative"
    enabled  = true

    retention_policy {
      enabled = true
      days    = 365
    }
  }

  log {
    category = "Security"
    enabled  = false

    retention_policy {
      enabled = false
    }
  }

  log {
    category = "ServiceHealth"
    enabled  = false

    retention_policy {
      enabled = false
    }
  }

  log {
    category = "Alert"
    enabled  = false

    retention_policy {
      enabled = false
    }
  }

  log {
    category = "Recommendation"
    enabled  = false

    retention_policy {
      enabled = false
    }
  }

  log {
    category = "Policy"
    enabled  = false

    retention_policy {
      enabled = false
    }
  }

  log {
    category = "Autoscale"
    enabled  = false

    retention_policy {
      enabled = false
    }
  }

  log {
    category = "ResourceHealth"
    enabled  = false

    retention_policy {
      enabled = false
    }
  }
}

Expected Behavior

After an initial terraform apply, when I run terraform plan or another terraform apply, I should see no changes.

Actual Behavior

After an initial terraform apply, when I run terraform plan or another terraform apply, I see settings for log categories that I defined in my configuration being changed.

Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.event_hub_activity_logs.azurerm_monitor_diagnostic_setting.export_activity_logs will be updated in-place
  ~ resource "azurerm_monitor_diagnostic_setting" "export_activity_logs" {
        eventhub_authorization_rule_id = "XXXXX"
        id                             = "XXXXX"
        name                           = "XXXXX"
        storage_account_id             = "XXXXX"
        target_resource_id             = "XXXXX"

      + log {
          + category = "Administrative"
          + enabled  = true

          + retention_policy {
              + days    = 365
              + enabled = true
            }
        }
      - log {
          - category = "Administrative" -> null
          - enabled  = true -> null
        }
      + log {
          + category = "Alert"
          + enabled  = false

          + retention_policy {
              + enabled = false
            }
        }
      - log {
          - category = "Alert" -> null
          - enabled  = false -> null
        }
      + log {
          + category = "Autoscale"
          + enabled  = false

          + retention_policy {
              + enabled = false
            }
        }
      - log {
          - category = "Autoscale" -> null
          - enabled  = false -> null
        }
      + log {
          + category = "Policy"
          + enabled  = false

          + retention_policy {
              + enabled = false
            }
        }
      - log {
          - category = "Policy" -> null
          - enabled  = false -> null
        }
      + log {
          + category = "Recommendation"
          + enabled  = false

          + retention_policy {
              + enabled = false
            }
        }
      - log {
          - category = "Recommendation" -> null
          - enabled  = false -> null
        }
      + log {
          + category = "ResourceHealth"
          + enabled  = false

          + retention_policy {
              + enabled = false
            }
        }
      - log {
          - category = "ResourceHealth" -> null
          - enabled  = false -> null
        }
      + log {
          + category = "Security"
          + enabled  = false

          + retention_policy {
              + enabled = false
            }
        }
      - log {
          - category = "Security" -> null
          - enabled  = false -> null
        }
      + log {
          + category = "ServiceHealth"
          + enabled  = false

          + retention_policy {
              + enabled = false
            }
        }
      - log {
          - category = "ServiceHealth" -> null
          - enabled  = false -> null
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Steps to Reproduce

Add the configuration block above
Run terraform apply to set the initial diagnostic settings
Run terraform apply (or terraform plan) again to observe the planned changes

References

azurerm_monitor_diagnostic_setting showing changes for logs/metrics not defined in the config #3572 azurerm_monitor_diagnostic_setting generates diff immediately after applying #2466

The text was updated successfully, but these errors were encountered:

jonmaestas · 2020-03-26T19:51:28Z

Are these issues related? #2466

rinjohn · 2020-04-14T16:57:55Z

Is there any solution for this? I am struggling with this issue while setting up the diagnostic settings of Recovery Service Vault and Azure SQL Database.
terraform version -0.12.13
Provider version- v1.27.1

Tommy-Ten · 2020-04-14T17:17:26Z

Are these issues related? #2466

Yes, it seems so.

batesandy · 2020-04-23T12:54:53Z

Seeing the same for diagnostic logs on subscription resource

nyuen · 2020-04-23T15:22:35Z

The issue is that the activity logs do not support Retention Policy which is mandatory on the Terraform provider

This one should probably be optional in the provider's code since it also seems optional (omitEmpty) in the Azure SDK

"retention_policy": {
Type: schema.TypeList,
Required: true,
MaxItems: 1,
Elem: &schema.Resource{
Schema: map[string]*schema.Schema{
"enabled": {
Type: schema.TypeBool,
Required: true,
},

  "days": {
    Type:         schema.TypeInt,
    Optional:     true,
    ValidateFunc: validation.IntAtLeast(0),
  },
},

},
},

…to Optional hashicorp#5673

rudolphjacksonm · 2020-07-21T14:27:01Z

I'm not sure I understand why my comment was marked as off-topic. This issue hasn't been fixed by @nyuen 's PR, since it's still appearing in provider version v2.13.0 and above. Therefore there's still an issue to be solved and whatever the bug is, it's still present in recent versions of the provider.

nyuen · 2020-07-22T03:26:50Z

Hi @rudolphjacksonm , the intent of my change was to make the retention policy optional as the new Activity log experience doesn't seem to provide the option to specify a retention policy anymore (as per the portal UI).

To make the workflow idempotent I've considered what is returned by the terraform state: if you have a look at what the diff indicates on the subsequent terraform apply you will see that the retention_policy is not stored at all which is causing a diff.

below is the Terraform code that I'm now using to create activity logs in Terraform as per the changes I've made to the azurerm provider

Sample code

provider "azurerm" {
  version = "=2.19.0"
  features {}
}

data "azurerm_subscription" "current" {
}

resource "azurerm_resource_group" "test_diag_rg" {
  name     = "rg-bug-terraform5673"
  location = "southeastasia"
}


resource "azurerm_storage_account" "tf_diag" {
  name                     = "tfbug5673stnyuen"
  resource_group_name      = azurerm_resource_group.test_diag_rg.name
  location                 = azurerm_resource_group.test_diag_rg.location
  account_tier             = "Standard"
  account_replication_type = "GRS"
}


resource "azurerm_monitor_diagnostic_setting" "export_activity_logs" {
  name                           = "demo_log_tf"
  target_resource_id             = data.azurerm_subscription.current.id
  storage_account_id             = azurerm_storage_account.tf_diag.id

  log {
    category = "Administrative"
    enabled  = true
  }

  log {
    category = "Security"
    enabled  = false
  }

  log {
    category = "ServiceHealth"
    enabled  = false
  }

  log {
    category = "Alert"
    enabled  = false
  }

  log {
    category = "Recommendation"
    enabled  = false
  }

  log {
    category = "Policy"
    enabled  = false
  }

  log {
    category = "Autoscale"
    enabled  = false
  }

  log {
    category = "ResourceHealth"
    enabled  = false
  }
}

rudolphjacksonm · 2020-07-22T13:56:23Z

Hi @nyuen, I've tried the same on my end but Terraform still wants to change the category for each entry. I've tried applying this several times and inspected the tfstate, which shows the retention_policy value is set to an empty array. Let me know if I'm doing something wrong here:

Sample Code

resource "azurerm_monitor_diagnostic_setting" "aks_cluster_diagnostics" {
  count                          = var.aks_enable_diagnostics == "true" && var.aks_diagnostic_event_hub_name != "" ? 1 : 0
  name                           = "aks-cluster-to-eventhub"
  target_resource_id             = azurerm_kubernetes_cluster.aks_with_aad_parameters.id
  eventhub_name                  = "aks-cluster-diagnostics"
  eventhub_authorization_rule_id = "${data.azurerm_subscription.current.id}/resourceGroups/${var.aks_rg_name}/providers/Microsoft.EventHub/namespaces/${var.aks_diagnostic_event_hub_name}/AuthorizationRules/RootManageSharedAccessKey"
  log {
    category = "kube-apiserver"
    enabled  = true
  }
  log {
    category = "kube-controller-manager"
    enabled  = true
  }
  log {
    category = "kube-scheduler"
    enabled  = true
  }
  log {
    category = "kube-audit"
    enabled  = true
  }
  log {
    category = "cluster-autoscaler"
    enabled  = true
  }
  metric {
    category = "AllMetrics"
    enabled  = true
  }
  depends_on = [azurerm_kubernetes_cluster.aks_with_aad_parameters]
}

resource "azurerm_monitor_diagnostic_setting" "aks_nsg_diagnostics" {
  count                          = var.aks_enable_diagnostics == "true" && var.aks_diagnostic_event_hub_name != "" ? 1 : 0
  name                           = "aks-nsg-to-eventhub"
  target_resource_id             = data.azurerm_resources.aks_cluster_managed_nsg.resources[0].id
  eventhub_name                  = "aks-nsg-diagnostics"
  eventhub_authorization_rule_id = "${data.azurerm_subscription.current.id}/resourceGroups/${var.aks_rg_name}/providers/Microsoft.EventHub/namespaces/${var.aks_diagnostic_event_hub_name}/AuthorizationRules/RootManageSharedAccessKey"
  log {
    category = "NetworkSecurityGroupEvent"
    enabled  = true
  }
  log {
    category = "NetworkSecurityGroupRuleCounter"
    enabled  = true
  }
  depends_on = [
    azurerm_kubernetes_cluster.aks_with_aad_parameters
  ]
}

Plan Output

# module.aks-cluster.azurerm_monitor_diagnostic_setting.aks_cluster_diagnostics[0] will be updated in-place
  ~ resource "azurerm_monitor_diagnostic_setting" "aks_cluster_diagnostics" {
        eventhub_authorization_rule_id = "/subscriptions/000000-00000-00000-00000/resourceGroups/devuks1/providers/Microsoft.EventHub/namespaces/devuks1-logging-ns-primary/AuthorizationRules/RootManageSharedAccessKey"
        eventhub_name                  = "aks-cluster-diagnostics"
        id                             = "/subscriptions/000000-00000-00000-00000/resourcegroups/devuks1/providers/Microsoft.ContainerService/managedClusters/devuks1|aks-cluster-to-eventhub"
        name                           = "aks-cluster-to-eventhub"
        target_resource_id             = "/subscriptions/000000-00000-00000-00000/resourcegroups/devuks1/providers/Microsoft.ContainerService/managedClusters/devuks1"

      - log {
          - category = "cluster-autoscaler" -> null
          - enabled  = true -> null

          - retention_policy {
              - days    = 0 -> null
              - enabled = false -> null
            }
        }
      + log {
          + category = "cluster-autoscaler"
          + enabled  = true
        }
      - log {
          - category = "kube-apiserver" -> null
          - enabled  = true -> null

          - retention_policy {
              - days    = 0 -> null
              - enabled = false -> null
            }
        }
      + log {
          + category = "kube-apiserver"
          + enabled  = true
        }
      - log {
          - category = "kube-audit" -> null
          - enabled  = true -> null

          - retention_policy {
              - days    = 0 -> null
              - enabled = false -> null
            }
        }
      + log {
          + category = "kube-audit"
          + enabled  = true
        }
      - log {
          - category = "kube-controller-manager" -> null
          - enabled  = true -> null

          - retention_policy {
              - days    = 0 -> null
              - enabled = false -> null
            }
        }
      + log {
          + category = "kube-controller-manager"
          + enabled  = true
        }
      - log {
          - category = "kube-scheduler" -> null
          - enabled  = true -> null

          - retention_policy {
              - days    = 0 -> null
              - enabled = false -> null
            }
        }
      + log {
          + category = "kube-scheduler"
          + enabled  = true
        }

      - metric {
          - category = "AllMetrics" -> null
          - enabled  = true -> null

          - retention_policy {
              - days    = 0 -> null
              - enabled = false -> null
            }
        }
      + metric {
          + category = "AllMetrics"
          + enabled  = true
        }
    }

  # module.aks-cluster.azurerm_monitor_diagnostic_setting.aks_nsg_diagnostics[0] must be replaced
-/+ resource "azurerm_monitor_diagnostic_setting" "aks_nsg_diagnostics" {
        eventhub_authorization_rule_id = "/subscriptions/000000-00000-00000-000009/resourceGroups/devuks1/providers/Microsoft.EventHub/namespaces/devuks1-logging-ns-primary/AuthorizationRules/RootManageSharedAccessKey"
        eventhub_name                  = "aks-nsg-diagnostics"
      ~ id                             = "/subscriptions/000000-00000-00000-00000/resourceGroups/mc_devuks1_uksouth/providers/Microsoft.Network/networkSecurityGroups/aks-agentpool-28835032-nsg|aks-nsg-to-eventhub" -> (known after apply)
        name                           = "aks-nsg-to-eventhub"
      ~ target_resource_id             = "/subscriptions/000000-00000-00000-00000/resourceGroups/mc_devuks1_uksouth/providers/Microsoft.Network/networkSecurityGroups/aks-agentpool-28835032-nsg" -> (known after apply) # forces replacement

      - log {
          - category = "NetworkSecurityGroupEvent" -> null
          - enabled  = true -> null

          - retention_policy {
              - days    = 0 -> null
              - enabled = false -> null
            }
        }
      + log {
          + category = "NetworkSecurityGroupEvent"
          + enabled  = true
        }
      - log {
          - category = "NetworkSecurityGroupRuleCounter" -> null
          - enabled  = true -> null

          - retention_policy {
              - days    = 0 -> null
              - enabled = false -> null
            }
        }
      + log {
          + category = "NetworkSecurityGroupRuleCounter"
          + enabled  = true
        }
    }

"attributes": {
            "eventhub_authorization_rule_id": "/subscriptions/00000-00000-00000-00000/resourceGroups/devuks1/providers/Microsoft.EventHub/namespaces/devuks1-logging-ns-primary/AuthorizationRules/RootManageSharedAccessKey",
            "eventhub_name": "aks-cluster-diagnostics",
            "id": "/subscriptions/00000-00000-00000-00000/resourcegroups/devuks1/providers/Microsoft.ContainerService/managedClusters/devuks1|aks-cluster-to-eventhub",
            "log": [
              {
                "category": "cluster-autoscaler",
                "enabled": true,
                "retention_policy": []
              },
              {
                "category": "kube-apiserver",
                "enabled": true,
                "retention_policy": []
              },
              {
                "category": "kube-audit",
                "enabled": true,
                "retention_policy": []
              },
              {
                "category": "kube-controller-manager",
                "enabled": true,
                "retention_policy": []
              },
              {
                "category": "kube-scheduler",
                "enabled": true,
                "retention_policy": []
              }
            ],

nyuen · 2020-07-22T15:52:42Z

My fix addresses specifically the Activity log which doesn't support the retention_policy even when stored in a storage account. For the Kubernetes related diagnostic settings it seems that the retention policy shouldn't be empty (even though you're not storing the settings to a storage account).

I would try:

log {
    category = "kube-apiserver"
    enabled  = true
 retention_policy {
               days    = 0
              enabled = false 
            }
}

rudolphjacksonm · 2020-07-31T08:18:40Z

@nyuen that worked! I've applied the same change for our Eventhub diagnostic settings which were getting recreated on every apply due to the same issue. Thanks so much for your help, that's been bothering me for ages!

magodo · 2020-08-04T03:12:41Z

The issue being discussed here is that though user has specified all the available diag settings, terraform still reports diff, which has been addressed by #6603. So I'm gonna close this issue for now.

For others who gets diff because of not specifying all the available diag settings, you can subscribe #7235 for any update for that issue.

ghost · 2020-09-03T14:50:13Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!

tracypholmes added bug service/monitor labels Feb 21, 2020

nyuen added a commit to nyuen/terraform-provider-azurerm that referenced this issue Apr 24, 2020

modifying the Diagnostic settings schema to set the retention policy …

3db7aa6

…to Optional hashicorp#5673

nyuen mentioned this issue Apr 24, 2020

azurerm_monitor_diagnostic_setting - mark retention_policy & retention_policy as optional #6603

Merged

This comment has been minimized.

Sign in to view

magodo mentioned this issue Aug 4, 2020

azurerm_monitor_diagnostic_setting generates diff immediately after applying #2466

Closed

magodo closed this as completed Aug 4, 2020

ghost locked and limited conversation to collaborators Sep 3, 2020

catriona-m added the v/1.x (legacy) label Jul 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Terraform wants to change azurerm_monitor_diagnostic_setting log category settings #5673

Terraform wants to change azurerm_monitor_diagnostic_setting log category settings #5673

Tommy-Ten commented Feb 11, 2020 •

edited

Loading

jonmaestas commented Mar 26, 2020

rinjohn commented Apr 14, 2020 •

edited

Loading

Tommy-Ten commented Apr 14, 2020

batesandy commented Apr 23, 2020

nyuen commented Apr 23, 2020 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

rudolphjacksonm commented Jul 21, 2020

nyuen commented Jul 22, 2020 •

edited

Loading

rudolphjacksonm commented Jul 22, 2020 •

edited

Loading

nyuen commented Jul 22, 2020

rudolphjacksonm commented Jul 31, 2020

magodo commented Aug 4, 2020

ghost commented Sep 3, 2020

Terraform wants to change azurerm_monitor_diagnostic_setting log category settings #5673

Terraform wants to change azurerm_monitor_diagnostic_setting log category settings #5673

Comments

Tommy-Ten commented Feb 11, 2020 • edited Loading

Community Note

Terraform (and AzureRM Provider) Version

Affected Resource(s)

Terraform Configuration Files

Expected Behavior

Actual Behavior

Steps to Reproduce

References

jonmaestas commented Mar 26, 2020

rinjohn commented Apr 14, 2020 • edited Loading

Tommy-Ten commented Apr 14, 2020

batesandy commented Apr 23, 2020

nyuen commented Apr 23, 2020 • edited Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

rudolphjacksonm commented Jul 21, 2020

nyuen commented Jul 22, 2020 • edited Loading

Sample code

rudolphjacksonm commented Jul 22, 2020 • edited Loading

Sample Code

Plan Output

nyuen commented Jul 22, 2020

rudolphjacksonm commented Jul 31, 2020

magodo commented Aug 4, 2020

ghost commented Sep 3, 2020

Tommy-Ten commented Feb 11, 2020 •

edited

Loading

rinjohn commented Apr 14, 2020 •

edited

Loading

nyuen commented Apr 23, 2020 •

edited

Loading

nyuen commented Jul 22, 2020 •

edited

Loading

rudolphjacksonm commented Jul 22, 2020 •

edited

Loading