Properly handle recurring downtimes definitions

[armcburney]

Overview

This PR implements a workaround for handling recurring downtimes with the Datadog Terraform provider. Recurring downtimes differ from regular 'one-off' downtimes in that subsequent recurrences are scheduled as new downtimes from the previous parent definition. Conceptually, we can think of this as a "linked list" of downtimes where each subsequent downtime is scheduled from the previous scheduled recurrence downtime definition. All fields are copied over from the previous parent with the exception of certain fields like start and end, which are calculated off of the recurrence attribute.

The Datadog Terraform provider keeps reference to this original parent downtime ID. Since subsequent recurrence downtimes are scheduled as new downtimes (with a new ID), updates in the UI/API to the existing 'child' downtime corresponding to the original recurrence would not previously be recognized when comparing the downtime's state with what we store in Terraform. Moreover, after downtimes expire, we delete them from our database after a certain period of time. This behavior was recently changed so that we don't delete downtimes after they expire if they are the first downtime in the recurrence chain (i.e., the original parent downtime).

Instead, we now return that downtime with a new active_child field in our GET /api/v1/downtime API - which we use to compare state with Terraform. This way updates from the UI/API will be propagated back to Terraform. Additionally, when making updates through Terraform, we call the PUT /api/v1/downtime/{downtime_id} endpoint with the active_child definition on the active_child's ID, so that changes from Terraform will be made to the current active recurrence downtime.

Caveats

Caveat One

UPDATE: We don't check the start/end boundaries for changes if the recurring downtime is a child to prevent superfluous diffs every time the downtime is rescheduled. The con to this approach is that if the start/end values are changed in the UI on a child recurring downtime, the diff will not be picked up by Terraform. We plan to iterate on this solution to address this shortcoming, but feel the benefits of the child/parent references being handled by Terraform are worth merging in.

Since new downtimes are scheduled each time a recurrence is rescheduled, fields like start and end will perpetually differ after the first schedule when running terraform plan/terraform apply.

$ terraform plan
datadog_downtime.recurring_downtime: Refreshing state... [id=1337]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # datadog_downtime.recurring_downtime will be updated in-place
  ~ resource "datadog_downtime" "recurring_downtime" {
      ~ end             = 1621016100 -> 1620929700
        id              = "1337"
      ~ message         = "test this out edit ui" -> "test this out"
      ~ start           = 1621015800 -> 1620929400
        # (6 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

I recommend we update the documentation around recurring downtimes to document this problem and recommended an approach to mitigating this (i.e., using ignore_changes to ignore the start and end updates).

resource "datadog_downtime" "recurring_downtime" {
  scope        = ["*"]
  start        = 1620929400
  end          = 1620929700
  timezone     = "EST"
  monitor_tags = ["github:armcburney"]
  message      = "test this out"

  recurrence {
    type   = "days"
    period = 1
  }

  lifecycle {
    ignore_changes = [
      start,
      end
    ]
  }
}

Caveat Two

Recurring downtimes created before 2021-05-13 using Terraform will need to be deleted and recreated for the references to work with the newest version of the Terraform provider. All recurring downtimes created after 2021-05-13 will have the updated parent/child references, ensuring they’ll work as expected with the latest version of the provider. We apologize for this inconvenience.

Fixes

• Feature request: Please support parent_id on recurring Downtimes #109
• Managing recurring datadog_downtime #393

[phillip-dd]

thanks @armcburney! This makes sense to me, except for this part:

fields like start and end will perpetually differ after the first schedule

From a customer perspective I don't think this is what we want - the provider should not be showing a diff when this is working as expected. As well, if customers use ignore_changes, then they won't be able to see any true diffs if the recurrence is updated in the UI.

Some other options:

• ignore start/end explicitly in the provider
• always compare start/end on the parent downtime (this would only catch changes if the original parent was changed)
• compare duration: e.g. end-start
• something else?

[phillip-dd]

👍 couple of questions, but this looks good to me. Definitely an improvement!

[phillip-dd]

I saw there was a slack conversation about this, @therve can you just confirm this is what the recommendation was?

[therve]

Yes we'll come back to fix it later.

[phillip-dd]

if we include this, I think it should be actually read only.

[phillip-dd]

Bumping on this, is this part auto generated or a copy/paste?

[skarimo]

The doc is auto generated. But to avoid this, we should remove the line below since the attribute is read only:

terraform-provider-datadog/datadog/resource_datadog_downtime.go

Line 253 in f06954f

Optional: true,

[phillip-dd]

lgtm - we may need to update the PR title for the change log

[therve]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).