DB option groups with a failed IAM-related option apply do not detect missing options in subsqeuent plan

[hashibot]

This issue was originally opened by @mattzimmerman as hashicorp/terraform#14417. It was migrated here as part of the provider split. The original body of the issue is below.

---

Terraform Version

0.9.4

Affected Resource(s)

• aws_db_option_group

Terraform Configuration Files

resource "aws_db_option_group" "sql-server-options" {
  name = "sql-server-${var.environment}-options"
  option_group_description = "SQL Server backup options"
  engine_name = "${var.db-engine}"
  major_engine_version = "${var.db-major-engine-version}"
  option {
    option_name = "SQLSERVER_BACKUP_RESTORE"
      option_settings {
        name = "IAM_ROLE_ARN"
        value = "${aws_iam_role.sql-server.arn}"
      }
  }
}

resource "aws_iam_instance_profile" "sql-server" {
  name = "sql-server.${var.namespace}"
  role = "${aws_iam_role.sql-server.id}"
}

resource "aws_iam_role" "sql-server" {
  name = "sql-server.${var.namespace}"
  path = "/"
  assume_role_policy = <removed for brevity
}

resource "aws_iam_role_policy" "sql-server-backup" {
  name = "sql-server-backup.${var.namespace}"
  role = "${aws_iam_role.sql-server.id}"
  policy = <removed for brevity>
}

Description

DB option groups with a failed option apply (I frequently see this on SQL Server DB option groups with an option that includes an IAM role for backing up/restoring to/from S3 buckets).

Terraform successfully creates my new IAM role, immediately applies that as part of an option to the aws_db_option_group, but then that apply fails because the IAM resource isn't available from the AWS API yet.

As an aside, it would be nice if terraform apply could wait for the role to become available before attempting to use it in a aws_db_option_group (but that is easy to work around by creating it and tracking the IAM role used for the option group in another terraform run prior to creating the option group).

However, when the Terraform run fails in this way (due to the newly-created IAM role and eventual consistency):

Error applying plan:
1 error(s) occurred:
* aws_db_option_group.sql-server-backup-options: 1 error(s) occurred:
* aws_db_option_group.sql-server-backup-options: Error modifying DB Option Group: InvalidParameterValue: IAM role ARN value is invalid or does not include the required permissions for: SQLSERVER_BACKUP_RESTORE
status code: 400, request id: d11b2c48-3426-11e7-977c-ddb68cb08401

I noticed that Terraform has lost track of the state of the option it was adding. Subsequent plan/apply attempts do not detect any changes, and the option group that is created from the initial run never actually gets the option added to it.

[SimplyMonk]

Having the same issue. Found a work around though by adding a wait after the IAM profile creation and then having the option_group depend on the role's instance profile:

provisioner "local-exec" {
    command = "ping 127.0.0.1 -n 11 > nul"
} 

[apeeters]

We just stumbled upon this issue. In our case we did not get any error while applying plan (as far as I know), but somehow the ARN on AWS is empty. The this missing ARN is however not being detected: according to the state, it is filled out, according to AWS console, it isn't.

[smastrorocco]

+1

[im-dofoborh]

@SimplyMonk how did you apply this.
Experiencing the same problem. I have tried various approach but can't make sense of yours. Would appreciate snippets of how you actually applied this.

[im-dofoborh]

Moved away from this but faith brought this up again.
A work around for me was to think of an even simplier approach, "Atlas" which we fortunately use.

Rather than dynamically generating a value, we pass it semi-automatically as the generated value is quite consistant. It's always the same.

resource "aws_db_option_group" "rds_backup_restore" {
  name                            = "microsoftworkinprogress-backup-restore"
  option_group_description        = "Ability restore Mssql natively from S3"
  engine_name                     = "sqlserver-se"
  major_engine_version            = "14.00"

  option {
    option_name                   = "SQLSERVER_BACKUP_RESTORE"
    option_settings {
      name                        = "IAM_ROLE_ARN"
      value                       = "${var.manual_populate_s3_role_arn}" <--- passed as a variable instead
    }
  }
}

Atlas is secure meaning we can pass (ARN) manually like so in the variable ${var.manual_populate_s3_role_arn}

arn:aws:iam::012345678910:role/microsoftworkinprogress-backup-restore

This resolved that problem. On to the next.

[antgel]

Confirming this annoying bug. Thanks to @SimplyMonk for the inspiration. Relevant parts of our configuration (untested as yet, but everything is created with no errors):

resource "aws_iam_role" "rds" {
  name               = "rds-${var.environment}"
  assume_role_policy = "${data.aws_iam_policy_document.sts-assume-role.json}"

  provisioner "local-exec" {
    # I wouldn't go for less than 10. I tried saving a few seconds and failed. :/
    command = "sleep 10"
  }
}

resource "aws_db_option_group" "mssql" {
  name                 = "mssql-web-14"
  engine_name          = "sqlserver-web"
  major_engine_version = "14.00"

  option {
    option_name = "SQLSERVER_BACKUP_RESTORE"

    option_settings {
      name  = "IAM_ROLE_ARN"
      value = "${aws_iam_role.rds.arn}"
    }
  }

  depends_on = ["aws_iam_role.rds"]
}

[im-dofoborh]

@antgel adapted your approach. works a treat. thanks.

[Sammmmm]

I dont see sleep as a long term reliable option, it would be nice if depends on worked here.

[bflad]

The fix for this has been merged into master and will release with version 1.30.0 of the AWS provider, likely on Wednesday. 👍

[bflad]

This has been released in version 1.30.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!