Unable to remove aws_kinesis_firehose_delivery_stream S3 processors

[camlow325]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.12.18
+ provider.archive v1.3.0
+ provider.aws v2.42.0

Affected Resource(s)

• aws_kinesis_firehose_delivery_stream

Terraform Configuration Files

resource "aws_s3_bucket" "test" {
  bucket = "test"
}

resource "aws_iam_role" "test" {
  assume_role_policy = data.aws_iam_policy_document.assume_role.json
}

data "aws_iam_policy_document" "assume_role" {
  statement {
    actions = [
      "sts:AssumeRole",
    ]

    principals {
      type = "Service"
      identifiers = [
        "firehose.amazonaws.com",
      ]
    }

    principals {
      type = "Service"
      identifiers = [
        "lambda.amazonaws.com",
      ]
    }

    effect = "Allow"
  }
}

data "archive_file" "lambda_zip" {
  type        = "zip"
  output_path = "/tmp/lambda.zip"
  source {
    content  = <<EOF
def lambda_handler(event,context):
    print("hello world")
EOF
    filename = "main.py"
  }
}
resource "aws_lambda_function" "test" {
  function_name    = "test"
  handler          = "main.lambda_handler"
  filename         = data.archive_file.lambda_zip.output_path
  source_code_hash = data.archive_file.lambda_zip.output_base64sha256
  role             = aws_iam_role.test.arn
  runtime          = "python3.6"
}

resource "aws_kinesis_firehose_delivery_stream" "test" {
  name        = "test"
  destination = "extended_s3"

  extended_s3_configuration {
    bucket_arn = aws_s3_bucket.test.arn
    role_arn   = aws_iam_role.test.arn
    processing_configuration {
      enabled = false
//  Uncomment the lines below for one apply then comment them back out for the next apply
//      processors {
//        type = "Lambda"
//        parameters {
//          parameter_name  = "LambdaArn"
//          parameter_value = aws_lambda_function.test.arn
//        }
//      }
    }
  }
}

Debug Output

Panic Output

Expected Behavior

When a terraform apply is attempted with no processors defined in the extended_s3_configuration.processing_configuration for the aws_kinesis_firehose_delivery_stream resource, previously applied processors should be removed and the apply should succeed.

Actual Behavior

The terraform apply with no processors defined fails with the following error:

aws_kinesis_firehose_delivery_stream.test: Modifying... [id=arn:aws:firehose:xxx:deliverystream/test]
                                                                                                           
Error: Error Updating Kinesis Firehose Delivery Stream: "test"
InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, UpdateDestinationInput.ExtendedS3DestinationUpdate.ProcessingConfiguration.Processors[0].Parameters[0].ParameterValue.

  on test.tf line 84, in resource "aws_kinesis_firehose_delivery_stream" "test":
  84: resource "aws_kinesis_firehose_delivery_stream" "test" {

Repeated terraform apply operations produce the same error. While in the error state, the Terraform state file contains the following content for the processing_configuration block, which appears to be invalid per the resource schema:

"processing_configuration": [
{
  "enabled": false,
  "processors": [
    {
      "parameters": [
        {
          "parameter_name": "",
          "parameter_value": ""
        }
      ],
      "type": ""
    }
  ]
}

The processors can be cleared by doing the following AWS CLI command:

aws firehose update-destination --delivery-stream-name test --cli-input-json file://test.json

The test.json file has the following content:

{
  "CurrentDeliveryStreamVersionId": "3",
  "DestinationId": "destinationId-000000000001",
  "ExtendedS3DestinationUpdate": {
    "ProcessingConfiguration": {
      "Enabled": false,
      "Processors": []
    }
  }
}

Once the processors have been cleared via the AWS CLI command, subsequent terraform apply commands with the processors still omitted from the configuration are successful.

Steps to Reproduce

1. Ensure that the extended_s3_configuration.processing_configuration block above is defined as follows:

   processing_configuration {
     enabled = false
   }

2. terraform apply

   The apply should be successful.

3. Modify the extended_s3_configuration.processing_configuration block above to the following:

   processing_configuration {
     enabled = false
     processors {
       type = "Lambda"
       parameters {
         parameter_name  = "LambdaArn"
         parameter_value = aws_lambda_function.test.arn
       }
     }
   }

4. terraform apply

   The apply should be successful.

5. Modify the extended_s3_configuration.processing_configuration block to have the same content as had been set in the first step:

   processing_configuration {
     enabled = false
   }

6. terraform apply

   This terraform apply encounters the error described in the Actual Behavior section above.

Important Factoids

References

[bflad]

The fix for this has been merged and will release with version 2.48.0 of the Terraform AWS Provider, later today. Thanks to @camlow325 for the implementation. 👍

[ghost]

This has been released in version 2.48.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!