r/aws_kinesis_firehose_delivery_stream: Delivery to Amazon Elasticsearch Service domain in VPC

[ewbankkit]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Amazon Kinesis Data Firehose can now deliver streaming data to an Amazon Elasticsearch Service domain in an Amazon VPC.

New or Affected Resource(s)

• aws_kinesis_firehose_delivery_stream

Potential Terraform Configuration

resource "aws_kinesis_firehose_delivery_stream" "example" {
  elasticsearch_configuration {
    vpc_configuration {
      subnet_ids = ["..."]
      security_group_ids = ["..."]
      role_arn = "..."
    }
  }
}

References

Announcement.

Requires AWS SDK v1.30.13:

• Update module aws/aws-sdk-go to v1.30.16 #12970

[martin-paulus]

Consistentcy note: The aws_elasticsearch_domain resource uses a vpc_options block. The aws_lambda_function resource uses a vpc_config block.

Please consider one of these two options to prevent introducing a third label. Hopefully, in time, we can use only one label for this type of block (in AWS resources).

[martin-paulus]

Prerequisite #12970 has been merged, and then released in v2.60.0 of the AWS provider.

[rajholla]

I just started working on this. I will link the PR once I am done.

[ddiawara]

still have the same error even if update provider

provider "aws" {
  version = "~> 2.60.0" # "~> 2.32"
  alias   = "ireland"
  region  = "eu-west-1"
  assume_role {
    role_arn = "arn:aws:iam::${var.aws_account}:role/ts-terraformRole"
  }
}

[martin-paulus]

@ddiawara Sure, because "aws" provider version 2.60.0 contains support for the AWS SDK version that supports VPC configuration. But the Terraform support for VPC configuration is present in the PR that @rajholla prepared. That PR is still unmerged.

[ddiawara]

is it possible to use it now while waiting for the merge to be accepted.
I have updated the provider 2.60.0 version and I have implemented the Aws-sdk version but I keep getting the same error message.

[martin-paulus]

@ddiawara In order to use (test) @rajholla 's contribution before it's merged, I believe you should follow this guide: https://www.terraform.io/docs/extend/writing-custom-providers.html

First, fetch the sources from: https://github.com/rajholla/terraform-provider-aws/tree/firehose-es-vpc-support
Ensure you checkout the firehose-es-vpc-support branch.

Then build the Terraform provider using the Writing Custom Providers guide: https://www.terraform.io/docs/extend/writing-custom-providers.html#building-the-plugin

Note: The essential changes are in: https://github.com/rajholla/terraform-provider-aws/blob/firehose-es-vpc-support/aws/resource_aws_kinesis_firehose_delivery_stream.go
But I am not sure whether you can stop after building this, or whether you should build the entire aws provider. (EDIT: It seems like you have to build the entire directory)

After a successful build, don't forget to install the "custom provider": https://www.terraform.io/docs/configuration/providers.html#third-party-plugins

Note: Terraform detects the provider version based on its filename: https://www.terraform.io/docs/configuration/providers.html#plugin-names-and-versions

And finally, you should configure your Terraform configuration to select your custom provider.

[martin-paulus]

Thank you @rajholla , I just now successfully deployed a Kinesis Firehose delivery stream with VPC access!

[ddiawara]

Thank you @rajholla , I just now successfully deployed a Kinesis Firehose delivery stream with VPC access!

Yo yeah what do you do because it doesn't work for me
this is what is do :

mkdir -p $HOME/development/terraform-providers/; cd $HOME/development/terraform-providers/
git clone https://github.com/rajholla/terraform-provider-aws.git
cd terraform-provider-aws && git checkout firehose-es-vpc-support
make tools
make build
cp $GOPATH/bin/terraform-provider-aws ~/.terraform.d/plugins
# in a terraform code

terraform init

terraform apply 

i don't know what i forget

[martin-paulus]

@ddiawara Please try renaming your provider binary with a filename that includes a version. Also, consider configuring Terraform to use aws as the provider name for your custom binary.

This is what I did:

git clone [email protected]:rajholla/terraform-provider-aws.git
cd terraform-provider-aws/
git checkout firehose-es-vpc-support

docker run -it --rm -v "$PWD":/usr/local/src/tf-provider -w /usr/local/src/tf-provider golang go build -o terraform-provider-awsCustomMain -v

mkdir ~/.terraform.d/plugins
cp terraform-provider-awsCustomMain ~/.terraform.d/plugins/terraform-provider-aws_v2.63.0

Note: Here I used a Docker container to compile the sources, if you like you could instead use a native golang compiler which you installed.

Select the custom build aws provider in Terraform:

terraform {
  required_providers {
    aws = "2.63.0"
  }
}

Finally, run:

terraform init
terraform apply

[martin-paulus]

@ddiawara Your VPC config block is misnamed, it should look something like:

resource "aws_kinesis_firehose_delivery_stream" "es-delivery" {
  elasticsearch_configuration {
    vpc_config {
      role_arn           = aws_iam_role.firehose-vpc.arn
      security_group_ids = [aws_security_group.firehose.id]
      subnet_ids         = slice(local.vpc.private_subnets, 0, var.es_instance_count)
    }
  }
}

So use vpc_config instead of vpc_configuration.

[ddiawara]

I thought I was going crazy, haahhaha thank you for your excellent work.

Creating for me
Awesome ::: = Process finished with exit code 0
Nice thanks all

[ghost]

Hi,
Any timeline when vpc_config will be released for elasticsearch_configuration ?

[Kevc1]

Is vpc_config in any currently available terraform version so I don't need to compile my own version ?
Thanks

[ghost]

Do we know when VPC block will be available for ES configuration?

[bflad]

Support for this functionality has been merged and will release with version 3.5.0 of the Terraform AWS Provider, later this week. Thanks to @rajholla for the implementation. 👍

[ghost]

This has been released in version 3.5.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!