[New Resource]: OpenSearch Ingestion Pipeline

[farhanangullia]

Description

Amazon OpenSearch Ingestion is a fully managed, serverless data collector that delivers real-time log, metric, and trace data to Amazon OpenSearch Service domains and OpenSearch Serverless collections.

Requested Resource(s) and/or Data Source(s)

• aws_osis_pipeline

Potential Terraform Configuration

resource "aws_osis_pipeline" "example" {
  name                        = "example"
  max_units                   = 5
  min_units                   = 1
  pipeline_configuration_body = file("${path.module}/pipeline.yaml")
  log_publishing_options {
    cloudwatch_log_destination {
      log_group = "/aws/OpenSearchService/IngestionService/my-pipeline"
    }
    is_logging_enabled = true
  }
}

References

https://pkg.go.dev/github.com/aws/[email protected]/service/osis

Would you like to implement a fix?

Yes

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[lawliet89]

awscc_osis_pipeline is available for use in the meantime.

[komgritz]

I am waiting for available on hashicorp/aws provider.

[DanielViglione]

I do not recommend the “awscc_osis_pipeline” resource. The awscc provider is pretty inflexible. If you use awscc_osis_pipeline, you will feel compelled to use awscc_iam_role_policy and awscc_iam_role, for example. awscc_iam_role doesn’t work like aws_iam_role with regards to tags, the outputs are named differently and worst of all the awscc_iam_role_policy only accepts a policy document as a map of strings, so you cannot pass it a json string, which makes it unsuitable for workloads where you want to create reusable modules and pass policies as json strings.

[HaroonSaid]

Any progress?

[edmondsiu0]

I echo @DanielViglione's opinion on not recommending awscc_osis_pipeline but for reason of access control / security.

Access control to AWS resources through the AWS Cloud Control API is very crude (pun intended) - while you can restrict actions based on Create / Read / Update / Delete or List, this access is not granular and is given to all resources within the AWS account, AWS CC does not currently support resource-level permissions or conditions.

Reference: https://docs.aws.amazon.com/cloudcontrolapi/latest/userguide/security.html

To me, creating resources with AWS Cloud Control API is like sudo vim'ing a config file as root.
Basically throwing the principle of least privilege out of the window with this resource.

We're in the process of evaluating the use of an alternative provider:
https://registry.terraform.io/providers/opensearch-project/opensearch/latest/docs/resources/ingest_pipeline

^ as @cameronattard pointed out this is something else that cannot substitute AWS Ingestion Pipeline.
(My understanding is that ^ is a pipeline within OS which applies processors to data after they've been ingested into OS, thus irrelevant)

[cameronattard]

We're in the process of evaluating the use of an alternative provider: https://registry.terraform.io/providers/opensearch-project/opensearch/latest/docs/resources/ingest_pipeline

FYI that is for a totally separate thing from AWS OpenSearch Ingestion Pipelines - https://opensearch.org/docs/latest/ingest-pipelines/create-ingest/

[thunder-spb]

Any updates on this?

[praddc]

Just curious if there are updates on this. Thanks!

[farhanangullia]

hi folks, i'll be raising the PR for this tomorrow.

[kamalika-majumder]

Will this be included in the next release of terraform provider for aws ?

[github-actions]

This functionality has been released in v5.36.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.