Add Support for Filtering AWS Namespaces

[brandonstevens]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Add support for filtering AWS namespaces (e.g. Lambda) using tags. It's currently only possible to limit resource collection using tags for Lambdas in the UI or through an undocumented API /api/v1/integration/aws/filtering. For organizations with a large number of Datadog accounts, limited resource collection by requiring an opt-in tag makes it a lot easier to manage Datadog costs. Making changes across a large number of Datadog accounts is tedious. Filters should be configurable with Terraform.

New or Affected Resource(s)

• datadog_integration_aws_filters
  or
• filters block inside datadog_integration_aws resource

Potential Terraform Configuration

resource "datadog_integration_aws" "sandbox" {
    account_id = "1234567890"
    role_name = "DatadogAWSIntegrationRole"
    filter_tags = ["key:value"]
    host_tags = ["key:value", "key2:value2"]
    account_specific_namespace_rules = {
        auto_scaling = false
        opsworks = false
    }
    excluded_regions = ["us-east-1", "us-west-2"]
    filters = {
        lambda = ["key:value"]
    }
}

References

$ curl -sX GET https://api.datadoghq.com/api/v1/integration/aws/filtering?account_id=XXXXXXXXXXXX -H "Content-Type: application/json" -H "DD-API-KEY: XXXXXXXXXXXX" -H "DD-APPLICATION-KEY: XXXXXXXXXXXX"

{
  "filters": [
    {
      "tag_filter_str": null,
      "namespace": "elb"
    },
    {
      "tag_filter_str": null,
      "namespace": "application_elb"
    },
    {
      "tag_filter_str": null,
      "namespace": "sqs"
    },
    {
      "tag_filter_str": null,
      "namespace": "rds"
    },
    {
      "tag_filter_str": null,
      "namespace": "custom"
    },
    {
      "tag_filter_str": null,
      "namespace": "network_elb"
    },
    {
      "tag_filter_str": "monitoring:true",
      "namespace": "lambda"
    }
  ]
}

[johnkost]

I'm interested in working on this

[johnkost]

This repository uses datadog-api-client-go library for all API interactions. I looked into adding the new endpoints to that but that repository is pretty much entirely auto-generated (only adding tests and documentation). I wasn't able to find where to make the updates so they are autogenerated. @therve - looks like you have interacted with both repositories recently. Do you know how I can get the new endpoints into the client library so I can add the feature to this repository?

[therve]

Hi @johnkost

I've talked to the backend team about that, it's on them to produce the API spec for their endpoint. Once we have that, we can autogenerate the client and add the feature to Terraform. I unfortunately don't have a timeline currently :/

[johnkost]

Thanks for checking. I'll keep an eye out on the client.

[brandonstevens]

@johnkost, looks like the the dependent changes have been merged in (see DataDog/datadog-api-client-go#589) and vendored (see #773). Are you still interested in contributing the changes to the provider?

[johnkost]

Yes I am. I'll go ahead and start working on it

[johnkost]

I looked into this over the weekend and I don't think the provider is the right place for this to live. It should be done by the API. The provider is currently setting the mapping it has of account_specific_namespace_rules to what is provided when it tries to create/modify the Integration and passes that to the API (https://github.com/DataDog/terraform-provider-datadog/blob/master/datadog/resource_datadog_integration_aws.go#L105). The API isn't handling it as we would expect.

I think we should open up a bug/feature request with DataDog and ask them to fix/modify their API rather than try and shim this in to the provider. What are your thoughts?

[skarimo]

@johnkost I think there might be a confusion between account_specific_namespace_rules vs the tag filtering rules which can be set via the endpoint integration/aws/filtering. account_specific_namespace_rules enables/disables metric collection for a given AWS namespaces. Whereas tag filtering rules set via the endpoint integration/aws/filtering apply tag filtering on a given AWS namespace.

For example, below would collect data on application_elb namespace resources that has the tag monitored:true.

curl -X POST 'https://app.datadoghq.com/api/v1/integration/aws/filtering?api_key=<API_KEY>&application_key=<APPLICATION_KEY>' --data '{"account_id": "<AWS_ACCOUNT_ID>", "namespace": "application_elb", "tag_filter_str": "monitored:true"}'  -H "Content-Type: text/plain"

Whereas if we were to apply the following, no application_elb would be collected:

resource "datadog_integration_aws" "sandbox" {
    ...
    account_specific_namespace_rules = {
        application_elb = false
    }
}

So ultimately, account_specific_namespace_rules and integration/aws/filtering are different and are set via different endpoint.

What do you think?

[johnkost]

Ah - that makes sense. Thanks for clarifying 😄

I can take a stab at this over the weekend/next week