Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add SNS subscription filter #2806

Merged
merged 7 commits into from
Feb 7, 2018
Merged

Add SNS subscription filter #2806

merged 7 commits into from
Feb 7, 2018

Conversation

yohei1126
Copy link
Contributor

@yohei1126 yohei1126 commented Dec 31, 2017 •
edited
Loading

fixes #2554

https://docs.aws.amazon.com/sns/latest/dg/message-filtering.html

Applying a Filter Policy with the AWS CLI

$ aws sns set-subscription-attributes --subscription-arn arn:aws:sns: ... --attribute-name FilterPolicy --attribute-value '{"store":["example_corp"],"event":["order_placed"]}'
$ aws sns get-subscription-attributes --subscription-arn arn:aws:sns: ...
{
    "Attributes": {
        "Endpoint": "endpoint . . .", 
        "Protocol": "https",
        "RawMessageDelivery": "false", 
        "EffectiveDeliveryPolicy": "delivery policy . . .",
        "ConfirmationWasAuthenticated": "true", 
        "FilterPolicy": "{\"store\": [\"example_corp\"], \"event\": [\"order_placed\"]}", 
        "Owner": "111122223333", 
        "SubscriptionArn": "arn:aws:sns: . . .", 
        "TopicArn": "arn:aws:sns: . . ."
    }
}

Acceptance test for this has passed.

$ make testacc TESTARGS="-run=TestAccAWSSNSTopicSubscription_attributes"
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test $(go list ./... |grep -v 'vendor') -v -run=TestAccAWSSNSTopicSubscription_attributes -timeout 120m
?       github.com/terraform-providers/terraform-provider-aws   [no test files]
=== RUN   TestAccAWSSNSTopicSubscription_attributes
--- PASS: TestAccAWSSNSTopicSubscription_attributes (41.49s)
PASS
ok      github.com/terraform-providers/terraform-provider-aws/aws       41.526s

@yohei1126
Copy link
Contributor Author

Could anyone review this PR?

@radeksimko radeksimko added the enhancement Requests to existing resources that expand the functionality or scope. label Jan 2, 2018
@yohei1126
Copy link
Contributor Author

acceptance test has not passed yet

$ make testacc TESTARGS="-run=TestAccAWSSNSTopicSubscription_attributes"
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test $(go list ./... |grep -v 'vendor') -v -run=TestAccAWSSNSTopicSubscription_attributes -timeout 120m
?       github.com/terraform-providers/terraform-provider-aws   [no test files]
=== RUN   TestAccAWSSNSTopicSubscription_attributes
--- FAIL: TestAccAWSSNSTopicSubscription_attributes (3.77s)
        testing.go:503: Step 0 error: Error loading configuration: Error parsing /var/folders/py/bt32yyqd33zg_gtj9nx6b6w80000gn/T/tf-test622793941/main.tf: At 14:12: Unknown token: 14:12 IDENT _policy
FAIL
FAIL    github.com/terraform-providers/terraform-provider-aws/aws       3.816s
make: *** [testacc] Error 1

@yohei1126
Copy link
Contributor Author

@radeksimko Would you help me fix this?

@atsushi-ishibashi
Copy link
Contributor

@yohei1126 There are helpful things in delivery_policy of resourceAwsSnsTopic

@yohei1126
Copy link
Contributor Author

yohei1126 commented Jan 5, 2018 •
edited
Loading

Thanks. I fixed acc test. delivery_policy in resourceAwsSnsTopic has much more properties than one in resourceAwsSnsTopicSubscription. Should I implement properties other than Type and Optional?

"delivery_policy": &schema.Schema{
    Type:             schema.TypeString,
    Optional:         true,
    ForceNew:         false,
    ValidateFunc:     validateJsonString,
    DiffSuppressFunc: suppressEquivalentJsonDiffs,
    StateFunc: func(v interface{}) string {
        json, _ := normalizeJsonString(v)
	return json
    },
}

@atsushi-ishibashi
Copy link
Contributor

Probably you should set the below theree

ValidateFunc:     validateJsonString,
DiffSuppressFunc: suppressEquivalentJsonDiffs,
StateFunc: func(v interface{}) string {
  json, _ := normalizeJsonString(v)
  return json
},

@yohei1126
Copy link
Contributor Author

@atsushi-ishibashi Thanks. I add some attributes you mentioned above.

@yohei1126
Copy link
Contributor Author

@radeksimko Would you review this PR?

@yohei1126
Copy link
Contributor Author

@radeksimko Would you check this? All of existing and new acceptance tests have already passed. Everything works well.

@matelang
Copy link

Do you guys have an estimate when this is going live? We are in desperate need for it 👍

@yohei1126
Copy link
Contributor Author

@radeksimko Everybody needs this feature! Would you review this PR?

@yohei1126
Copy link
Contributor Author

@bflad Could you review this?

@yohei1126
Copy link
Contributor Author

@radeksimko Would you review this PR?

@bflad bflad added the service/sns Issues and PRs that pertain to the sns service. label Jan 16, 2018
@bflad
Copy link
Contributor

bflad commented Jan 16, 2018

Hi, everyone. Just to provide an update here, we are actively working on only crash and bug fixes this week for a v1.7.1 release. I cannot guarantee a timeframe for review/acceptance of this PR until after then, but we do see this PR and will get to it.

PS: Please vote on issues/PRs with 👍 reactions on the original post. We use this to help us prioritize our work.

@yohei1126
Copy link
Contributor Author

@bflad Thanks. I am waiting for your review. Please let me know if I can help you.

@yohei1126
Copy link
Contributor Author

@bflad any update?

@yohei1126
Copy link
Contributor Author

It would be great if this PR is merged this month :)

@yohei1126
Copy link
Contributor Author

@bflad could you check it?

1 similar comment
@yohei1126
Copy link
Contributor Author

@bflad could you check it?

bflad
bflad reviewed Feb 6, 2018
View reviewed changes
Copy link
Contributor

@bflad bflad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yohei1126 this is off to a great start! Thanks for submitting this functionality, for which lots of people are excited.

Please see my feedback below (mostly just testing related), letting me know if you have any questions and when its good to look at again or if you do not have time to address these! 🚀

aws/resource_aws_sns_topic_subscription.go Outdated
_, err := snsconn.SetSubscriptionAttributes(req)

if err != nil {
return fmt.Errorf("Unable to set filter policy attribute on subscription")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick: We should probably return the err as well so people know why this happened 😄
return fmt.Errorf("Unable to set filter policy attribute on subscription: %s", err)

aws/resource_aws_sns_topic_subscription_test.go Outdated
@@ -31,6 +31,25 @@ func TestAccAWSSNSTopicSubscription_basic(t *testing.T) {
})
}

func TestAccAWSSNSTopicSubscription_attributes(t *testing.T) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please adjust this acceptance test:

  • Rename to _filterPolicy (there are other attributes we can/should be testing)
  • Add an update step and verify attributes

For the second item, you can check out TestAccAWSIAMUserPolicy_multiplePolicies for an example of this in action.

aws/resource_aws_sns_topic_subscription_test.go Outdated
@@ -157,6 +207,25 @@ resource "aws_sns_topic_subscription" "test_subscription" {
`, i, i)
}

func testAccAWSSNSTopicSubscriptionConfig_attributes(i int) string {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In line with renaming the test itself, we should update this as well please: _filterPolicy

aws/resource_aws_sns_topic_subscription_test.go Outdated
topic_arn = "${aws_sns_topic.test_topic.arn}"
protocol = "sqs"
endpoint = "${aws_sqs_queue.test_queue.arn}"
filter_policy = "{\"key1\": [\"val1\"], \"key2\": [\"val2\"]}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you parameterize the filter_policy here so we can easily add an update step to the acceptance testing? filter_policy = %s

We can easily handle the extra quoting conversions (and checking the attribute is properly set to the right value in the state) with strconv.Quote() in the test function. Check out TestAccAWSIAMUserPolicy_multiplePolicies for an example of this in action. 😄

aws/resource_aws_sns_topic_subscription_test.go Outdated
@@ -124,6 +143,37 @@ func testAccCheckAWSSNSTopicSubscriptionExists(n string) resource.TestCheckFunc
}
}

func testAccCheckAWSSNSTopicSubscriptionAttributesExists(n string) resource.TestCheckFunc {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function is unnecessary, see above. 😄

aws/resource_aws_sns_topic_subscription_test.go Outdated
Check: resource.ComposeTestCheckFunc(
testAccCheckAWSSNSTopicExists("aws_sns_topic.test_topic"),
testAccCheckAWSSNSTopicSubscriptionExists("aws_sns_topic_subscription.test_subscription"),
testAccCheckAWSSNSTopicSubscriptionAttributesExists("aws_sns_topic_subscription.test_subscription"),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be done much easier with using some of our acceptance testing helper functions. Can you please switch them over to something like this (once the filter_policy is parameterized in the configuration)?

resource.TestCheckResourceAttr("aws_sns_topic_subscription", "filter_policy", strconv.Quote(filterPolicy)),

@bflad bflad added the waiting-response Maintainers are waiting on response from community or contributor. label Feb 6, 2018
This was referenced Feb 6, 2018
Closed
Closed
@ghost ghost added the size/M Managed by automation to categorize the size of a PR. label Feb 7, 2018
@yohei1126
Copy link
Contributor Author

Sorry. The test failed.

68.66s$ make test
==> Checking that code complies with gofmt requirements...
go test ./... -timeout=30s -parallel=4
# github.com/terraform-providers/terraform-provider-aws/aws
aws/resource_aws_sns_topic_subscription.go:86:17: undefined: normalizeJsonString
# github.com/terraform-providers/terraform-provider-aws/aws
aws/resource_aws_sns_topic_subscription.go:86:17: undefined: normalizeJsonString
make: *** [test] Error 2

@ghost ghost added the size/M Managed by automation to categorize the size of a PR. label Feb 7, 2018
@yohei1126
Copy link
Contributor Author 

$ make test
==> Checking that code complies with gofmt requirements...
go test ./... -timeout=30s -parallel=4
# github.com/terraform-providers/terraform-provider-aws/aws
aws/resource_aws_sns_topic_subscription.go:87:17: cannot refer to unexported name structure.normalizeJsonString
aws/resource_aws_sns_topic_subscription.go:87:17: undefined: structure.normalizeJsonString
# github.com/terraform-providers/terraform-provider-aws/aws
aws/resource_aws_sns_topic_subscription.go:87:17: cannot refer to unexported name structure.normalizeJsonString
aws/resource_aws_sns_topic_subscription.go:87:17: undefined: structure.normalizeJsonString
make: *** [test] Error 2

@yohei1126
Copy link
Contributor Author

@bflad I fixed the test. Could you check it again?

@bflad bflad removed the waiting-response Maintainers are waiting on response from community or contributor. label Feb 7, 2018
bflad
bflad approved these changes Feb 7, 2018
View reviewed changes
Copy link
Contributor

@bflad bflad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM -- thanks for the extra effort, very clean! 🚀

make testacc TEST=./aws TESTARGS='-run=TestAccAWSSNSTopicSubscription'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -run=TestAccAWSSNSTopicSubscription -timeout 120m
=== RUN   TestAccAWSSNSTopicSubscription_importBasic
--- PASS: TestAccAWSSNSTopicSubscription_importBasic (17.75s)
=== RUN   TestAccAWSSNSTopicSubscription_basic
--- PASS: TestAccAWSSNSTopicSubscription_basic (15.72s)
=== RUN   TestAccAWSSNSTopicSubscription_filterPolicy
--- PASS: TestAccAWSSNSTopicSubscription_filterPolicy (28.20s)
=== RUN   TestAccAWSSNSTopicSubscription_autoConfirmingEndpoint
--- PASS: TestAccAWSSNSTopicSubscription_autoConfirmingEndpoint (45.97s)
=== RUN   TestAccAWSSNSTopicSubscription_autoConfirmingSecuredEndpoint
--- PASS: TestAccAWSSNSTopicSubscription_autoConfirmingSecuredEndpoint (75.95s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	183.649s

@bflad bflad added this to the v1.9.0 milestone Feb 7, 2018
@bflad bflad merged commit 02b2848 into hashicorp:master Feb 7, 2018
bflad added a commit that referenced this pull request Feb 7, 2018
@bflad
Update CHANGELOG for #2806
47f1091
@yohei1126 yohei1126 deleted the sub-attributes branch February 7, 2018 22:01
@bflad
Copy link
Contributor

bflad commented Feb 9, 2018

This has been released in terraform-provider-aws version 1.9.0. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

@yohei1126 yohei1126 mentioned this pull request Feb 10, 2018
Merged
@yohei1126
Copy link
Contributor Author

@bflad Sorry. I forgot to update the document. I sent another PR to update the document. #3322

@oanasabau
Copy link

We found a problem with the functionality and I've created an issue for it
#3328

@ghost
Copy link

ghost commented Apr 8, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators Apr 8, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@bflad bflad bflad approved these changes

Assignees
No one assigned
Labels
enhancement Requests to existing resources that expand the functionality or scope. service/sns Issues and PRs that pertain to the sns service. size/M Managed by automation to categorize the size of a PR.
Projects
None yet
Milestone
v1.9.0
Development

Successfully merging this pull request may close these issues.

How to apply AWS SNS/SQS subscription filter
6 participants
@yohei1126 @atsushi-ishibashi @matelang @bflad @oanasabau @radeksimko