GH-1294 Add support to FIFO #3437
Conversation
lillo42
requested review from
iancooper,
holytshirt,
DevJonny and
preardon
as code owners
|
Hi @lillo42 - thanks for this. To lift out of inline above, the usual strategy would be: Folders for each queue type:
Put the existing tests in Standard and duplicate the tests in Fifo, but with the changed semantics. I tend to prefer being explicit in tests, over worrying about duplication there. |
|
PS @lillo42 if you have time. One weakness in Brighter is that you have to produce to SNS and can't target a queue. It would be good to allow the queue to be the target. We now do something similar for ASB. Probably pop that in a separate PR though? |
tests/Paramore.Brighter.AWS.Tests/MessagingGateway/When_infastructure_exists_can_assume.cs
Outdated
Show resolved
Hide resolved
Sure no problem, I can add it to this PR. |
|
Thanks @lillo42. Apologies in advance for the merge issues caused by the Reactor(Blocking)/Proactor(Non-Blocking) enhancement |
… localstack and rename SQS to SNS
|
We still have a credentials issue on the AWS tests, that means they don't show up here. I need to fix that too. Let me see if I can fix that and merge to main, so that we can at least see what is passing |
| RoutingKey topic, | ||
| TopicFindBy topicFindBy, | ||
| SnsAttributes? attributes, | ||
| OnMissingChannel makeTopic = OnMissingChannel.Create, |
There was a problem hiding this comment.
Can we make makeTopic the last parameter? That keeps it in sync with other producers
src/Paramore.Brighter.MessagingGateway.AWSSQS/SnsMessageProducer.cs
Outdated
Show resolved
Hide resolved
src/Paramore.Brighter.MessagingGateway.AWSSQS/SqsMessageConsumer.cs
Outdated
Show resolved
Hide resolved
| namespace System.Diagnostics.CodeAnalysis; | ||
|
|
||
| [AttributeUsage(AttributeTargets.Method | AttributeTargets.Property, Inherited = false, AllowMultiple = true)] | ||
| public sealed class MemberNotNullWhenAttribute : Attribute |
There was a problem hiding this comment.
Out of interest, we have an open conversation on using polyfills vs dropping netstandard20
There was a problem hiding this comment.
I tend to prefer drop support for netstandard20, but depending on how many downloads we have today on .NET standard maybe we still have people using Brighter with .NET Framework 4
Maybe on V11 we can remove the support for .NET Standard completed, and put a warning on V10 saying that is the last version of supporting .NET Standard/Framework
There was a problem hiding this comment.
Yeah, you may be right that it would be 0-warning if we dropped it with V10 and we should declare that will happen with V11.
![codescene-delta-analysis[bot]](https://avatars.githubusercontent.com/in/53921?s=60&v=4){kind=small}
| public Message CreateMessage(Amazon.SQS.Model.Message sqsMessage) | ||
| { | ||
| var topic = HeaderResult<RoutingKey>.Empty(); | ||
| var messageId = HeaderResult<string?>.Empty(); | ||
|
|
||
| Message message; | ||
| try |
There was a problem hiding this comment.
❌ Getting worse: Complex Method
CreateMessage increases in cyclomatic complexity from 10 to 12, threshold = 9
| publishRequest.MessageAttributes = messageAttributes; | ||
|
|
||
| var response = await _client.PublishAsync(publishRequest); | ||
| if (response.HttpStatusCode == System.Net.HttpStatusCode.OK || response.HttpStatusCode == System.Net.HttpStatusCode.Created || response.HttpStatusCode == System.Net.HttpStatusCode.Accepted) |
There was a problem hiding this comment.
✅ No longer an issue: Complex Conditional
PublishAsync no longer has a complex conditional
| private HeaderResult<RoutingKey> ReadTopic() | ||
| { | ||
| if (_messageAttributes.TryGetValue(HeaderNames.Topic, out var topicArn)) |
There was a problem hiding this comment.
❌ New issue: Bumpy Road Ahead
ReadTopic has 2 blocks with nested conditional logic. Any nesting of 2 or deeper is considered. Threshold is one single, nested block per function
|
|
||
| public Message CreateMessage(Amazon.SQS.Model.Message sqsMessage) |
There was a problem hiding this comment.
❌ Getting worse: Complex Method
CreateMessage increases in cyclomatic complexity from 10 to 13, threshold = 9
| private static HeaderResult<RoutingKey> ReadTopic(Amazon.SQS.Model.Message sqsMessage) | ||
| { | ||
| if (sqsMessage.MessageAttributes.TryGetValue(HeaderNames.Topic, out MessageAttributeValue? value)) | ||
| { | ||
| //we have an arn, and we want the topic | ||
| var topic = value.StringValue; | ||
| if (Arn.TryParse(value.StringValue, out var arn)) | ||
| { | ||
| topic = arn.Resource; | ||
| } | ||
| else |
There was a problem hiding this comment.
❌ New issue: Bumpy Road Ahead
ReadTopic has 2 blocks with nested conditional logic. Any nesting of 2 or deeper is considered. Threshold is one single, nested block per function
| public SqsSubscription( | ||
| Type dataType, | ||
| SubscriptionName? name = null, | ||
| ChannelName? channelName = null, | ||
| RoutingKey? routingKey = null, | ||
| int bufferSize = 1, | ||
| int noOfPerformers = 1, | ||
| TimeSpan? timeOut = null, | ||
| int requeueCount = -1, | ||
| TimeSpan? requeueDelay = null, | ||
| int unacceptableMessageLimit = 0, | ||
| MessagePumpType messagePumpType = MessagePumpType.Proactor, | ||
| IAmAChannelFactory? channelFactory = null, | ||
| int lockTimeout = 10, | ||
| int delaySeconds = 0, | ||
| int messageRetentionPeriod = 345600, | ||
| TopicFindBy findTopicBy = TopicFindBy.Name, | ||
| string? iAmPolicy = null, | ||
| RedrivePolicy? redrivePolicy = null, | ||
| SnsAttributes? snsAttributes = null, | ||
| Dictionary<string, string>? tags = null, | ||
| OnMissingChannel makeChannels = OnMissingChannel.Create, | ||
| bool rawMessageDelivery = true, | ||
| TimeSpan? emptyChannelDelay = null, | ||
| TimeSpan? channelFailureDelay = null, | ||
| SnsSqsType sqsType = SnsSqsType.Standard, | ||
| bool contentBasedDeduplication = true, | ||
| DeduplicationScope? deduplicationScope = null, | ||
| int? fifoThroughputLimit = null, | ||
| RoutingKeyType routingKeyType = RoutingKeyType.PubSub, | ||
| QueueFindBy queueFindBy = QueueFindBy.Name | ||
| ) | ||
| : base(dataType, name, channelName, routingKey, bufferSize, noOfPerformers, timeOut, requeueCount, | ||
| requeueDelay, unacceptableMessageLimit, messagePumpType, channelFactory, makeChannels, emptyChannelDelay, | ||
| channelFailureDelay) |
There was a problem hiding this comment.
ℹ Getting worse: Constructor Over-Injection
SqsSubscription increases from 24 to 30 arguments, threshold = 5
| public SqsSubscription( | ||
| SubscriptionName? name = null, | ||
| ChannelName? channelName = null, | ||
| RoutingKey? routingKey = null, | ||
| int bufferSize = 1, | ||
| int noOfPerformers = 1, | ||
| TimeSpan? timeOut = null, | ||
| int requeueCount = -1, | ||
| TimeSpan? requeueDelay = null, | ||
| int unacceptableMessageLimit = 0, | ||
| MessagePumpType messagePumpType = MessagePumpType.Proactor, | ||
| IAmAChannelFactory? channelFactory = null, | ||
| int lockTimeout = 10, | ||
| int delaySeconds = 0, | ||
| int messageRetentionPeriod = 345600, | ||
| TopicFindBy findTopicBy = TopicFindBy.Name, | ||
| string? iAmPolicy = null, | ||
| RedrivePolicy? redrivePolicy = null, | ||
| SnsAttributes? snsAttributes = null, | ||
| Dictionary<string, string>? tags = null, | ||
| OnMissingChannel makeChannels = OnMissingChannel.Create, | ||
| bool rawMessageDelivery = true, | ||
| TimeSpan? emptyChannelDelay = null, | ||
| TimeSpan? channelFailureDelay = null, | ||
| SnsSqsType sqsType = SnsSqsType.Standard, | ||
| bool contentBasedDeduplication = true, | ||
| DeduplicationScope? deduplicationScope = null, | ||
| int? fifoThroughputLimit = null, | ||
| RoutingKeyType routingKeyType = RoutingKeyType.PubSub, | ||
| QueueFindBy queueFindBy = QueueFindBy.Name | ||
| ) | ||
| : base(typeof(T), name, channelName, routingKey, bufferSize, noOfPerformers, timeOut, requeueCount, | ||
| requeueDelay, | ||
| unacceptableMessageLimit, messagePumpType, channelFactory, lockTimeout, delaySeconds, | ||
| messageRetentionPeriod, findTopicBy, | ||
| iAmPolicy, redrivePolicy, snsAttributes, tags, makeChannels, rawMessageDelivery, emptyChannelDelay, | ||
| channelFailureDelay, sqsType, contentBasedDeduplication, deduplicationScope, fifoThroughputLimit, | ||
| routingKeyType) |
There was a problem hiding this comment.
ℹ Getting worse: Constructor Over-Injection
SqsSubscription increases from 23 to 29 arguments, threshold = 5
| private async Task SubscribeToTopicAsync( | ||
| string topicArn, | ||
| string queueUrl, | ||
| SqsAttributes? sqsAttributes, | ||
| AmazonSQSClient sqsClient, | ||
| AmazonSimpleNotificationServiceClient snsClient) | ||
| { | ||
| var arn = await snsClient.SubscribeQueueAsync(topicArn, sqsClient, queueUrl); | ||
| if (string.IsNullOrEmpty(arn)) | ||
| { | ||
| throw new InvalidOperationException( | ||
| $"Could not subscribe to topic: {topicArn} from queue: {queueUrl} in region {AwsConnection.Region}"); | ||
| } | ||
|
|
||
| var response = await snsClient.SetSubscriptionAttributesAsync( | ||
| new SetSubscriptionAttributesRequest(arn, | ||
| "RawMessageDelivery", | ||
| sqsAttributes?.RawMessageDelivery.ToString()) | ||
| ); | ||
|
|
||
| if (response.HttpStatusCode != HttpStatusCode.OK) | ||
| { | ||
| throw new InvalidOperationException("Unable to set subscription attribute for raw message delivery"); | ||
| } | ||
| } |
There was a problem hiding this comment.
ℹ New issue: Excess Number of Function Arguments
SubscribeToTopicAsync has 5 arguments, threshold = 4
| private async Task CreateQueueAsync(AmazonSQSClient sqsClient) | ||
| { | ||
| if (_subscription is null) | ||
| throw new InvalidOperationException("ChannelFactory: Subscription cannot be null"); | ||
|
|
||
| s_logger.LogDebug("Queue does not exist, creating queue: {ChannelName} subscribed to {Topic} on {Region}", _subscription.ChannelName.Value, _subscription.RoutingKey.Value, AwsConnection.Region); | ||
| _queueUrl = null; | ||
| try | ||
| { | ||
| var attributes = new Dictionary<string, string>(); | ||
| if (_subscription.RedrivePolicy != null && _dlqARN != null) | ||
| { | ||
| var policy = new { maxReceiveCount = _subscription.RedrivePolicy.MaxReceiveCount, deadLetterTargetArn = _dlqARN }; | ||
| attributes.Add("RedrivePolicy", JsonSerializer.Serialize(policy, JsonSerialisationOptions.Options)); | ||
| } | ||
|
|
||
| attributes.Add("DelaySeconds", _subscription.DelaySeconds.ToString()); | ||
| attributes.Add("MessageRetentionPeriod", _subscription.MessageRetentionPeriod.ToString()); | ||
| if (_subscription.IAMPolicy != null) attributes.Add("Policy", _subscription.IAMPolicy); | ||
| attributes.Add("ReceiveMessageWaitTimeSeconds", _subscription.TimeOut.Seconds.ToString()); | ||
| attributes.Add("VisibilityTimeout", _subscription.LockTimeout.ToString()); | ||
|
|
||
| var tags = new Dictionary<string, string> { { "Source", "Brighter" } }; | ||
| if (_subscription.Tags != null) | ||
| { | ||
| foreach (var tag in _subscription.Tags) | ||
| { | ||
| tags.Add(tag.Key, tag.Value); | ||
| } | ||
| } | ||
|
|
||
| var request = new CreateQueueRequest(_subscription.ChannelName.Value) | ||
| { | ||
| Attributes = attributes, | ||
| Tags = tags | ||
| }; | ||
| var response = await sqsClient.CreateQueueAsync(request); | ||
| _queueUrl = response.QueueUrl; | ||
|
|
||
| if (!string.IsNullOrEmpty(_queueUrl)) | ||
| { | ||
| s_logger.LogDebug("Queue created: {URL}", _queueUrl); | ||
| using var snsClient = new AmazonSimpleNotificationServiceClient(AwsConnection.Credentials, AwsConnection.Region); | ||
| await CheckSubscriptionAsync(_subscription.MakeChannels, sqsClient, snsClient); | ||
| } | ||
| else | ||
| { | ||
| throw new InvalidOperationException($"Could not create queue: {_subscription.ChannelName.Value} subscribed to {ChannelTopicArn} on {AwsConnection.Region}"); | ||
| } | ||
| } | ||
| catch (QueueDeletedRecentlyException ex) | ||
| { | ||
| var error = $"Could not create queue {_subscription.ChannelName.Value} because {ex.Message} waiting 60s to retry"; | ||
| s_logger.LogError(ex, "Could not create queue {ChannelName} because {ErrorMessage} waiting 60s to retry", _subscription.ChannelName.Value, ex.Message); | ||
| Thread.Sleep(TimeSpan.FromSeconds(30)); | ||
| throw new ChannelFailureException(error, ex); | ||
| } | ||
| catch (AmazonSQSException ex) | ||
| { | ||
| var error = $"Could not create queue {_queueUrl} subscribed to topic {_subscription.RoutingKey.Value} in region {AwsConnection.Region.DisplayName} because {ex.Message}"; | ||
| s_logger.LogError(ex, "Could not create queue {URL} subscribed to topic {Topic} in region {Region} because {ErrorMessage}", _queueUrl, _subscription.RoutingKey.Value, AwsConnection.Region.DisplayName, ex.Message); | ||
| throw new InvalidOperationException(error, ex); | ||
| } | ||
| catch (HttpErrorResponseException ex) | ||
| { | ||
| var error = $"Could not create queue {_queueUrl} subscribed to topic {_subscription.RoutingKey.Value} in region {AwsConnection.Region.DisplayName} because {ex.Message}"; | ||
| s_logger.LogError(ex, "Could not create queue {URL} subscribed to topic {Topic} in region {Region} because {ErrorMessage}", _queueUrl, _subscription.RoutingKey.Value, AwsConnection.Region.DisplayName, ex.Message); | ||
| throw new InvalidOperationException(error, ex); | ||
| } | ||
| } |
There was a problem hiding this comment.
✅ No longer an issue: Complex Method
CreateQueueAsync is no longer above the threshold for cyclomatic complexity
| private async Task EnsureQueueAsync() | ||
| { | ||
| if (_subscription is null) | ||
| throw new InvalidOperationException("ChannelFactory: Subscription cannot be null"); | ||
|
|
||
| if (_subscription.MakeChannels == OnMissingChannel.Assume) | ||
| return; | ||
|
|
||
| using var sqsClient = new AmazonSQSClient(AwsConnection.Credentials, AwsConnection.Region); | ||
| var queueName = _subscription.ChannelName.ToValidSQSQueueName(); | ||
| var topicName = _subscription.RoutingKey.ToValidSNSTopicName(); | ||
|
|
||
| (bool exists, _) = await QueueExistsAsync(sqsClient, queueName); | ||
| if (!exists) | ||
| { | ||
| if (_subscription.MakeChannels == OnMissingChannel.Create) | ||
| { | ||
| if (_subscription.RedrivePolicy != null) | ||
| { | ||
| await CreateDLQAsync(sqsClient); | ||
| } |
There was a problem hiding this comment.
✅ No longer an issue: Bumpy Road Ahead
EnsureQueueAsync is no longer above the threshold for logical blocks with deeply nested code
| private async Task CheckSubscriptionAsync(OnMissingChannel makeSubscriptions, AmazonSQSClient sqsClient, AmazonSimpleNotificationServiceClient snsClient) | ||
| { | ||
| if (makeSubscriptions == OnMissingChannel.Assume) | ||
| return; | ||
|
|
||
| if (!await SubscriptionExistsAsync(sqsClient, snsClient)) | ||
| { | ||
| if (makeSubscriptions == OnMissingChannel.Validate) | ||
| { | ||
| throw new BrokerUnreachableException($"Subscription validation error: could not find subscription for {_queueUrl}"); | ||
| } | ||
| else if (makeSubscriptions == OnMissingChannel.Create) | ||
| { | ||
| await SubscribeToTopicAsync(sqsClient, snsClient); | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
✅ No longer an issue: Bumpy Road Ahead
CheckSubscriptionAsync is no longer above the threshold for logical blocks with deeply nested code
| @@ -1,4 +1,5 @@ | |||
| #region Licence | |||
|
|
|||
There was a problem hiding this comment.
✅ No longer an issue: Overall Code Complexity
The mean cyclomatic complexity in this module is no longer above the threshold
| else if (_publication.Topic is not null) | ||
| { | ||
| routingKey = _publication.Topic; | ||
| } | ||
|
|
||
| if (routingKey is null) |
There was a problem hiding this comment.
There is a RoutingKey.IsNullOrEmpty()
|
|
||
| var arnElements = s!.Split(':'); | ||
| var topic = arnElements[(int)ARNAmazonSNS.TopicName]; | ||
| var topic = topicArn.GetValueInString() ?? string.Empty; |
There was a problem hiding this comment.
The earlier version reduces nesting by returning early, which is generally preferable
| } | ||
| } | ||
|
|
||
| var messageAttributes = new Dictionary<string, MessageAttributeValue> |
There was a problem hiding this comment.
Typically we factor this into a publisher, to keep the producer clean of all the header manipulation
Hohohoho 🎅
How should I handle the tests? My current approach is to duplicate them. Should I continue on this path?Duplicate the test