chore: Minor cleanup of DynamoDB OffsetStoreDao #1262
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
patriknw
commented
Nov 22, 2024
| @@ -13,7 +13,8 @@ akka.projection.dynamodb { | |||
| time-window = 5 minutes | |||
|
|
|||
| # Trying to batch insert offsets in batches of this size. | |||
| offset-batch-size = 20 | |||
| # Must be less than or equal to 25 (hard limit in DynamoDB) | |||
| offset-batch-size = 25 | |||
There was a problem hiding this comment.
I would probably not have made this configurable since it's a hard limit in DynamoDB, but since we had this already in the config/settings and it wasn't used.
| @@ -63,7 +62,7 @@ import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse | |||
| val managementStateBySlicePid = AttributeValue.fromS("_mgmt") | |||
| } | |||
|
|
|||
| final class BatchWriteFailed(val lastResponse: BatchWriteItemResponse) extends Exception | |||
| final class BatchWriteFailed(val lastResponse: BatchWriteItemResponse) extends RuntimeException | |||
There was a problem hiding this comment.
doesn't matter in Scala, but just to follow our conventions
| @@ -143,13 +140,14 @@ import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse | |||
| if (log.isDebugEnabled) { | |||
| val count = unprocessed.asScala.valuesIterator.map(_.size).sum | |||
| log.debug( | |||
| "Not all writes in batch were applied, retrying in [{}]: [{}] unapplied writes, [{}/{}] retries", | |||
| delay.toCoarsest, | |||
There was a problem hiding this comment.
doesn't work great for exponential backoff delays since those will become full random nanos
| val unprocessedSeqNrs = unprocessedSeqNrItems.map { item => | ||
| import OffsetStoreDao.OffsetStoreAttributes._ | ||
| s"${item.get(NameSlice).s}: ${item.get(Pid).s}" | ||
| } |
There was a problem hiding this comment.
it was kind of double iterations and collection constructions of those
|
I looked at the test failure. It's unrelated and I see what is wrong, in the test. I'll follow up on that in separate PR |
johanandren
approved these changes
Nov 22, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.