Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This fixes a bug where the records of the last GetRecords call in a closed Shard are dropped from the Coordinator's buffer immediately. See #111 for investigation details.
Now, the Coordinator keeps a count of buffered records for each closed shard that it previously tracked; as each record is popped off the buffer, the count is decremented until it's dropped. The Shards can't be tracked in the same place as active shards (
Coordinator.active
) because those are periodically polled for more records. However, we can't stop tracking them because then the renderedShard.token
would skip ahead while the records are still buffered.In addition to unit tests, I ran some hacked together tests for a few hours; records were inserted every 30mins/1min/10sec in 3 different streams and then counted on the way out of the
Stream
processor.