Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix/111 #112

Merged
merged 6 commits into from
Feb 4, 2018
Merged

Fix/111 #112

merged 6 commits into from
Feb 4, 2018

Conversation

numberoverzero
Copy link
Owner

This fixes a bug where the records of the last GetRecords call in a closed Shard are dropped from the Coordinator's buffer immediately. See #111 for investigation details.

Now, the Coordinator keeps a count of buffered records for each closed shard that it previously tracked; as each record is popped off the buffer, the count is decremented until it's dropped. The Shards can't be tracked in the same place as active shards (Coordinator.active) because those are periodically polled for more records. However, we can't stop tracking them because then the rendered Shard.token would skip ahead while the records are still buffered.

In addition to unit tests, I ran some hacked together tests for a few hours; records were inserted every 30mins/1min/10sec in 3 different streams and then counted on the way out of the Stream processor.

@numberoverzero numberoverzero self-assigned this Feb 4, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant