Honor ingest-percentage for bulks #768
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
With this commit we honor the parameter `ingest-percentage` again for bulk requests. Before elastic#763, the method `size()` was used to determine upfront when a parameter source is exhausted. When we changed the API then in elastic#763 the parameter source's progress indication was instead done with the `percent_completed` property but this is only informational and the parameter source is required to raise `StopIteration` to indicate completion. This means that `ingest-percentage` was not considered anymore for bulks but instead we always read the input file(s) completely and then terminated with `StopIteration`. Relates elastic#763
danielmitterdorfer
added
bug
dliappis
approved these changes
Sep 10, 2019
|
For the record, this was observed with a reproduction like: where progress for |
|
Thanks for the review! |
novosibman
pushed a commit
to novosibman/rally
that referenced
this pull request
Oct 2, 2019
With this commit we honor the parameter `ingest-percentage` again for bulk requests. Before elastic#763, the method `size()` was used to determine upfront when a parameter source is exhausted. When we changed the API then in elastic#763 the parameter source's progress indication was instead done with the `percent_completed` property but this is only informational and the parameter source is required to raise `StopIteration` to indicate completion. This means that `ingest-percentage` was not considered anymore for bulks but instead we always read the input file(s) completely and then terminated with `StopIteration`. Relates elastic#763 Relates elastic#768
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With this commit we honor the parameter
ingest-percentageagain for bulkrequests. Before #763, the method
size()was used to determine upfront when aparameter source is exhausted. When we changed the API then in #763 the
parameter source's progress indication was instead done with the
percent_completedproperty but this is only informational and the parametersource is required to raise
StopIterationto indicate completion. This meansthat
ingest-percentagewas not considered anymore for bulks but instead wealways read the input file(s) completely and then terminated with
StopIteration.Relates #763