Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiget and scan iteration count bug #808

Closed
hycdong opened this issue Sep 2, 2021 · 0 comments
Closed

Multiget and scan iteration count bug #808

hycdong opened this issue Sep 2, 2021 · 0 comments
Labels
type/bug This issue reports a bug. type/enhancement Indicates new feature requests
Milestone

Comments

@hycdong
Copy link
Contributor

hycdong commented Sep 2, 2021

Feature enhancemant and bug report

As #486 shows, Pegasus version 1.12.3 adds iteration count and iteration time check while calling rocksdb range read.

For example, client use multiget interface and set max_kv_count option to control the expected value count this round would like to get. If this round don't get completed data, server will return false and the last read key-value pair, the next round can specify that key to start next round.

Case1. When total_count < max_kv_count <= max_iteration_count

Assuming: total_count = 50, max_kv_count=100, server max_iteration_count=3000

  1. [1, 50] value are expired -> can't get any value, return true
  2. [1, 20] value are expired, [21, 50] is valid -> get value [21, 50], count = 30, return true
  3. [1, 50] value are all valid -> get value [1, 50], count = 50, return true

Case2. When total_count > max_kv_count >= max_iteration

Assuming: total_count = 4000, max_kv_count=3500, server max_iteration_count=3000

  1. [1, 3000] value are all expired, [3001, 4000] are valid -> can't get any value, return false
  2. [1, 500] value are expired, [501, 4000] are valid -> get value [501,3000], count = 2500, return false
  3. [1, 4000] value are all valid -> get value [1, 3000], count = 3000, return false

Case3. When the total_count > max_iteration_count > max_kv_count

Assuming: total_count = 4000, max_kv_count=100, server max_iteration_count=3000

  1. [1, 3000] value are all expired, [3001, 4000] are valid -> can't get any value, return false
  2. [1, 2950] value are expired, [2951, 4000] are valid -> can't get any value, return false
  3. [1, 100] value are all expired, [101, 4000] are valid -> can't get any value, return false
  4. [1, 20] value are expired, [21, 4000] is valid -> get value [21,100], count = 80, return false
  5. [1, 4000] value are all valid -> get value [1, 100], count = 100, return false

However, the Bold cases above don't meet the client expectation, the expected result are:

  • [1, 2950] value are expired, [2951, 4000] are valid -> value[2951, 3000], count = 50, return false
  • [1, 100] value are all expired, [101, 4000] are valid -> value [101, 200], count = 100, return false
  • [1, 20] value are expired, [21, 4000] is valid -> value[21, 120], count = 100, return false

In particular, for case2-1 and case3-1, if too many value expired or filtered (exceed the max_iteration_count), server SHOULD NOT iterate more to get a valid value.

@hycdong hycdong added type/bug This issue reports a bug. type/enhancement Indicates new feature requests labels Sep 2, 2021
@hycdong hycdong added this to the v2.3.0 milestone Sep 6, 2021
@hycdong hycdong closed this as completed Oct 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug This issue reports a bug. type/enhancement Indicates new feature requests
Projects
None yet
Development

No branches or pull requests

1 participant