Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Further simplify the offset validation #1770

Closed
HaoYang670 opened this issue Jun 1, 2022 · 1 comment
Closed

Further simplify the offset validation #1770

HaoYang670 opened this issue Jun 1, 2022 · 1 comment
Labels
arrow Changes to the arrow crate

Comments

@HaoYang670
Copy link
Contributor

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
This is the follow-up of #1684 (comment).

Describe the solution you'd like

How do we validate offsets now?

  1. check if each offset can be converted to usize
  2. check if each offset exceeds the offset_limit
  3. check if offsets are monotonically increasing
  4. validate each slot.

How could we simplify this process?

  1. check if the last offset exceeds the offset_limit
  2. check if each offset can be converted to usize (actually, this could also be removed. But we need to convert the type of offset to usize, so reserve it)
  3. check if offsets are monotonically increasing
  4. validate each slot.

What will be impacted?

  1. We hope the validation could be somewhat faster (we need the benchmark result)
  2. The Error message will change for some test cases. For example, given an offset buffer
[0, 1, 6, 3, 4, 5]

and offset_limit == 5
We will get an error "Offset invariant failure: offset at position 2 out of bounds: 6 > 5" now, but
Offset invariant failure: non-monotonic offset at slot 2: 6 > 3 if we do the simplification.

@HaoYang670 HaoYang670 added the enhancement Any new improvement worthy of a entry in the changelog label Jun 1, 2022
@HaoYang670
Copy link
Contributor Author

I have done the simplification but no obvious performance improvement is gotten.
So close this issue

@HaoYang670 HaoYang670 closed this as not planned Won't fix, can't repro, duplicate, stale Jun 2, 2022
@alamb alamb added arrow Changes to the arrow crate and removed arrow Changes to the arrow crate enhancement Any new improvement worthy of a entry in the changelog labels Jun 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate
Projects
None yet
Development

No branches or pull requests

2 participants