Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] check decimal inputs before multiply and divide operations #4729

Open
revans2 opened this issue Feb 9, 2022 · 1 comment
Open

[FEA] check decimal inputs before multiply and divide operations #4729

revans2 opened this issue Feb 9, 2022 · 1 comment
Assignees
Labels
feature request New feature or request P1 Nice to have for release performance A performance related task/issue

Comments

@revans2
Copy link
Collaborator

revans2 commented Feb 9, 2022

Is your feature request related to a problem? Please describe.
This is another one of my crazy ideas. I have not verified anything with it to see how performant it might be. We know that multiply and divide are very expensive to do as decimal 128 values. But we also know that Spark often will make values grow a lot larger than they need to be. It would be great if we could come up with a way to very cheaply check if what is stored in a column needs 128-bits or if we could do the computation as 64-bits instead. We might be able to use a min/max reduction on the column vector to detect this.

@revans2 revans2 added feature request New feature or request ? - Needs Triage Need team to review and classify performance A performance related task/issue labels Feb 9, 2022
@ttnghia
Copy link
Collaborator

ttnghia commented Feb 9, 2022

How about binop comparison? Comparing all elements to the int64_t::max and do a reduction to see if all the numbers are fit within 64 bit rep.

@sameerz sameerz removed the ? - Needs Triage Need team to review and classify label Feb 15, 2022
@jlowe jlowe self-assigned this Feb 18, 2022
@mattahrens mattahrens added the P1 Nice to have for release label May 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request P1 Nice to have for release performance A performance related task/issue
Projects
None yet
Development

No branches or pull requests

5 participants