Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Decimal 128 Support: Add a "Trust me I know it will not overflow config" #3888

Closed
revans2 opened this issue Oct 21, 2021 · 0 comments
Closed
Assignees
Labels
feature request New feature or request

Comments

@revans2
Copy link
Collaborator

revans2 commented Oct 21, 2021

Spark is very conservative in calculating the worst case situation for various math operations with Decimals. This can result in a lot of operations that have a precision of 38. We also are very careful when doing decimal operations and only accept operations where we have a very high confidence that we will not overflow, or that we can detect when an overflow has happened. When you combine these together there are a lot of decimal operations that we cannot do on the GPU. We should have a config that lets the user tell us that they take full responsibility of making sure that there will be no overflows when actually processing the data. As a part of this we should be extra careful with multiply and divide that we have documented exactly what ranges of values still might result in an overflow. For these it is related to #3886

@revans2 revans2 added feature request New feature or request ? - Needs Triage Need team to review and classify labels Oct 21, 2021
@revans2 revans2 self-assigned this Oct 22, 2021
@Salonijain27 Salonijain27 removed the ? - Needs Triage Need team to review and classify label Oct 26, 2021
@revans2 revans2 closed this as completed Oct 29, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants