You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Spark is very conservative in calculating the worst case situation for various math operations with Decimals. This can result in a lot of operations that have a precision of 38. We also are very careful when doing decimal operations and only accept operations where we have a very high confidence that we will not overflow, or that we can detect when an overflow has happened. When you combine these together there are a lot of decimal operations that we cannot do on the GPU. We should have a config that lets the user tell us that they take full responsibility of making sure that there will be no overflows when actually processing the data. As a part of this we should be extra careful with multiply and divide that we have documented exactly what ranges of values still might result in an overflow. For these it is related to #3886
The text was updated successfully, but these errors were encountered:
Spark is very conservative in calculating the worst case situation for various math operations with Decimals. This can result in a lot of operations that have a precision of 38. We also are very careful when doing decimal operations and only accept operations where we have a very high confidence that we will not overflow, or that we can detect when an overflow has happened. When you combine these together there are a lot of decimal operations that we cannot do on the GPU. We should have a config that lets the user tell us that they take full responsibility of making sure that there will be no overflows when actually processing the data. As a part of this we should be extra careful with multiply and divide that we have documented exactly what ranges of values still might result in an overflow. For these it is related to #3886
The text was updated successfully, but these errors were encountered: