Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] NDS q58 Decimal scale (59) cannot be greater than precision (38). #1984

Closed
andygrove opened this issue Mar 22, 2021 · 1 comment · Fixed by #1991
Closed

[BUG] NDS q58 Decimal scale (59) cannot be greater than precision (38). #1984

andygrove opened this issue Mar 22, 2021 · 1 comment · Fixed by #1991
Assignees
@razajafri
Labels
bug Something isn't working
Milestone
Mar 15 - March 26

Comments

@andygrove
Copy link
Contributor

Describe the bug

NDS q58 fails in CI when running against Spark 3.1.1 against the 1TB parquet dataset with decimals.

org.apache.spark.sql.AnalysisException: Decimal scale (59) cannot be greater than precision (38).
  at org.apache.spark.sql.types.DecimalType.<init>(DecimalType.scala:48)
  at com.nvidia.spark.rapids.GpuOverrides$$anon$97.tagExprForGpu(GpuOverrides.scala:1736
@andygrove andygrove added bug Something isn't working ? - Needs Triage Need team to review and classify labels Mar 22, 2021
@andygrove
Copy link
Contributor Author

I can reproduce this with Spark 3.0.2 as well

@razajafri razajafri self-assigned this Mar 22, 2021
razajafri added a commit to razajafri/spark-rapids that referenced this issue Mar 22, 2021
@razajafri
fixes NVIDIA#1984
770131e 
Signed-off-by: Raza Jafri <[email protected]>
@razajafri razajafri mentioned this issue Mar 22, 2021
Merged
@sameerz sameerz removed the ? - Needs Triage Need team to review and classify label Mar 23, 2021
@sameerz sameerz added this to the Mar 15 - March 26 milestone Mar 23, 2021
razajafri added a commit that referenced this issue Mar 23, 2021
@razajafri
Fixes creation of invalid DecimalType in GpuDivide.tagExprForGpu (#1991)
cc91b54 
* fixes #1984

Signed-off-by: Raza Jafri <[email protected]>

* added unit tests

Signed-off-by: Raza Jafri <[email protected]>

* make sure the exception is not AnalysisException

Signed-off-by: Raza Jafri <[email protected]>

Co-authored-by: Raza Jafri <[email protected]>
nartal1 pushed a commit to nartal1/spark-rapids that referenced this issue Jun 9, 2021
@razajafri
Fixes creation of invalid DecimalType in GpuDivide.tagExprForGpu (NVI…
244b5ca 
…DIA#1991)

* fixes NVIDIA#1984

Signed-off-by: Raza Jafri <[email protected]>

* added unit tests

Signed-off-by: Raza Jafri <[email protected]>

* make sure the exception is not AnalysisException

Signed-off-by: Raza Jafri <[email protected]>

Co-authored-by: Raza Jafri <[email protected]>
nartal1 pushed a commit to nartal1/spark-rapids that referenced this issue Jun 9, 2021
@razajafri
Fixes creation of invalid DecimalType in GpuDivide.tagExprForGpu (NVI…
bd22164 
…DIA#1991)

* fixes NVIDIA#1984

Signed-off-by: Raza Jafri <[email protected]>

* added unit tests

Signed-off-by: Raza Jafri <[email protected]>

* make sure the exception is not AnalysisException

Signed-off-by: Raza Jafri <[email protected]>

Co-authored-by: Raza Jafri <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@razajafri razajafri

Labels
bug Something isn't working
Projects
None yet
Milestone
Mar 15 - March 26
Development

Successfully merging a pull request may close this issue.

Fixes creation of invalid DecimalType in GpuDivide.tagExprForGpu razajafri/spark-rapids
3 participants
@andygrove @sameerz @razajafri