Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add in more generalized support for casting nested types #3162

Merged
merged 8 commits into from
Aug 6, 2021
Merged

Add in more generalized support for casting nested types #3162

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
f93e7ce
Update nested cast support to be more generic
revans2 Aug 6, 2021
861f60a
Struct to Struct casting support
revans2 Aug 6, 2021
674fc57
Map to Map cast support
revans2 Aug 6, 2021
c68b7d4
Missed signoff
revans2 Aug 6, 2021
a878c0d
Addressed review comments
revans2 Aug 6, 2021
b6272aa
Forgot to regenerate docs
revans2 Aug 6, 2021
dfd783d
Forgot to remove tests that are now not needed
revans2 Aug 6, 2021
ba1715c
Fixed a bug and made sure no more like it exist
revans2 Aug 6, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions docs/supported_ops.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20587,7 +20587,7 @@ and the accelerator produces the same result.
<td> </td>
<td> </td>
<td> </td>
<td><em>PS<br/>missing nested BOOLEAN, BYTE, SHORT, LONG, DATE, TIMESTAMP, STRING, DECIMAL, NULL, BINARY, CALENDAR, MAP, STRUCT, UDT</em></td>
<td><em>PS<br/>The array's child type must also support being cast to to desired child type;<br/>max nested DECIMAL precision of 18;<br/>UTC is only supported TZ for nested TIMESTAMP;<br/>missing nested CALENDAR, UDT</em></td>
<td> </td>
<td> </td>
<td> </td>
Expand All @@ -20609,7 +20609,7 @@ and the accelerator produces the same result.
<td> </td>
<td> </td>
<td> </td>
<td><b>NS</b></td>
<td><em>PS<br/>the map's kay and value must also support being cast to the desired child types;<br/>max nested DECIMAL precision of 18;<br/>UTC is only supported TZ for nested TIMESTAMP;<br/>missing nested CALENDAR, UDT</em></td>
<td> </td>
<td> </td>
</tr>
Expand All @@ -20631,7 +20631,7 @@ and the accelerator produces the same result.
<td> </td>
<td> </td>
<td> </td>
<td><b>NS</b></td>
<td><em>PS<br/>the struct's children must also support being cast to the desired child type(s);<br/>max nested DECIMAL precision of 18;<br/>UTC is only supported TZ for nested TIMESTAMP;<br/>missing nested CALENDAR, UDT</em></td>
<td> </td>
</tr>
<tr>
Expand Down Expand Up @@ -20991,7 +20991,7 @@ and the accelerator produces the same result.
<td> </td>
<td> </td>
<td> </td>
<td><em>PS<br/>missing nested BOOLEAN, BYTE, SHORT, LONG, DATE, TIMESTAMP, STRING, DECIMAL, NULL, BINARY, CALENDAR, MAP, STRUCT, UDT</em></td>
<td><em>PS<br/>The array's child type must also support being cast to to desired child type;<br/>max nested DECIMAL precision of 18;<br/>UTC is only supported TZ for nested TIMESTAMP;<br/>missing nested CALENDAR, UDT</em></td>
<td> </td>
<td> </td>
<td> </td>
Expand All @@ -21013,7 +21013,7 @@ and the accelerator produces the same result.
<td> </td>
<td> </td>
<td> </td>
<td><b>NS</b></td>
<td><em>PS<br/>the map's kay and value must also support being cast to the desired child types;<br/>max nested DECIMAL precision of 18;<br/>UTC is only supported TZ for nested TIMESTAMP;<br/>missing nested CALENDAR, UDT</em></td>
<td> </td>
<td> </td>
</tr>
Expand All @@ -21035,7 +21035,7 @@ and the accelerator produces the same result.
<td> </td>
<td> </td>
<td> </td>
<td><b>NS</b></td>
<td><em>PS<br/>the struct's children must also support being cast to the desired child type(s);<br/>max nested DECIMAL precision of 18;<br/>UTC is only supported TZ for nested TIMESTAMP;<br/>missing nested CALENDAR, UDT</em></td>
<td> </td>
</tr>
<tr>
Expand Down
24 changes: 24 additions & 0 deletions integration_tests/src/main/python/cast_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,27 @@ def test_cast_empty_string_to_int():
'CAST(a as INTEGER)',
'CAST(a as LONG)'))

# These tests are not intended to be exhaustive. The scala test CastOpSuite should cover
# just about everything for non-nested values. This is intended to check that the
# recursive code in nested type checks, like arrays, is working properly. So we are going
# pick child types that are simple to cast. Upcasting integer values and casting them to strings
@pytest.mark.parametrize('data_gen,to_type', [
(ArrayGen(byte_gen), ArrayType(IntegerType())),
(ArrayGen(byte_gen), ArrayType(StringType())),
(ArrayGen(byte_gen), ArrayType(DecimalType(6, 2))),
(ArrayGen(ArrayGen(byte_gen)), ArrayType(ArrayType(IntegerType()))),
(ArrayGen(ArrayGen(byte_gen)), ArrayType(ArrayType(StringType()))),
(ArrayGen(ArrayGen(byte_gen)), ArrayType(ArrayType(DecimalType(6, 2)))),
(StructGen([('a', byte_gen)]), StructType([StructField('a', IntegerType())])),
(StructGen([('a', byte_gen), ('c', short_gen)]), StructType([StructField('b', IntegerType()), StructField('c', ShortType())])),
(StructGen([('a', ArrayGen(byte_gen)), ('c', short_gen)]), StructType([StructField('a', ArrayType(IntegerType())), StructField('c', LongType())])),
(ArrayGen(StructGen([('a', byte_gen), ('b', byte_gen)])), ArrayType(StringType())),
(MapGen(ByteGen(nullable=False), byte_gen), MapType(StringType(), StringType())),
(MapGen(ShortGen(nullable=False), ArrayGen(byte_gen)), MapType(IntegerType(), ArrayType(ShortType()))),
(MapGen(ShortGen(nullable=False), ArrayGen(StructGen([('a', byte_gen)]))), MapType(IntegerType(), ArrayType(StructType([StructField('b', ShortType())]))))
], ids=idfn)
def test_cast_nested(data_gen, to_type):
assert_gpu_and_cpu_are_equal_collect(
lambda spark : unary_op_df(spark, data_gen).select(f.col('a').cast(to_type)))


15 changes: 12 additions & 3 deletions shims/spark311/src/main/scala/com/nvidia/spark/rapids/shims/spark311/Spark311Shims.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,13 +147,22 @@ class Spark311Shims extends Spark301Shims {

// calendarChecks are the same

override val arrayChecks: TypeSig = none
override val arrayChecks: TypeSig =
ARRAY.nested(commonCudfTypes + DECIMAL_64 + NULL + ARRAY + BINARY + STRUCT) +
psNote(TypeEnum.ARRAY, "The array's child type must also support being cast to " +
"to desired child type")
jlowe marked this conversation as resolved.
Show resolved Hide resolved
override val sparkArraySig: TypeSig = ARRAY.nested(all)

override val mapChecks: TypeSig = none
override val mapChecks: TypeSig =
MAP.nested(commonCudfTypes + DECIMAL_64 + NULL + ARRAY + BINARY + STRUCT + MAP) +
psNote(TypeEnum.MAP, "the map's kay and value must also support being cast to the " +
jlowe marked this conversation as resolved.
Show resolved Hide resolved
"desired child types")
override val sparkMapSig: TypeSig = MAP.nested(all)

override val structChecks: TypeSig = none
override val structChecks: TypeSig =
STRUCT.nested(commonCudfTypes + DECIMAL_64 + NULL + ARRAY + BINARY + STRUCT) +
psNote(TypeEnum.STRUCT, "the struct's children must also support being cast to the " +
"desired child type(s)")
override val sparkStructSig: TypeSig = STRUCT.nested(all)

override val udtChecks: TypeSig = none
Expand Down
Loading