Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ORC reader supports struct #2887

Merged
merged 17 commits into from
Jul 28, 2021
Merged

ORC reader supports struct #2887

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
5f0ceb5
orc reader supports struct and list
firestarman Jul 8, 2021
5b32a8d
Update the tests
firestarman Jul 9, 2021
a2749c6
Address comments
firestarman Jul 14, 2021
3d37d6e
Merge branch 'branch-21.08' into orc-read
firestarman Jul 16, 2021
9f46068
support only struct
firestarman Jul 16, 2021
c9a1e84
Merge branch 'branch-21.08' into orc-read
firestarman Jul 21, 2021
3e1d6b6
tool module update
firestarman Jul 21, 2021
cecadba
Add tests for nested column pruning
firestarman Jul 22, 2021
b724f00
default to false
firestarman Jul 23, 2021
2a30234
Fallback to cpu for nested schema pruning.
firestarman Jul 26, 2021
ac2149a
Merge branch 'branch-21.08' into orc-read
firestarman Jul 26, 2021
b7fdeb7
Updated tests
firestarman Jul 26, 2021
79dbfe4
Fall back to the CPU only finding struct column
firestarman Jul 26, 2021
4d18885
Add tests for nested predicate pushdown.
firestarman Jul 26, 2021
de93d07
correct the fallback condition
firestarman Jul 26, 2021
4506d0e
remove extra line
firestarman Jul 26, 2021
a1240e0
Address comments
firestarman Jul 27, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/supported_ops.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20918,9 +20918,9 @@ dates or timestamps, or for a lack of type coercion support.
<td> </td>
<td><b>NS</b></td>
<td> </td>
<td><em>PS* (missing nested DECIMAL, BINARY, MAP, UDT)</em></td>
<td><b>NS</b></td>
<td><b>NS</b></td>
<td><b>NS</b></td>
<td><em>PS* (missing nested DECIMAL, BINARY, MAP, UDT)</em></td>
<td><b>NS</b></td>
</tr>
<tr>
Expand Down
25 changes: 23 additions & 2 deletions integration_tests/src/main/python/orc_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,9 +45,30 @@ def test_basic_read(std_input_path, name, read_func, v1_enabled_list, orc_impl,
read_func(std_input_path + '/' + name),
conf=all_confs)

orc_gens_list = [[byte_gen, short_gen, int_gen, long_gen, float_gen, double_gen,
orc_basic_gens = [byte_gen, short_gen, int_gen, long_gen, float_gen, double_gen,
string_gen, boolean_gen, DateGen(start=date(1590, 1, 1)),
TimestampGen(start=datetime(1590, 1, 1, tzinfo=timezone.utc))],
TimestampGen(start=datetime(1590, 1, 1, tzinfo=timezone.utc))]

# Set to true after the issue https://github.com/rapidsai/cudf/issues/8704 is fixed.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't check this in as-is -- it can silently corrupt data on an ORC read. Either we need to make the struct support qualified by a separate incompatible config that defaults to off or wait until it is fixed.

Copy link
Collaborator Author

@firestarman firestarman Jul 13, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue rapidsai/cudf#8704 is marked as " v21.08 Release", so I chose to wait for the fix.

struct_gen_nullable = False

orc_basic_struct_gen = StructGen([['child'+str(ind), sub_gen] for ind, sub_gen in enumerate(orc_basic_gens)],
struct_gen_nullable)

# Some array gens, but not all because of nesting
orc_array_gens_sample = [ArrayGen(sub_gen) for sub_gen in orc_basic_gens] + [
ArrayGen(ArrayGen(short_gen, max_length=10)),
ArrayGen(ArrayGen(string_gen, max_length=10)),
ArrayGen(StructGen([['child0', byte_gen], ['child1', string_gen], ['child2', float_gen]], struct_gen_nullable))]

# Some struct gens, but not all because of nesting
orc_struct_gens_sample = [orc_basic_struct_gen,
StructGen([['child0', byte_gen], ['child1', orc_basic_struct_gen]], struct_gen_nullable),
StructGen([['child0', ArrayGen(short_gen)], ['child1', double_gen]], struct_gen_nullable)]

orc_gens_list = [orc_basic_gens,
orc_array_gens_sample,
orc_struct_gens_sample,
pytest.param([date_gen], marks=pytest.mark.xfail(reason='https://github.com/NVIDIA/spark-rapids/issues/131')),
pytest.param([timestamp_gen], marks=pytest.mark.xfail(reason='https://github.com/NVIDIA/spark-rapids/issues/131'))]

Expand Down
3 changes: 2 additions & 1 deletion sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -769,7 +769,8 @@ object GpuOverrides {
sparkSig = (TypeSig.atomics + TypeSig.STRUCT + TypeSig.ARRAY + TypeSig.MAP +
TypeSig.UDT).nested())),
(OrcFormatType, FileFormatChecks(
cudfReadWrite = TypeSig.commonCudfTypes,
cudfRead = (TypeSig.commonCudfTypes + TypeSig.STRUCT + TypeSig.ARRAY).nested(),
cudfWrite = TypeSig.commonCudfTypes,
sparkSig = (TypeSig.atomics + TypeSig.STRUCT + TypeSig.ARRAY + TypeSig.MAP +
TypeSig.UDT).nested())))

Expand Down