Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GLUTEN-1490] refactor substrait literals using generics, and support map/struct/array literals based on it #1494

Merged
merged 5 commits into from
May 18, 2023
Merged

[GLUTEN-1490] refactor substrait literals using generics, and support map/struct/array literals based on it #1494

merged 5 commits into from
May 18, 2023

Conversation

taiyang-li
Copy link
Contributor

@taiyang-li taiyang-li commented Apr 24, 2023 •
edited
Loading

What changes were proposed in this pull request?

(Fixes: #1490, #1418)

@github-actions
Copy link

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/oap-project/gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

@github-actions
Copy link

Run Gluten Clickhouse CI

@taiyang-li taiyang-li changed the title [CH][VL] literals with scalar types use generics [CH][VL] step1: literals with scalar types use generics Apr 24, 2023
@taiyang-li taiyang-li changed the title [CH][VL] step1: literals with scalar types use generics [GLUTEN-1490] step1: literals with scalar types use generics Apr 24, 2023
@github-actions
Copy link

#1490

@github-actions
Copy link

Run Gluten Clickhouse CI

1 similar comment
@github-actions
Copy link

Run Gluten Clickhouse CI

@taiyang-li taiyang-li marked this pull request as draft April 25, 2023 06:57
@github-actions
Copy link

Run Gluten Clickhouse CI

@taiyang-li
Copy link
Contributor Author

I had adding some new changes, work in progress, pls do not review until this pr is out of draft state.

@github-actions
Copy link

Run Gluten Clickhouse CI

@taiyang-li taiyang-li changed the title [GLUTEN-1490] step1: literals with scalar types use generics [GLUTEN-1490] refactor substrait literals using generics Apr 25, 2023
@taiyang-li taiyang-li marked this pull request as ready for review April 25, 2023 08:48
@taiyang-li taiyang-li mentioned this pull request Apr 25, 2023
Closed
@github-actions
Copy link

Run Gluten Clickhouse CI

@taiyang-li taiyang-li marked this pull request as draft April 25, 2023 09:45
@github-actions
Copy link

Run Gluten Clickhouse CI

1 similar comment
@github-actions
Copy link

Run Gluten Clickhouse CI

@github-actions
Copy link

Run Gluten Clickhouse CI

@taiyang-li taiyang-li changed the title [GLUTEN-1490] refactor substrait literals using generics [GLUTEN-1490] refactor substrait literals using generics, and support map/struct/array literals based on it Apr 26, 2023
@taiyang-li taiyang-li marked this pull request as ready for review April 26, 2023 03:37
@taiyang-li
Copy link
Contributor Author

@zzcclp @rui-mo Do you have time to review this pr, thanks!

rui-mo
rui-mo reviewed Apr 26, 2023
View reviewed changes
Copy link
Contributor

@rui-mo rui-mo left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We also need some time to check the failed unit tests on Velox backend.

gluten-core/src/main/java/io/glutenproject/substrait/expression/StructLiteralNode.java Outdated Show resolved Hide resolved
@github-actions
Copy link

Run Gluten Clickhouse CI

3 similar comments
@github-actions
Copy link

Run Gluten Clickhouse CI

@github-actions
Copy link

Run Gluten Clickhouse CI

@github-actions
Copy link

Run Gluten Clickhouse CI

@rui-mo rui-mo mentioned this pull request May 5, 2023
Merged
@github-actions
Copy link

github-actions bot commented May 5, 2023

Run Gluten Clickhouse CI

@rui-mo
Copy link
Contributor

rui-mo commented May 5, 2023

Below error was found on CI when executing tpcds q8. I am not sure whether it is related to the BinaryStructNode change in this PR. @jinchengchenghh Do you have any clue for that?

Error running query q8.  Error: FATAL: java.io.IOException: Unexpected Bloom filter version number (16777218)
2023-05-05T08:34:30.4115446Z    at org.apache.spark.util.sketch.BloomFilterImpl.readFrom0(BloomFilterImpl.java:256)
2023-05-05T08:34:30.4116319Z    at org.apache.spark.util.sketch.BloomFilterImpl.readFrom(BloomFilterImpl.java:265)
2023-05-05T08:34:30.4117202Z    at org.apache.spark.util.sketch.BloomFilter.readFrom(BloomFilter.java:178)
2023-05-05T08:34:30.4118333Z    at org.apache.spark.sql.catalyst.expressions.BloomFilterMightContain.deserialize(BloomFilterMightContain.scala:111)

@jinchengchenghh
Copy link
Contributor

The TPCDS Q8 has offloaded to velox, this PR may fallback might_contain and offload bloom_filter_agg to velox. @rui-mo

@github-actions
Copy link

github-actions bot commented May 6, 2023

Run Gluten Clickhouse CI

@github-actions
Copy link

Run Gluten Clickhouse CI

3 similar comments
@github-actions
Copy link

Run Gluten Clickhouse CI

@github-actions
Copy link

Run Gluten Clickhouse CI

@github-actions
Copy link

Run Gluten Clickhouse CI

@github-actions
Copy link

Run Gluten Clickhouse CI

4 similar comments
@github-actions
Copy link

Run Gluten Clickhouse CI

@rui-mo
Copy link
Contributor

rui-mo commented May 16, 2023

Run Gluten Clickhouse CI

@github-actions
Copy link

Run Gluten Clickhouse CI

@rui-mo
Copy link
Contributor

rui-mo commented May 16, 2023

Run Gluten Clickhouse CI

@rui-mo
Copy link
Contributor

rui-mo commented May 16, 2023

@taiyang-li CH CI failed, could you take a look?

@github-actions
Copy link

Run Gluten Clickhouse CI

1 similar comment
@lwz9103
Copy link
Contributor

lwz9103 commented May 17, 2023

Run Gluten Clickhouse CI

@github-actions
Copy link

Run Gluten Clickhouse CI

@rui-mo rui-mo merged commit 82e7c41 into apache:main May 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rui-mo rui-mo rui-mo approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Implement LiteralNode in generics
4 participants
@taiyang-li @rui-mo @jinchengchenghh @lwz9103