Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added documentation for assertion on non-empty nulls [skip ci] #8271

Closed
wants to merge 2 commits into from
Closed

Added documentation for assertion on non-empty nulls [skip ci] #8271

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d4b1f59
Added documentation for assertion on non-empty nulls
razajafri May 10, 2023
fa7f3af
change wording
razajafri May 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/compatibility.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,14 @@ problem. Please also look at the current list of
[bugs](https://github.com/NVIDIA/spark-rapids/issues?q=is%3Aopen+is%3Aissue+label%3Abug) which are
typically incompatibilities that we have not yet addressed.

## Non-empty nulls

The SQL plugin doesn't support nested types to have non-empty nulls and if the data being processed
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"non-empty nulls" sounds cryptic. Can you rephrase or illustrate with an example?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I agree. I can't think of another word, let me explain with an example

contains non-empty nulls, it will throw an AssertionError. If you still want to continue using this
disable assertions by setting `-da:ai.rapids.cudf.AssertEmptyNulls` in extra Java parameters for the
Driver and the Executor but know that this will result in undefined behavior as
[cudf](https://github.com/rapidsai/cudf) doesn't support non-empty nulls.

## Ordering of Output

There are some operators where Spark does not guarantee the order of the output.
Expand Down