Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add JNI methods for detecting and purging non-empty nulls from LIST and STRUCT #12742

Merged
merged 19 commits into from
Feb 27, 2023
Merged

Add JNI methods for detecting and purging non-empty nulls from LIST and STRUCT #12742

Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
0f72ac7
Added Jni methods for detecting and purging non-empty nulls
razajafri Feb 9, 2023
2b29f3b
Merge remote-tracking branch 'origin/branch-23.04' into SP-5430-non-e…
razajafri Feb 9, 2023
7abf384
fix formatting
razajafri Feb 9, 2023
097a09a
addressed review comments
razajafri Feb 9, 2023
dac0b21
removed the unused native method
razajafri Feb 10, 2023
9c1ecc0
Merge remote-tracking branch 'origin/branch-23.04' into SP-5430-non-e…
razajafri Feb 10, 2023
d9bd29a
reformatted
razajafri Feb 10, 2023
0a77f17
reformat
razajafri Feb 13, 2023
a70910c
Merge remote-tracking branch 'origin/branch-23.04' into SP-5430-non-e…
razajafri Feb 13, 2023
faa5e76
Merge branch 'branch-23.04' into SP-5430-non-empty
razajafri Feb 16, 2023
7cd46bd
Merge branch 'branch-23.04' into SP-5430-non-empty
razajafri Feb 17, 2023
7378c6a
Merge branch 'branch-23.04' into SP-5430-non-empty
ttnghia Feb 17, 2023
01688bb
Merge branch 'branch-23.04' into SP-5430-non-empty
razajafri Feb 22, 2023
6b12700
Merge branch 'branch-23.04' into SP-5430-non-empty
ajschmidt8 Feb 24, 2023
2843bc8
Merge branch 'branch-23.04' into SP-5430-non-empty
razajafri Feb 24, 2023
fbb5e49
closing the vectors
razajafri Feb 24, 2023
508b9e6
release local allocation
razajafri Feb 25, 2023
a8a1106
Return an Array of ColumnViews so they can be closed by the caller
razajafri Feb 25, 2023
b1f84cb
Merge branch 'branch-23.04' into SP-5430-non-empty
razajafri Feb 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 22 additions & 17 deletions java/src/test/java/ai/rapids/cudf/ColumnVectorTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6693,6 +6693,9 @@ void testApplyBooleanMaskFromListOfStructure() {
}
}

/**
* The caller needs to make sure to close the returned ColumnView
ttnghia marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

@ttnghia ttnghia Feb 24, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only the returned ColumnView is not enough. There are more local variables/columns need to be closed.

Suggested change
* The caller needs to make sure to close the returned ColumnView
* The caller needs to make sure to call this function in a `try` block as local variables need to be closed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ugh, I am rushing through this and context-switching isn't helping either. I think it will make sense for me to close all the local allocations and leave only the returned vector to be closed by the caller.

*/
private ColumnView getColumnViewWithNonEmptyNulls() {
List<Integer> list0 = Arrays.asList(1, 2, 3);
List<Integer> list1 = Arrays.asList(4, 5, null);
Expand Down Expand Up @@ -6722,26 +6725,28 @@ private ColumnView getColumnViewWithNonEmptyNulls() {

@Test
void testPurgeNonEmptyNullsList() {
ColumnView colWithNonEmptyNulls = getColumnViewWithNonEmptyNulls();
// purge non-empty nulls
assertTrue(colWithNonEmptyNulls.hasNonEmptyNulls());
ColumnView colWithEmptyNulls = colWithNonEmptyNulls.purgeNonEmptyNulls();
ColumnVector expectedOffsetsAfterPurge = ColumnVector.fromInts(0, 3, 3, 6, 6);
ColumnView offsetsCvAfterPurge = colWithEmptyNulls.getListOffsetsView();
assertColumnsAreEqual(expectedOffsetsAfterPurge, offsetsCvAfterPurge);
assertFalse(colWithEmptyNulls.hasNonEmptyNulls());
try (ColumnView colWithNonEmptyNulls = getColumnViewWithNonEmptyNulls();
// purge non-empty nulls
ColumnView colWithEmptyNulls = colWithNonEmptyNulls.purgeNonEmptyNulls();
ColumnVector expectedOffsetsAfterPurge = ColumnVector.fromInts(0, 3, 3, 6, 6);
ColumnView offsetsCvAfterPurge = colWithEmptyNulls.getListOffsetsView()) {
assertTrue(colWithNonEmptyNulls.hasNonEmptyNulls());
assertColumnsAreEqual(expectedOffsetsAfterPurge, offsetsCvAfterPurge);
assertFalse(colWithEmptyNulls.hasNonEmptyNulls());
}
}

@Test
void testPurgeNonEmptyNullsStruct() {
ColumnView listCol = getColumnViewWithNonEmptyNulls();
ColumnView stringsCol = ColumnVector.fromStrings("A", "col", "of", "Strings");
ColumnView structView = ColumnView.makeStructView(stringsCol, listCol);
ColumnView structWithEmptyNulls = structView.purgeNonEmptyNulls();
ColumnView newListChild = structWithEmptyNulls.getChildColumnView(1);
ColumnVector expectedOffsetsAfterPurge = ColumnVector.fromInts(0, 3, 3, 6, 6);
ColumnView offsetsCvAfterPurge = newListChild.getListOffsetsView();
assertColumnsAreEqual(expectedOffsetsAfterPurge, offsetsCvAfterPurge);
assertFalse(newListChild.hasNonEmptyNulls());
try (ColumnView listCol = getColumnViewWithNonEmptyNulls();
ColumnView stringsCol = ColumnVector.fromStrings("A", "col", "of", "Strings");
ColumnView structView = ColumnView.makeStructView(stringsCol, listCol);
ColumnView structWithEmptyNulls = structView.purgeNonEmptyNulls();
ColumnView newListChild = structWithEmptyNulls.getChildColumnView(1);
ColumnVector expectedOffsetsAfterPurge = ColumnVector.fromInts(0, 3, 3, 6, 6);
ColumnView offsetsCvAfterPurge = newListChild.getListOffsetsView()) {
assertColumnsAreEqual(expectedOffsetsAfterPurge, offsetsCvAfterPurge);
assertFalse(newListChild.hasNonEmptyNulls());
}
}
}