Update JNI JSON reader column compatability for Spark #13477
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
revans2
added
bug
jlowe
approved these changes
May 31, 2023
Comment on lines
+82
to
+86
| DType[] ret = new DType[types.size()]; | ||
| for (int i = 0; i < types.size(); i++) { | ||
| ret[i] = types.get(i); | ||
| } | ||
| return ret; |
There was a problem hiding this comment.
Suggested change
| DType[] ret = new DType[types.size()]; | |
| for (int i = 0; i < types.size(); i++) { | |
| ret[i] = types.get(i); | |
| } | |
| return ret; | |
| return types.toArray(new DType[types.size()]); |
There was a problem hiding this comment.
Neat. That would pack nicely to:
return types == null ? null : types.toArray(new DType[types.size()]);
Comment on lines
240
to
242
| int[] dTypeIds, int[] dTypeScales, | ||
| String filePath, long address, long length, | ||
| boolean dayFirst, boolean lines) throws CudfException; |
There was a problem hiding this comment.
Nit: These parameters are now mis-indented
| // We might need to rearrange the columns to match what we want. | ||
| DType[] types = schema.getTypes(); | ||
| ColumnVector[] columns = new ColumnVector[neededColumns.length]; | ||
| try (Table tbl = twm.releaseTable()) { |
There was a problem hiding this comment.
Nit: Seems like this should be the first thing in the method so no matter what happens (NPE, whatever) we're closing the table.
There was a problem hiding this comment.
TableWithMetadata will close the table if we don't pull it out first.
| @@ -968,6 +971,42 @@ public static Table readJSON(Schema schema, JSONOptions opts, byte[] buffer) { | |||
| return readJSON(schema, opts, buffer, 0, buffer.length); | |||
| } | |||
|
|
|||
| private static Table gatherJSONColumns(Schema schema, TableWithMeta twm) { | |||
There was a problem hiding this comment.
Would be nice to have a comment on this method explaining it will release and close the underlying table of twm as a side-effect.
mythrocks
reviewed
May 31, 2023
mythrocks
approved these changes
May 31, 2023
|
/merge |
|
/merge |
This was referenced Jun 8, 2023
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This moves the logic to update the columns returned from the JSON reader to java. It also updated the code to be able to deal with requested columns that were not in the data. It is not perfect because it will not work if the input file had no columns at all in it.
But it fixes issues for a file that has valid columns in it, but none of them are the columns that we requested.
This is a work around for #13473, but is not perfect.
Checklist