Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix map column can not be non-nullable for java #14003

Merged
merged 9 commits into from
Sep 6, 2023
29 changes: 29 additions & 0 deletions java/src/main/java/ai/rapids/cudf/ColumnWriterOptions.java
Original file line number Diff line number Diff line change
Expand Up @@ -522,7 +522,10 @@ protected String[] getFlatColumnNames(String[] ret) {
* Maps are List columns with a Struct named 'key_value' with a child named 'key' and a child
* named 'value'. The caller of this method doesn't need to worry about this as this method will
* take care of this without the knowledge of the caller.
* Note: this method always returns a nullabe column, can not return non-nullable column.
ttnghia marked this conversation as resolved.
Show resolved Hide resolved
ttnghia marked this conversation as resolved.
Show resolved Hide resolved
* Do not use this, use the next function with the parameter `isNullable`.
*/
@Deprecated
public static ColumnWriterOptions mapColumn(String name, ColumnWriterOptions key,
ColumnWriterOptions value) {
StructColumnWriterOptions struct = structBuilder("key_value").build();
Expand All @@ -537,6 +540,32 @@ public static ColumnWriterOptions mapColumn(String name, ColumnWriterOptions key
return opt;
}

/**
* Add a Map Column to the schema.
* <p>
* Maps are List columns with a Struct named 'key_value' with a child named 'key' and a child
* named 'value'. The caller of this method doesn't need to worry about this as this method will
* take care of this without the knowledge of the caller.
*
* Note: If this map column is a key of another map, should pass isNullable = false.
* e.g.: map1(map2(int, int), int) the map2 should be non-nullable.
*
* @param isNullable is the returned map nullable.
*/
public static ColumnWriterOptions mapColumn(String name, ColumnWriterOptions key,
ColumnWriterOptions value, Boolean isNullable) {
if (key.isNullable) {
throw new IllegalArgumentException("key column can not be nullable");
}
StructColumnWriterOptions struct = structBuilder("key_value").build();
struct.childColumnOptions = new ColumnWriterOptions[]{key, value};
ColumnWriterOptions opt = listBuilder(name, isNullable)
revans2 marked this conversation as resolved.
Show resolved Hide resolved
.withStructColumn(struct)
.build();
opt.isMap = true;
return opt;
}

/**
* Creates a ListBuilder for column called 'name'
*/
Expand Down
6 changes: 4 additions & 2 deletions java/src/test/java/ai/rapids/cudf/TableTest.java
Original file line number Diff line number Diff line change
Expand Up @@ -8064,7 +8064,8 @@ void testParquetWriteMap() throws IOException {
ParquetWriterOptions options = ParquetWriterOptions.builder()
.withMapColumn(mapColumn("my_map",
new ColumnWriterOptions("key0", false),
new ColumnWriterOptions("value0"))).build();
new ColumnWriterOptions("value0"),
true)).build();
File f = File.createTempFile("test-map", ".parquet");
List<HostColumnVector.StructData> list1 =
Arrays.asList(new HostColumnVector.StructData(Arrays.asList("a", "b")));
Expand Down Expand Up @@ -8562,7 +8563,8 @@ void testORCWriteMapChunked() throws IOException {
ORCWriterOptions options = ORCWriterOptions.builder()
.withMapColumn(mapColumn("my_map",
new ColumnWriterOptions("key0", false),
new ColumnWriterOptions("value0"))).build();
new ColumnWriterOptions("value0"),
true)).build();
File f = File.createTempFile("test-map", ".parquet");
List<HostColumnVector.StructData> list1 =
Arrays.asList(new HostColumnVector.StructData(Arrays.asList("a", "b")));
Expand Down