PARQUET-2417: Add support for geometry logical type

[zhangfengcdt]

This PR is to provide a POC to support the proposed changes to the parquet-format to add geometry type to parquet.

Here is the proposal: apache/parquet-format#240

Jira

• My PR addresses the following Parquet Jira issues and references them in the PR title. For example, "PARQUET-1234: My Parquet PR"
  • https://issues.apache.org/jira/browse/PARQUET-2471

Tests

• My PR adds the following unit tests: TestGeometryTypeRoundTrip

Commits

• My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain Javadoc that explain what it does

[zhangfengcdt]

CC: @jiayuasu @Kontinuation @wgtmac

[wgtmac]

Thanks for the update! I have left some comments. I think we are reaching the finish line!

[wgtmac]

Thanks for adding these tests!

I think we are missing tests in following cases:

• verify geometry type metadata is well preserved.
• verify all kinds of geometry stats are preserved, including bbox, covering and geometry types.
• verify geo stats in the column index have been generated.

I can do these later.

[jiayuasu]

@wgtmac please take a look :-)

[wgtmac]

Sure, I will take a look. Thanks!

[Kontinuation]

I am depending on this PR to build geo support for iceberg. I got lots of test failures when building this branch locally:

java.lang.NullPointerException
	at org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetStatistics(ParquetMetadataConverter.java:965)
	at org.apache.parquet.format.converter.ParquetMetadataConverter.buildColumnChunkMetaData(ParquetMetadataConverter.java:1750)
	at org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetMetadata(ParquetMetadataConverter.java:1848)
	at org.apache.parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:1728)
	at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:629)
	at org.apache.parquet.hadoop.ParquetFileReader.<init>(ParquetFileReader.java:934)
	at org.apache.parquet.hadoop.ParquetFileReader.<init>(ParquetFileReader.java:925)
	at org.apache.parquet.hadoop.ParquetFileReader.open(ParquetFileReader.java:698)

NPE is thrown when reading parquet files without geo columns. Can we apply the following patch to resolve this problem?

diff --git a/parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java b/parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java
index 3efc9345..22e51783 100644
--- a/parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java
+++ b/parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java
@@ -961,6 +961,9 @@ public class ParquetMetadataConverter {
 
   static org.apache.parquet.column.statistics.geometry.GeospatialStatistics fromParquetStatistics(
       GeospatialStatistics formatGeomStats, PrimitiveType type) {
+    if (formatGeomStats == null) {
+      return null;
+    }
     org.apache.parquet.column.statistics.geometry.BoundingBox bbox = null;
     if (formatGeomStats.isSetBbox()) {
       BoundingBox formatBbox = formatGeomStats.getBbox();

[Kontinuation]

Should we define an enum for edge interpolation algorithms?

[zhangfengcdt]

They are already auto generated from the thrift protocol in EdgeInterpolationAlgorithm.

[zhangfengcdt]

The reason it is not used here is because if we do, it will generate loop dependency.

[Kontinuation]

parquet-column defines its own enum for the enums that are defined by thrift. Examples are enum Encoding and enum CompressionCodecName. I'm not sure if we should follow the same convention for EdgeInterpolationAlgorithm.

[Kontinuation]

We'd better use equals instead of == to compare Strings to be safe, even when the String instances are being reused. I think this is left out when reverting edgeAlgorithm from enum to String.

[Kontinuation]

Hashing both crs and edgeAlgorithm.