PARQUET-2413: Support configurable extraMetadata in ParquetWriter

[clairemcginty]

https://issues.apache.org/jira/browse/PARQUET-2413

Adds support for Configurable extraMetadata in Parquet file footer. This makes it easier for users to migrate from Avro to Parquet (since Avro supports custom metadata keys).

I chose this approach (parsing values from a preset Configuration key prefix) because (a) Configuration is already Stringable, so no need to worry about object-to-String conversion, and (b) it doesn't require any API changes (i.e. adding withExtraMetadata Builder options to all Parquet writers/implementations).

Alternate approaches:

• Add withExtraMetadata Builder method to ParquetWriter; in ParquetWriter#build, append all to the value of WriteContext#getExtraMetadata().
• Add extraMetadata class variables to all WriteSupport implementations, and pass to WriteContext in WriteSupport#init (i.e. AvroWriteSupport).

Let me know if either of those approaches are preferable to this one!

Jira

• My PR addresses the following Parquet Jira issues and references
  them in the PR title. For example, "PARQUET-1234: My Parquet PR"
  • https://issues.apache.org/jira/browse/PARQUET-XXX
  • In case you are adding a dependency, check if the license complies with
    the ASF 3rd Party License Policy.

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

• My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines
  from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Style

• [z] My contribution adheres to the code style guidelines and Spotless passes.
  • To apply the necessary changes, run mvn spotless:apply -Pvector-plugins

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain Javadoc that explain what it does

[clairemcginty]

This method is now available through the ParquetWriter.Builder superclass.

[wgtmac]

Is it possible to add an overload here to suppress the complaint of japicmp? That's more preferable than an exclusion.

[clairemcginty]

great idea! updated + removed the exclusion.

[wgtmac]

@Fokko @ConeyLiu Do you have any comment?

[ConeyLiu]

+1 thanks @wgtmac ping me. And thanks @clairemcginty for the contribution.

[wgtmac]

@clairemcginty It seems that we need to add an exclusion to japicmp to make the CI happy.

[clairemcginty]

@clairemcginty It seems that we need to add an exclusion to japicmp to make the CI happy.

Added! The CI on my fork's GHA seems to be happy now

[wgtmac]

Thanks for the change! I've left some minor comments.

[wgtmac]

Can we avoid specializing any key? IIUC, it can also be caught at line 422 if OBJECT_MODEL_NAME_PROP has been set already.

[clairemcginty]

This key is kind of a special case since it's added by the delegated InternalParqueRecordWriter at the end of writing: https://github.com/apache/parquet-mr/blob/945836c/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/InternalParquetRecordWriter.java#L132-L134

So if we remove this extra check and there's a conflicting OBJECT_MODEL_NAME_PROP key in extraMetaData, InternalParquetRecordWriter will silently overwrite it at the end.

[wgtmac]

Is it possible to add an overload here to suppress the complaint of japicmp? That's more preferable than an exclusion.

[wgtmac]

+1

Thanks @clairemcginty and @ConeyLiu!