Optimized Writing of Arrow Byte Array to Parquet

[tustvold]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

A significant amount of effort has been put into making the reading of byte arrays from parquet fast:

• Generify ColumnReaderImpl and RecordReader (#1040)  #1041
• parquet: Optimized ByteArrayReader, Add UTF-8 Validation (#1040)  #1082
• Preserve dictionary encoding when decoding parquet into Arrow arrays, 60x perf improvement (#171) #1180

We should invest some effort in making the writer performance comparable.

Describe the solution you'd like

Currently in order to write byte array types from arrow:

• Any dictionaries are hydrated
• Each value from a string array is separately allocated into a Vec<ByteArray>
• These values are then written using the ColumnWriter

It would be a significant performance win to be able to elide these first two steps. This would likely involve much the same process as was followed for the reader:

• Generify ColumnWriter to allow writing from different buffers
• Add the ability to write from an arrow ByteArray directly
• Add the ability to write from an arrow dictionary array directly

Describe alternatives you've considered

We could not do this