Use ChunkedSliceOutput to chunk and buffer writes in parquet

[raunaqmorarka]

Description

Avoids separate calls to output stream for writing page headers and page data.
Avoids holding on to extra memory for each buffered page due to over-allocation by compressors.
Adds memory usage accounting for buffered pages.

Additional context and related issues

Should help with the memory usage accounting problems described in #18557

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:

# Hive, Hudi, Iceberg, Delta
* Improve memory usage accounting of parquet writer. ({issue}`18564`)

[findepi]

Avoids separate calls to output stream for writing page headers and page data.

not sure i understand the goal here.
are we saving on http requests? i think the output stream is buffering internally?

can you expand a bit the commit rationale so that it's more obvious what's the objective?

[raunaqmorarka]

Avoids separate calls to output stream for writing page headers and page data.

not sure i understand the goal here. are we saving on http requests? i think the output stream is buffering internally?

can you expand a bit the commit rationale so that it's more obvious what's the objective?

Yes, there is varying levels of buffering in different output stream implementations. It's not clear to me that every implementation is doing that or that we should rely on that always being the case. In any case, that's not the main benefit of this change.

The main objective here is

Avoids holding on to extra memory for each buffered page due to over-allocation by compressors.
Adds memory usage accounting for buffered pages.

Also, ORC writer uses ChunkedSliceOutput in OrcOutputBuffer#writeChunkToOutputStream in a similar way.

[raunaqmorarka]

Insert benchmarks.pdf

There is a big increase in the peak memory usage due to the improved accounting and small decrease in perf due to the extra byte array copy required to get rid of over-allocation in compression buffer.

[lukasz-stec]

lgtm % comments

[sopel39]

Is it called after a flush? (is there something like a flush here)?

[raunaqmorarka]

There is no "flush" here, there is a "close" followed by a getBuffer to extract the buffered pages.
This is called by connector page sink for every writer after writing each page.