new feature: Expose API to configure chunk size for java binding

[BaurzhanSakhariev]

Feature Description

Ability to configure chunk size for Operator

Problem and Solution

OperatorOutputStream uses hardcoded chunk size which leads to BlockCountExceedsLimit error when uploading a large file using azblob service. Increasing chunk size helps to have fewer blocks and avoid this error.

Also, upload time decreased significantly with buffer size 10x of the current size.

I used a test

@Test
public void testLargeFile() {
  final String path = UUID.randomUUID().toString();
  final Random random = new Random();
  final int size = 16384 * 10; // 10 x OperatorOutputStream.MAX_BYTES (10 flushes per write)
  final byte[] content = new byte[size];
  random.nextBytes(content);
  try (Operator op = op(); OperatorOutputStream operatorOutputStream = op.createOutputStream(path)) {
      for (int i = 0; i < 20000; i++) { // More iterations in case BlockCountExceedsLimit doesn't pop up exactly after 100K blocks.
          operatorOutputStream.write(content);
      }
  } catch (IOException e) {
      throw new RuntimeException(e);
  }
}

against a real Azure container and got an error after exactly 100K iterations.

Then manually changed OperatorOutputStream.MAX_BYTES to make it 10x.
Test passed and file has been uploaded to the container.

Additional Context

#5420 and crate/crate#17136

[Xuanwo]

Thank you @tisonkun for fixing this!

[BaurzhanSakhariev]

This will be part of the 0.51.0, right?