[Java] BaseVariableWidthVector only supports 1 GiB through safe interfaces

[Kimahriman]

Describe the bug, including details regarding any error messages, version, and platform.

apache/arrow#13815 fixed an issue with a misleading error caused by going over the 2GiB limit of a variable width vector, but it also introduced a 1 GiB limit when using the safe interfaces. Basically whenever you try to add data beyond 1 GiB, the vector will try to double itself to the next power of two, which would be 2147483648, which is greater than Integer.MAX_VALUE which is 2147483647, thus throwing a OversizedAllocationException. This effectively limits the total size allowed in the buffer to 1 GiB, even though it should only really be bound by the max offset being less than Integer.MAX_VALUE.

See apache/spark#39572 (comment) and the comment above it for how I could recreate the issue.

Component(s)

Java

[jjbskir]

Hey! I ran into the same issue recently. I wanted to see if there was any progress to support variables between 1GB to 2GB? Or any supported work arounds?

Happy to open a PR to support this. I think an solution would to add an additional check in reallocDataBuffer if the desiredAllocSize is greater then 1GB, but less then 2GB to just set the newAllocationSize to the max value.