[Parquet] DELTA_BINARY_PACKED constraint on num_bits is too restrict?

[asfimport]

Consider the sequence of (int32) values

[863490391,-816295192,1613070492,-1166045478,1856530847]

This sequence can be encoded as a single block, single miniblock with a bit_width of 33.

However, we currently require [1] the bit_width of each miniblock to be smaller than the bitwidth of the type it encodes.

We could consider lifting this constraint, as, as shown in the example above, the values representation's bit_width can be smaller than the delta's representation's bit_width.

[1]

arrow/cpp/src/parquet/encoding.cc

Line 2173 in a376968

if (bit_width_data[i] > kMaxDeltaBitWidth) {

Reporter: Jorge Leitão / @jorgecarleitao

Original Issue Attachments:

• test.parquet

_{Note: This issue was originally created as ARROW-17465. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
It would be useful to find out what parquet-mr (the Java Parquet implementation) does about this.
In particular, we don't want to produce files that wouldn't be readable by parquet-mr, I think.

[asfimport]

Jorge Leitão / @jorgecarleitao:
Agree. The issue was not very clear - we currently only support reading DELTA_BINARY_PACKED, and it is in the read path that currently bails.

Note that pyspark can read it. Attached to this comment is a minimal parquet file reproducing the issue. The code example results in

Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/08/18 21:14:28 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
spark reads it                                                                  
Traceback (most recent call last):
 File "bla.py", line 16, in <module>
   table = pyarrow.parquet.read_table("test.parquet")
 File "/home/azureuser/projects/arrow2/arrow-parquet-integration-testing/venv/lib/python3.8/site-packages/pyarrow/parquet/__init__.py", line 2827, in read_table
   ...
OSError: delta bit width larger than integer bit width

which how I concluded that this is something specific in arrow :)

import pyarrow.parquet
import pyspark.sql


spark = pyspark.sql.SparkSession.builder.config(
    # see https://stackoverflow.com/a/62024670/931303
    "spark.sql.parquet.enableVectorizedReader",
    "false",
).getOrCreate()

result = spark.read.parquet("test.parquet").collect()
assert [r["c1"] for r in result] == [863490391, -816295192, 1613070492, -1166045478, 1856530847]
print("spark reads it")

table = pyarrow.parquet.read_table("test.parquet")
print("pyarrow reads it")

test.parquet

[asfimport]

Antoine Pitrou / @pitrou:
Hmm, it seems it will be complicated for Parquet C++ to accept this without some rearchitecturing of the decoder.
(roughly, if we decode Int32 values, we first decode the deltas as a buffer of Int32 as well...)

I would suggest first ask on the Parquet ML whether this is intended to be supported.

(note that of course for such data, DELTA_BINARY_PACKED should not be used at all, as it produces an expansion... while being more CPU-intensive to decode as well)

[asfimport]

Antoine Pitrou / @pitrou:
Also, for Int64 data, it would mean we would have to accept 65-bit deltas.

[mapleFU]

@etseidl After #37940 this is applied, would we prevent from generating new 33/65 bits data? Though we can still read from legacy data.

[etseidl]

@mapleFU AFAIK the arrow writer does not produce 33/65 bit data either before or after #37940, and still cannot read the bitwidths larger than the physical type. This reminds me that I still would like to add some clarification to the specification around this issue.

[etseidl]

This issue should be closed given the changes to the Parquet specification in PARQUET-2435.

cc @pitrou @mapleFU

[pitrou]

Thanks @etseidl ! Closing then.