Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to read date format columns (int96 type) from avro-parquet schema #22

Open
jeremihas-caruso opened this issue Mar 25, 2024 · 1 comment

Comments

@jeremihas-caruso
Copy link

I am facing the following exception when reading the parquet file having date column:

java.lang.IllegalArgumentException: INT96 is deprecated. As interim enable READ_INT96_AS_FIXED flag to read as byte array.

at org.apache.parquet.avro.AvroSchemaConverter$1.convertINT96(AvroSchemaConverter.java:331)
at org.apache.parquet.avro.AvroSchemaConverter$1.convertINT96(AvroSchemaConverter.java:313)
at org.apache.parquet.schema.PrimitiveType$PrimitiveTypeName$7.convert(PrimitiveType.java:341)
at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:312)
at org.apache.parquet.avro.AvroSchemaConverter.convertFields(AvroSchemaConverter.java:290)
at org.apache.parquet.avro.AvroSchemaConverter.convert(AvroSchemaConverter.java:279)
at org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:134)
at org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:190)
at org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:166)
at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135)
at com.databricks.labs.delta.sharing.java.format.parquet.TableReader.read(TableReader.java:57)
@jeremihas-caruso
Copy link
Author

jeremihas-caruso commented Mar 25, 2024

INT96 is deprecated as we can see in stackoverflow https://stackoverflow.com/questions/55829202/unable-to-read-date-format-columns-int96-type-from-avro-parquet-schema-in-apac
The solutions is set "parquet.avro.readInt96AsFixed" configuration to "true" when build the reader.

jeremihas-caruso added a commit to jeremihas-caruso/delta-sharing-java-connector that referenced this issue Mar 25, 2024
…ema (databrickslabs#22)

INT96 is deprecated so we must set "parquet.avro.readInt96AsFixed" configuration to "true" when build the reader.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant