Provide Arrow Schema Hint to Parquet Reader

[tustvold]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

The parquet reader automatically uses an embedded arrow schema to hint type inference for decode. In particular if the hinted type is compatible with the underlying parquet type, it performs a cast.

Describe the solution you'd like

In situations where the writer was not an arrow writer this schema is not available, and therefore the arrow types are inferred from the parquet schema. This is not always desirable:

• Coerce parquet int96 timestamps to microsecond precision #5655
• Account for Timezone when Casting Timestamp to Date32 #5605 (comment)

Describe alternatives you've considered

Additional context

[alamb]

Here is one potential API

let file = File::open("data.parquet").unwrap();

let reader = ParquetRecordBatchReaderBuilder::try_new(file)?
  // specify column "time" should be UTC
  // will error if this type can not be read from parquet
  .with_column_type("time", DateTime::Timestamp(Nanoseconds, Some("UTC"))

println!("Converted arrow schema is: {}", builder.schema());

I am not quite sure how to handle identifying nested types with a single column name

Like if the parquet file has

{
  "my_object": { 
    "time": "12-01-02"
  }
}

maybe we would refer to the time field like "my_object.time"?

[tustvold]

I think my expectation would be for you to provide the SchemaRef for the entire file

[Lordworms]

Let me try the remaining part if it is ok

[liukun4515]

I think my expectation would be for you to provide the SchemaRef for the entire file

Basically agree with your idea
In the datafusion, the ParquetExec of FileScanConfig contains the schema for the parquet file, but I think the provided SchemaRef should be the optional for the parquet reader when infer the data type

[alamb]

Something like

let file = File::open("data.parquet").unwrap();

let reader = ParquetRecordBatchReaderBuilder::try_new(file)?
  // specify the arrow schema to read from this parquet file
  // will error if the types in the parquet file can not be converted
  // into the specific types. 
  // Will ignore any embedded metadata about types when written
  .schema(schema)

println!("Converted arrow schema is: {}", builder.schema());

[liukun4515]

Something like

let file = File::open("data.parquet").unwrap();

let reader = ParquetRecordBatchReaderBuilder::try_new(file)?
  // specify the arrow schema to read from this parquet file
  // will error if the types in the parquet file can not be converted
  // into the specific types. 
  // Will ignore any embedded metadata about types when written
  .schema(schema)

println!("Converted arrow schema is: {}", builder.schema());

Do we need to add some checker in the function of the schema which is used to compare the input schema with schema inferred from the parquet file?

The compatibility is very important for the parquet reader

[tustvold]

The inference logic is already setup to use the arrow schema as a hint as opposed to authoritative , if you give it something invalid it will just ignore it

[liukun4515]

The inference logic is already setup to use the arrow schema as a hint as opposed to authoritative , if you give it something invalid it will just ignore it

thanks, got it.

[efredine]

I noticed this seemed to have stalled, so I thought I would have a go at it.

The implemented API is different from the one discussed. In particular, the schema is supplied as an ArrowReaderOption.

let file = File::open("file.parquet");
let schema = Arc::new(Schema::new(vec![Fields::new("col", DataType::Int32, false)]));
let options = ArrowReaderOptions::new().with_schema(schema);
// This may fail as described above.
let builder = ParquetRecordBatchReaderBuilder::try_new_with_options(file, options).unwrap();
let reader = builder.build().unwrap();

This is necessary because it needs to be provided as a hint when the metadata is read from the parquet file.

The tests are incomplete and there are some questions in the PR with respect to how error handling should be handled.

[alamb]

label_issue.py automatically added labels {'arrow'} from #5605

[alamb]

label_issue.py automatically added labels {'parquet'} from #5939