You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
ProjectionExec does not preserve field metadata when generating the projected schema. I believe this will also cause it to lose dictionary information for the field.
I think fixing this will require extending PhysicalExpr with a fn field(&self, input_schema: &Schema) -> Result<Field>; or similar to replace the current logic that just constructs a new Field
To Reproduce
Create a RecordBatch with a schema with field metadata and pass it through ProjectionExec.
Expected behavior
There is potentially discussion to be had w.r.t schema-level metadata, but I would expect field-level metadata to be preserved for the fields that are projected
The text was updated successfully, but these errors were encountered:
I guess I was expecting that the Column expression would preserve field metadata, and all other expressions would continue to do what they currently do. Part of the challenge is that ProjectionExec can be used for more than just basic column projection due to its formulation in terms of expressions. This isn't necessarily a problem, but I'd just like to be able to project a schema without losing field metadata - as within IOx we use this for additional field type information
Describe the bug
ProjectionExec
does not preserve field metadata when generating the projected schema. I believe this will also cause it to lose dictionary information for the field.I think fixing this will require extending PhysicalExpr with a
fn field(&self, input_schema: &Schema) -> Result<Field>;
or similar to replace the current logic that just constructs a new FieldTo Reproduce
Create a RecordBatch with a schema with field metadata and pass it through ProjectionExec.
Expected behavior
There is potentially discussion to be had w.r.t schema-level metadata, but I would expect field-level metadata to be preserved for the fields that are projected
The text was updated successfully, but these errors were encountered: