You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
I uncovered a small bug in schema.go, the recursiveFix func overwrites the passed in colPath by using it in the append function to add the column name to the column path on L684. Since go slices may or may not reuse the underlying array depending on capacity, this does not always necessarily return a slice that points to a newly allocated array (I have been bitten by this bug before, it's one of go's "gotchas"). This resulted in the go schema to have the same column path for all fields in a list element and thus invalid parquet.
Using this schema, all the column paths for the list elements were ["results", "list", "element", "test_end_time"] because test_end_time is the last field in the group and the recursive function continuously overwrites the colPath argument and assigns it to each column's path field.
I can put up a PR for the fix, it is a one-line change:
Describe the bug
I uncovered a small bug in schema.go, the
recursiveFix
func overwrites the passed incolPath
by using it in theappend
function to add the column name to the column path on L684. Since go slices may or may not reuse the underlying array depending on capacity, this does not always necessarily return a slice that points to a newly allocated array (I have been bitten by this bug before, it's one of go's "gotchas"). This resulted in the go schema to have the same column path for all fields in a list element and thus invalid parquet.Using this schema, all the column paths for the list elements were
["results", "list", "element", "test_end_time"]
becausetest_end_time
is the last field in the group and the recursive function continuously overwrites thecolPath
argument and assigns it to each column'spath
field.I can put up a PR for the fix, it is a one-line change:
Let me know if I can provide any more info, thanks!
The text was updated successfully, but these errors were encountered: