Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panic when read parquet file with field "__index_level_0__" #233

Closed
icefed opened this issue Apr 23, 2020 · 2 comments
Closed

Panic when read parquet file with field "__index_level_0__" #233

icefed opened this issue Apr 23, 2020 · 2 comments

Comments

@icefed
Copy link

icefed commented Apr 23, 2020

We get panic when read parquet file, the file is created by pandas.
apache/arrow#2244

panic: reflect.StructOf: field "__index_level_0__" is unexported but missing PkgPath

goroutine 1 [running]:
reflect.runtimeStructField(0xc00001a7e0, 0x11, 0x0, 0x0, 0x1512880, 0x13d28e0, 0x0, 0x0, 0x0, 0x0, ...)
	/usr/local/Cellar/go/1.14.2_1/libexec/src/reflect/type.go:2769 +0x1df
reflect.StructOf(0xc000106000, 0x14, 0x27, 0x0, 0x0)
	/usr/local/Cellar/go/1.14.2_1/libexec/src/reflect/type.go:2385 +0x1e3f
github.com/xitongsys/parquet-go/schema.(*SchemaHandler).GetTypes(0xc000013260, 0xc00001c1c2, 0x6, 0xc00001c1c2)
	.../pkg/mod/github.com/xitongsys/[email protected]/schema/gettype.go:78 +0x7ae
@xitongsys
Copy link
Owner

xitongsys commented Apr 24, 2020

hi, @icefed
This issue is due to the field name with underline prefix __index_level_0__.
It doesn't have an upper case prefix which is unexported and the reflection can't assign value.
Fix in latest commits.
I have checked it and it works now

parquet-tools.exe -cmd cat
-file .\a.parquet
[{"One":-1,"Two":"foo","Three":true,"P___index_level_0__":"a"},{"One":null,"Two":"bar","Three":
false,"P___index_level_0__":"b"},{"One":2.5,"Two":"baz","Three":true,"P___index_level_0__":"c"}
]

@icefed
Copy link
Author

icefed commented Apr 24, 2020

hi, @icefed
This issue is due to the field name with underline prefix __index_level_0__.
It doesn't have an upper case prefix which is unexported and the reflection can't assign value.
Fix in latest commits.
I have checked it and it works now

parquet-tools.exe -cmd cat
-file .\a.parquet
[{"One":-1,"Two":"foo","Three":true,"P___index_level_0__":"a"},{"One":null,"Two":"bar","Three":
false,"P___index_level_0__":"b"},{"One":2.5,"Two":"baz","Three":true,"P___index_level_0__":"c"}
]

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants