Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support reading parquet maps from older writers #95

Open
parsnips opened this issue Nov 30, 2022 · 0 comments · May be fixed by #96
Open

Support reading parquet maps from older writers #95

parsnips opened this issue Nov 30, 2022 · 0 comments · May be fixed by #96

Comments

@parsnips
Copy link

Describe the bug
AWS Kinesis produces maps in an old way, most notably using map instead of key_value

example:

optional group new (MAP) {
    repeated group map (MAP_KEY_VALUE) {
      required binary key (UTF8);
      optional group value {
        optional binary b (UTF8);
        optional binary n (UTF8);
      }
    }
  }

When trying to populate a struct from a parquet file with this, it'll throw an error.

Unit test to reproduce

https://github.com/parsnips/parquet-go-1/tree/parsnips/support-filling-map has unit test and fix.

parquet-go specific details
reproduced on the latest version on main.

Misc Details
This is parquet generated by AWS Kinesis Firehose ParquetSerDe V2.

I have a test file here: https://gist.github.com/parsnips/928c14d850331dd21c8d917227f77c5c#file-kinesisserde-v2-parquet

@parsnips parsnips linked a pull request Nov 30, 2022 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant