Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix addMetadataFields() to carry over 'props' #483

Closed
shangxinli opened this issue Oct 11, 2018 · 0 comments
Closed

Fix addMetadataFields() to carry over 'props' #483

shangxinli opened this issue Oct 11, 2018 · 0 comments

Comments

@shangxinli
Copy link
Contributor

shangxinli commented Oct 11, 2018

Inside addMetadataFields(), it trunks the 'props' after adding the metadata. This blocks scenario where it relies those 'props' to let Parquet do some work like encryption. The fix is simple by adding the 'props'.

 for (Schema.Field field : schema.getFields()) {
   if (!isMetadataField(field.name())) {
  •    Schema.Field newField = new Schema.Field(field.name(), field.schema(), field.doc(), null);
    
  •    for (Map.Entry<String, JsonNode> prop : field.getJsonProps().entrySet()) {
    
  •      newField.addProp(prop.getKey(), prop.getValue());
    
  •    }
    
  •    parentFields.add(newField);
     }
    
    }

This passed the unit test and verified working in that the 'props' can pass to Parquet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants