Return per-file metadata from readers #10782
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vuule
added
feature request
Codecov Report
@@ Coverage Diff @@
## branch-22.06 #10782 +/- ##
================================================
+ Coverage 86.40% 86.45% +0.04%
================================================
Files 143 143
Lines 22448 22493 +45
================================================
+ Hits 19396 19446 +50
+ Misses 3052 3047 -5
Continue to review full report at Codecov.
|
PointKernel
reviewed
May 4, 2022
Comment on lines
+472
to
+475
| // Assumes that all input files have the same metadata | ||
| // TODO: verify this assumption | ||
| auto it = keyval_maps[0].find("pandas"); | ||
| if (it != keyval_maps[0].end()) { |
There was a problem hiding this comment.
Not sure how it works if the first file is written by pandas (thus has index_columns) while the second one is written by cudf (has no index_columns). To be verified once the python changes have been finished.
PointKernel
approved these changes
May 4, 2022
vuule
added
breaking
karthikeyann
reviewed
May 6, 2022
|
rerun tests |
karthikeyann
approved these changes
May 9, 2022
|
@gpucibot merge |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #10775
C++ side of the fix the the issue above.
Adds
pref_file_user_datatotable_metadataso that readers can return a map per file instead of merging maps from multiple input file into a single map, overwriting elements with the same key.The original
user_datamember now holds the metadata from the first input file, instead of trying (and failing) to merge the maps.Will be removed in the future. Got no good way to deprecate, as the
table_metadatastruct does not have encapsulation :("breaking" label because the logic of
user_datachanged. Not expected to impact and working code.