-
Notifications
You must be signed in to change notification settings - Fork 915
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add metadata_builder helper class #13232
Add metadata_builder helper class #13232
Conversation
Signed-off-by: Alessandro Bellina <[email protected]>
I have tested with this patch in the past, and I know the cpp tests passed before I posted the PR. I had a linker error with pyarrow when it was trying to dlopen one of its internal libraries (so not even to the tests). As a result I have been recreating the conda environment and still waiting on that. Marking this as reviewable for now as I think getting some 👀 should be good and I don't expect new issues to show up in the python tests. |
Local build finished and |
Thanks @mythrocks, updated. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for the slow review. I'll look through this some more tomorrow.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mostly const additions requested.
Co-authored-by: Mike Wilson <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Thanks for the reviews so far @mythrocks and @hyperbolic2346. Let me know if there's anything else @hyperbolic2346 |
/merge |
This PR introduces a helper class
metadata_builder
within thecudf::detail
namespace, that allowspack
andcontiguous_split
to create metadata for a packed buffer without having access to metadata internals (e.g.serialized_column
).The class makes it possible for callers to build metadata independent of the current implementations in
pack
, that are relying oncolumn_view
s being defined (actual valid cuDF columns). The chunked pack work will use this class to create metadata without needing to create acolumn_view
in the first place, since it doesn't have a valid base pointer for the columns to be instantiated.