Generic serialization of all column types

[wence-]

Prior to this change, not all column types were serializable, with
serialization implemented in an ad-hoc way for each type in turn. The
inheritance is such that we can just implement generic serialization on the
base column class (no subclass has a specialized constructor), so do
that (closes #10766).

To support this implement serialization for all dtypes. These must
individually implement the Serializable interface since in contrast to
columns every dtype has its own constructor (closes #10785).

[codecov]

Codecov Report

Merging #10784 (2171cd7) into branch-22.06 (8d861ce) will decrease coverage by 0.01%.
The diff coverage is 98.40%.

❗ Current head 2171cd7 differs from pull request most recent head 94bc231. Consider uploading reports for the commit 94bc231 to get more accurate results

@@               Coverage Diff                @@
##           branch-22.06   #10784      +/-   ##
================================================
- Coverage         86.40%   86.38%   -0.02%     
================================================
  Files               143      143              
  Lines             22448    22442       -6     
================================================
- Hits              19396    19387       -9     
- Misses             3052     3055       +3     

Impacted Files	Coverage Δ
python/cudf/cudf/core/column/lists.py	90.68% <ø> (-1.40%)	⬇️
python/cudf/cudf/core/column/string.py	88.66% <ø> (-0.44%)	⬇️
python/cudf/cudf/core/indexed_frame.py	91.70% <ø> (ø)
python/cudf/cudf/core/dataframe.py	93.73% <96.29%> (+0.03%)	⬆️
python/cudf/cudf/core/buffer.py	88.52% <100.00%> (+0.19%)	⬆️
python/cudf/cudf/core/column/categorical.py	89.37% <100.00%> (-0.61%)	⬇️
python/cudf/cudf/core/column/column.py	89.76% <100.00%> (+0.32%)	⬆️
python/cudf/cudf/core/column/decimal.py	90.60% <100.00%> (-0.50%)	⬇️
python/cudf/cudf/core/dtypes.py	97.19% <100.00%> (-0.11%)	⬇️
python/cudf/cudf/testing/_utils.py	94.05% <100.00%> (+0.06%)	⬆️
... and 1 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8d861ce...94bc231. Read the comment docs.

[vyasr]

@wence- thanks for this! I didn't get around to taking a look today, but I should be able to tomorrow.

[wence-]

I think this is now ready for review. Some points to note:

1. The applied "non-breaking" label is incorrect; this changes the header format for serialized columns which means that on-disk pickle files are not loadable across a merge of this branch. I can institute backwards-compat if that is required: what's the policy on how best to do this? If more generally serialization is seen as a public API that needs backwards compatibility it is probably worthwhile versioning the metadata so that it's easy to detect moving through breaking change boundaries.
2. frame_count is now used everywhere in the header metadata for dtypes (needed for this PR) and columns that are serialized; but not yet in other objects (like dataframes etc...). I think it makes sense to push changes there to a subsequent PR.

[madsbk]

rerun tests

[shwina]

This looks great to me. Thank you @wence-! I'd let @vyasr take a look before merging.

[madsbk]

Great work @wence-. My only suggestion is that we rename frame_count to frame-count.

[wence-]

Great work @wence-. My only suggestion is that we rename frame_count to frame-count.

I'd prefer to do this in a separate round that just does renaming (since it would pervade further than just these changes).

[wence-]

I'm just going to flag again (since I do not have permissions to change the labels), that this is a breaking change if the serialize interface is considered to be a stable mechanism for storage across releases. As it stands, someone pickling a dataframe in 22.04 and attempting to load it in 22.06 (assuming this PR makes it in) will not be able to in most cases.

I can add backwards compat and deprecation warnings, but would like to know if that's necessary first...

[vyasr]

Some very minor comments here, but everything LGTM and the only open question the backwards compatibility layer. The last couple of times that I changed serialization I introduced a backwards compatibility layer that was removed in the next release. @galipremsagar do you think it was valuable? It shouldn't matter for Dask purposes since this would only help data that is serialized and stored over longer periods of time. Do you think that's worth doing again? My 2c: it's a nice-to-have but it's not critical to have.

[galipremsagar]

Some very minor comments here, but everything LGTM and the only open question the backwards compatibility layer. The last couple of times that I changed serialization I introduced a backwards compatibility layer that was removed in the next release. @galipremsagar do you think it was valuable? It shouldn't matter for Dask purposes since this would only help data that is serialized and stored over longer periods of time. Do you think that's worth doing again? My 2c: it's a nice-to-have but it's not critical to have.

I think this one is okay to be a breaking change. Pickling & un-pickling is the only thing that will probably break and that is not guaranteed across versions in general, though it's the opposite for IO readers and writers. So okay to go ahead and not have a backward compatible change here.

[vyasr]

LGTM

[vyasr]

@wence- in addition to my suggestions for the assertions, my only other request is that you update the PR description. That description goes into the commit message, and since you are not planning to address the frame-count question in this PR it would be better to remove that checklist before we merge.

[wence-]

@wence- in addition to my suggestions for the assertions, my only other request is that you update the PR description. That description goes into the commit message, and since you are not planning to address the frame-count question in this PR it would be better to remove that checklist before we merge.

Thanks, done.

[vyasr]

rerun tests

[madsbk]

@gpucibot merge