[ENH] Add metadata to block

[HammadB]

Description of changes

Summarize the changes made by this PR.

• Improvements & Bug fixes
  • Introduces the ability to attach metadata to a block
  • We use the RecordBatch ability to store a custom HashMap of metadata in its schema.
  • The delta.finish() method is used to pass the metadata. So metadata is a one-time writeable piece of data.
  • into_arrow returns the arrow fields, not the actual record batch, so that schema can be attached.
• New functionality
  • /

Test plan

How are these changes tested?
Tests for this functionality are in a higher PR in the stack

• Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes

Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs repository?

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[HammadB]

• [ENH] Use sparse index count in blockfile #2939
• [ENH] Add count index #2938
• [ENH] Sparse Index Reader/Writer Split #2918
• [ENH] Add blockfile root #2916
• [ENH] Add metadata to block #2915 👈
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @HammadB and the rest of your teammates on Graphite

[HammadB]

Merge activity

• Oct 16, 9:29 AM EDT: A user started a stack merge that includes this pull request via Graphite.
• Oct 16, 9:30 AM EDT: Graphite rebased this pull request as part of a merge.
• Oct 16, 9:31 AM EDT: A user merged this pull request with Graphite.