Move to keyset pagination

[d33bs]

Description

This PR modifies "chunk" behavior to follow keyset pagination patterns. We gather sets of pages as pagesets which are based on a key per table which is used to help create pages of rows from tables. pagesets are gathered ahead of time to retain the ability to parallelize where appropriate. This change required a new configuration parameter to be specified per table and the overall join.

Operationally this reduces the amount of data stored in tables throughout, reduces the complexity of the sorting performed, and avoids unnecessary data loading which occurs during the use of OFFSET.

Changes here made it unnecessary to create metadata columns and as a result I've removed those from all spots I could find.

Changes here are intended to address #214 , which turned out to be more related to data size than the sorting implementation. This is confirmed by a now passing large_data_tests test of JUMP data using the full dataset (instead of a truncated version).

Closes #214

What is the nature of your change?

• Bug fix (fixes an issue).
• Enhancement (adds functionality).
• Breaking change (fix or feature that would cause existing functionality to not work as expected).
• This change requires a documentation update.

Checklist

Please ensure that all boxes are checked before indicating that a pull request is ready for review.

• I have read the CONTRIBUTING.md guidelines.
• My code follows the style guidelines of this project.
• I have performed a self-review of my own code.
• I have commented my code, particularly in hard-to-understand areas.
• I have made corresponding changes to the documentation.
• My changes generate no new warnings.
• New and existing unit tests pass locally with my changes.
• I have added tests that prove my fix is effective or that my feature works.
• I have deleted all non-relevant text in this pull request template.

[falquaddoomi]

Neat solution to the problem you were facing; I agree that it's less brittle than using LIMIT and OFFSET since you're now relying on the data rather than implementation-specific definitions of LIMIT and OFFSET.

I think you could add a bit more description of what columns are good candidates for keys; from my reading of your code I assume they'd be unique integers.

Anyway, nice work, and glad to see this issue put to rest!

[d33bs]

Thanks @falquaddoomi for the review and comments! I made some adjustments based on what you mentioned.

[jenna-tomkinson]

I left a few comments, mainly clarification about the fix. Nice job! 🎉

[gwaybio]

Exciting to see this enhancement! I made several comments, mostly aimed at improving clarity in comments and variable names. After addressing, feel free to merge 👍

[d33bs]

Thank you @gwaybio, @jenna-tomkinson and @falquaddoomi for your reviews! After making adjustments for all comments I believe this is now ready and I'll merge this in.