Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add sortColumnsBy DataFraneExt #162

Merged
merged 14 commits into from
Oct 8, 2024
Merged

Conversation

zeotuan
Copy link
Collaborator

@zeotuan zeotuan commented Sep 29, 2024

This was inspired by https://stackoverflow.com/questions/57821538/how-to-sort-columns-of-nested-structs-alphabetically-in-pyspark#:~:text=You%20can%20first%20flatten%20your%20DF%20with%20the%20nice%20colname.*

Here we support more flexible sorting by allowing user to provide a custom transformation and implicit ordering.

  • add support to sorting of ArrayType(StructType) column which was not available in original implementation

@zeotuan zeotuan changed the title selectSortedExpr Add sortColumnsBy DataFraneExt Sep 30, 2024
@zeotuan zeotuan marked this pull request as ready for review September 30, 2024 03:49
* // +---+----+------+
* }}}
*/
def sortColumnsBy[A](f: StructField => A)(implicit ord: Ordering[A]): DataFrame =
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can only see to use it with alphabetical order, if you want to allow it to use it with datatypes we should provide an instance of Ordering[DataType] or limit it only to order by string value.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. I think another use case would be to perform specific ordering based on some field metadata but we can allow the user the flexibility to provide their own

@zeotuan zeotuan self-assigned this Sep 30, 2024
@alfonsorr
Copy link
Collaborator

Let me enable CI on pr first

@zeotuan zeotuan merged commit 617e062 into mrpowers-io:main Oct 8, 2024
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants