-
Notifications
You must be signed in to change notification settings - Fork 153
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add sortColumnsBy DataFraneExt #162
Conversation
src/main/scala/com/github/mrpowers/spark/daria/sql/DataFrameExt.scala
Outdated
Show resolved
Hide resolved
* // +---+----+------+ | ||
* }}} | ||
*/ | ||
def sortColumnsBy[A](f: StructField => A)(implicit ord: Ordering[A]): DataFrame = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can only see to use it with alphabetical order, if you want to allow it to use it with datatypes we should provide an instance of Ordering[DataType]
or limit it only to order by string value.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure. I think another use case would be to perform specific ordering based on some field metadata but we can allow the user the flexibility to provide their own
src/main/scala/com/github/mrpowers/spark/daria/sql/types/StructTypeHelpers.scala
Outdated
Show resolved
Hide resolved
src/test/scala/com/github/mrpowers/spark/daria/sql/DataFrameExtTest.scala
Outdated
Show resolved
Hide resolved
Let me enable CI on pr first |
src/test/scala/com/github/mrpowers/spark/daria/sql/DataFrameExtTest.scala
Outdated
Show resolved
Hide resolved
src/main/scala/com/github/mrpowers/spark/daria/sql/types/StructTypeHelpers.scala
Outdated
Show resolved
Hide resolved
62fb840
to
ffcc14a
Compare
This was inspired by https://stackoverflow.com/questions/57821538/how-to-sort-columns-of-nested-structs-alphabetically-in-pyspark#:~:text=You%20can%20first%20flatten%20your%20DF%20with%20the%20nice%20colname.*
Here we support more flexible sorting by allowing user to provide a custom transformation and implicit ordering.