feat: add more Spark Expressions #1724
Conversation
| def _all(_input: Column) -> Column: | ||
| from pyspark.sql import functions as F # noqa: N812 | ||
|
|
||
| return F.bool_and(_input) |
There was a problem hiding this comment.
bool_and and bool_or are only available from pyspark 3.5, while narwhals minimum dependency is 3.3
Is there a way to do write these for older versions of spark?
There was a problem hiding this comment.
this would be new functionality, maybe we could bump the minimum pyspark version?
There was a problem hiding this comment.
not opposed ^ pyspark 3.5 introduced a whole batch of functions that would let us work on other expressions like the ones mentioned in #1714
There was a problem hiding this comment.
This seems the easier way forward. Alternatively, raise for those methods and backend_version pre 3.5. E.g.
if self._backend_version < (3, 5):
msg = ...
raise NotImplementedError(msg)
...There was a problem hiding this comment.
It'd definetly make it easier for us :) Happy to make a PR for increasing the version later this afternoon
My only worry is that it may hurt adoption a bit. For what I have seen, updating to the latest pyspark is not always easy in enterprise (may be dependent on vendors,may require update of the cluster, etc.). But we can worry about loosening the dependency after we implement all the methods 👌
There was a problem hiding this comment.
thanks for the perspective - sure, if it's not too painful to support 3.3+, i'd say that that's ok
There was a problem hiding this comment.
moving the discussion to #1744
|
Hey @EdAbati , Took an initial swing at implementing the |
What type of PR is this? (check all applicable)
Related issues
Checklist
If you have comments or can explain your changes, please do so below
Added the following methods to
SparkLikeExprandSparkLikeNamespace:anyallnull_countany_horizontalCopied respective tests over - couldn't run them without Java on my machine but running them locally on their respective test datasets worked for me.
Let me know if anything needs to be updated!