Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA]Support function sha1 #7035

Open
viadea opened this issue Nov 9, 2022 · 4 comments
Open

[FEA]Support function sha1 #7035

viadea opened this issue Nov 9, 2022 · 4 comments
Labels
cudf_dependency An issue or PR with this label depends on a new feature in cudf feature request New feature or request

Comments

@viadea
Copy link
Collaborator

viadea commented Nov 9, 2022

I wish we can support function sha1.

eg:

spark-sql> select sha1(c_customer_id) from customer limit 10;

      ! <Sha1> sha1(cast(c_customer_id#25 as binary)) cannot run on GPU because GPU does not currently support the operator class org.apache.spark.sql.catalyst.expressions.Sha1
@viadea viadea added feature request New feature or request ? - Needs Triage Need team to review and classify labels Nov 9, 2022
@sameerz sameerz added cudf_dependency An issue or PR with this label depends on a new feature in cudf and removed ? - Needs Triage Need team to review and classify labels Nov 9, 2022
@sameerz
Copy link
Collaborator

sameerz commented Nov 9, 2022

Related cudf issue rapidsai/cudf#8641

@revans2
Copy link
Collaborator

revans2 commented Nov 9, 2022

We need to be a bit careful here to be sure that our requirements match that of pandas/python. In Spark the output of all sha hashes is a string that is a lowercase HEX encoded representation of the binary hash result. This corresponds to the hexdigest() method. Also rapidsai/cudf#8641 calls out sha256 and sha512, not sha1. Spark supports sha1 and sha2. sha2 supports bit lengths of 224, 256, 384, and 512. sha2 matches sha256 and sha512.

@sameerz
Copy link
Collaborator

sameerz commented Nov 10, 2022

The PR rapidsai/cudf#9215 attached to rapidsai/cudf#8641 appears to be adding support for SHA-1.

@sameerz
Copy link
Collaborator

sameerz commented Jan 12, 2024

Depends on rapidsai/cudf#14391

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cudf_dependency An issue or PR with this label depends on a new feature in cudf feature request New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants