You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
It looks like postgresql and spark have a different, (probably much faster) implementation of calculating the string length, which doesn't depend on calculating the grapheme clusters, but on utf8 code points.
As an example: select length('ä')
PostgreSQL | Spark | DataFusion
Postgres
Spark
DataFusion
2
2
1
Describe the solution you'd like
Probably .chars().count() is a faster solution and matches the implementation of PostgreSQL (and Spark).
It can be applied to other string functions as well.
Describe alternatives you've considered
Accepting our own implementation being slower but "superior" to other solutions.
This is something that we could potentially add a configuration option for as well, defaulting to the fast version but allowing the user to select a slower and more accurate version.
This is something that we could potentially add a configuration option for as well, defaulting to the fast version but allowing the user to select a slower and more accurate version.
In general, I think it would be better to choose one version of the algorithm rather than have the result depend on the configuration.
As we aim to be mostly compatible with PostgreSQL, it makes sense to me to choose counting the code points instead of grapheme clusters.
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
It looks like postgresql and spark have a different, (probably much faster) implementation of calculating the string length, which doesn't depend on calculating the grapheme clusters, but on utf8 code points.
As an example:
select length('ä')
PostgreSQL | Spark | DataFusion
Describe the solution you'd like
Probably
.chars().count()
is a faster solution and matches the implementation of PostgreSQL (and Spark).It can be applied to other string functions as well.
Describe alternatives you've considered
Accepting our own implementation being slower but "superior" to other solutions.
Additional context
This came up as being quite slow when profiling this benchmark: https://github.com/DataPsycho/data-pipelines-in-rust/tree/main/amazon_review_pipeline
We're using the
grapheme
option in other places as well, maybe here we could use the fasterThe text was updated successfully, but these errors were encountered: