Unable to load Jaro-Winkler Similarity in Databricks

[thejoemar]

Experiencing the error below when trying to load Jaro-Winkler Similarity in Databricks on a Spark 3.1.2 cluster.

from pyspark.sql import SparkSession, types
from pyspark.context import SparkConf, SparkContext

conf = SparkConf()
conf.set('spark.jars', '/databricks/python/lib/python3.8/site-packages/splink/jars/scala-udf-similarity-0.0.9.jar')
spark.udf.registerJavaFunction('jaro_winkler_sim','uk.gov.moj.dash.linkage.JaroWinklerSimilarity',types.DoubleType())

sc = SparkContext.getOrCreate(conf=conf)
spark_session = SparkSession(sc)

Returns the Error message:
AnalysisException: Can not load class uk.gov.moj.dash.linkage.JaroWinklerSimilarity, please make sure it is on the classpath

[RobinL]

Have you verified that /databricks/python/lib/python3.8/site-packages/splink/jars/scala-udf-similarity-0.0.9.jar exists at that location?

If you move scala-udf-similarity-0.0.9.jar into your home directory, and use that path instead, does it work?

[thejoemar]

Thanks for the response, Robin.

That appeared to be part of the issue. I'll outline my steps to fix below for other Databricks users:

1. Manually install scala-udf-similarity-0.0.9.jar as a library in the relevant cluster.
2. Find the correct path to the jar by running dbutils.fs.ls('/FileStore/jars/'). This will be of the form /FileStore/jars/{random_string_of_characters}-scala_udf_similarity_0_0_9-3e057.jar
3. Modify the suggested code from

from pyspark.sql import SparkSession, types
from pyspark.context import SparkConf, SparkContext
conf.set('spark.driver.extraClassPath', '/databricks/python/lib/python3.8/site-packages/splink/jars/scala-udf-similarity-0.0.9.jar') # Not needed in spark 3
conf.set('spark.jars', '/databricks/python/lib/python3.8/site-packages/splink/jars/scala-udf-similarity-0.0.9.jar')
spark.udf.registerJavaFunction('jaro_winkler_sim','uk.gov.moj.dash.linkage.JaroWinklerSimilarity',types.DoubleType())
sc = SparkContext.getOrCreate(conf=conf)
spark = SparkSession(sc)

to

from pyspark.sql import SparkSession, types
from pyspark.context import SparkConf, SparkContext
conf = SparkConf()
conf.set('spark.driver.extraClassPath', '{path_to_your_jar}')
conf.set('spark.jars', '{path_to_your_jar}')
spark.udf.registerJavaFunction('jaro_winkler','uk.gov.moj.dash.linkage.JaroWinklerSimilarity',types.DoubleType())
sc = SparkContext.getOrCreate(conf=conf)
spark_session = SparkSession(sc)

Robin, could I suggest adding conf = SparkConf() and the change from jaro_winker_sim to jaro_winkler to the recommendation in the package? (I think I can do this myself, but I'm new to GitHub so please forgive me)

[RobinL]

Thanks for the info, that's very useful.

We don't have access to DataBricks, and it's very difficult to figure out a jar config that is agnostic to the env in which it's running. So this feedback is invaluable.

The suggested code actually populates where we 'think' your jar is, and seems to work on most systems. The source code that 'autodetects' the location is here

My hypothesis is that /databricks/python/lib/python3.8/site-packages/splink/jars/scala-udf-similarity-0.0.9.jar does exist on your system, but it is somehow inaccessible (perhaps file access permissions?) by Spark. It'd be a great help if you could just check that for me (i.e. confirm that /databricks/python/lib/python3.8/site-packages/splink/jars/scala-udf-similarity-0.0.9.jar is present).

We can then think about how to accurately reflect this in the warning/suggestion that's printed for the user

[mamonu]

Hi.
Thanks for the databricks-only information. Indeed this is very useful.

Btw on your comment above...Im not sure i agree with the change proposed...

In this line

spark.udf.registerJavaFunction('jaro_winkler_sim','uk.gov.moj.dash.linkage.JaroWinklerSimilarity',
types.DoubleType())

jaro_winkler_sim is needed to be registered like that since that is what is then called by in the spark sql when splink.linker runs.

if its declared as jaro_winkler then it will not be found when splink will be calling it.

of course if you want to use it on other spark things you can call it in anyway you like.
but at some point splink expects to find a jaro_winkler_sim function that is actually uk.gov.moj.dash.linkage.JaroWinklerSimilarity

[RobinL]

Moving this to a discussion so we have a record of how to register these functions in databricks.