CTE definition can't have duplicate names

[alexdaustin]

Hi. Just getting started with Splink. Looks to be a great tool. Any ideas what I've done wrong to get the below error message? I'm posting the settings used. Thank you!

[RobinL]

Thanks for reporting this. We need to look into it in more detail. For now, restarting the kernel and rerunning might work (let me know if it doesn't).

Something else that would probably work would be turning term frequency adjustments on for at least one column

Finally, just to note, that blocking rule will result in every input row being compared to every other. That's fine for small input data, but will run extremely slowly for larger input datasets. It's equality conditions which reduce the number of comparisons that get generated. (This is unrelated to the error)

[alexdaustin]

Thanks for getting back to me. Unfortunately, that didn't work. Though strictly speaking I got a different error. Online I've seen output like this associated with being OOM. But I've deliberately kept things small. The df itself has been limited to 100 rows, and I'm targeting 2500 rows to estimate the u values. This is being done on a k8s cluster with two cpu & 48 gb of memory. I would think it was more than enough.

[RobinL]

That one's simple -you need to set a checkpoint directory in your Spark context because of the way Splink caches some intermediate outputs. In this instance it's not related to out of memory issues

[RobinL]

Re: the original error, please could you post the line of code where you're creating the linker object using SparkLinker(), might help us get to the bottom of it

[alexdaustin]

After establishing a checkpoint directory I was able to successfully run linker.estimate_u_using_random_sampling(target_rows=2500)
The linker object was created using
linker = SparkLinker(spark_samp, settings)
where spark_samp is made using spark.createDataFrame(a pandas df)
The reason I am using a Jaro-Winkler comparison as a blocking rule is that my data is comprised of fullname and dob only. I believe equality-type rules will be too restrictive. If anyone has any best-practice suggestions for data of this nature, I'd be glad to hear them!
Thanks for all your help!