Splink match 3 times slower with 1.0 compare to 0.4.2

[Khallil]

Hi team,

Again thanks for this magnificent tool that you developed.

I updated Splink to 1.0, was 0.4.2 before, and now the Splink match takes 3 times longer.

I just changed the required typo which are,

• changing the parameter df = df, to df_or_dfs=[]
• adding a source_column lit with the source name

The settings and the data is the same.
I have no idea why that happen and I don't know where to look for to tackle the issue.

Thanks again for your time !

[RobinL]

This is certainly a suprise. What type of job is it, dedupe only, link and dedupe or link only?

[Khallil]

Thank you for your answer.

It's a link only type of job.
To add a bit more on the context, I tried with 100k rows sample, with 0.4.2 it took 10min, 30min with 1.0. And I ran the jobs multiple times to make sure.
Again same settings, same data. I just do a :

df = Splink(...)
df.get_scored_comparaisons()

In the cell that I measured runtime on.
Let me know if you need additional information, I can reproduce anytime.

Thanks again !

[RobinL]

Thanks for the feedback. - link_only is a setting we rarely use, so it's very useful that you've spotted this.

I've just had a look at the code and I think I can see why this might be the case. Before, after

I'm going to convert this into an issue and we can think about whether we can revert back to the previous approach. Unfortunately it's not trivial because in 1.0, link_only supports >2 datasets, and the performance gain I think happens specifically for the two dataset case (which I admit is most common).

Issue here

[Khallil]

Great, I will keep an eye on it !
Thanks a lot

[RobinL]

For what it's worth, this now fixed - upgrade to the latest version of Splink for the faster code