MinHashing is a very efficient way of finding similar records in a dataset based on Jaccard similarity. PyMinHash implements efficient minhashing for Pandas dataframes. See instructions below or look at the example notebook to get started.
Developed by Frits Hermans
Documentation can be found here
Using PyPI
pip install pyminhash
Using conda
conda install -c conda-forge pyminhash
Clone this Github repo and install in editable mode:
python -m pip install -e ".[dev]"
python setup.py develop
Apply record matching to column name of your Pandas dataframe df as follows:
myHasher = MinHash(n_hash_tables=10)
myHasher.fit_predict(df, 'name')This will return the row pairs from df that have non-zero Jaccard similarity.