Exposing lucene 6.x minhash filter.

[a2lin]

I've tried to expose the 6.x minhash filter, and wrote some documentation from my (very inexpert) understanding on how this works.

I was mostly playing around with it using the following scenario that I cribbed off of the lucene ticket.

result (hits)

{
  "hits":[
    {
      "_index":"lsh_test",
      "_type":"doc_type",
      "_id":"2",
      "_score":1.4709876,
      "_source":{
        "doc_lsh":"elasticsearch is also an open source enterprise search engine based on Lucene"
      }
    },
    {
      "_index":"lsh_test",
      "_type":"doc_type",
      "_id":"1",
      "_score":0.61897373,
      "_source":{
        "doc_lsh":"elasticsearch is also an open source search engine based on Lucene"
      }
    },
    {
      "_index":"lsh_test",
      "_type":"doc_type",
      "_id":"3",
      "_score":0.4758651,
      "_source":{
        "doc_lsh":"elasticsearch is also a popular open source enterprise search engine based on Lucene"
      }
    }
  ]
}

query:

{
  "match": "elasticsearch is also an open source search engine"
}

(bulk) indexed documents:

{ "index" : { "_index" : "lsh_test", "_type" : "doc_type", "_id" : "1" }}
{"doc_lsh": "elasticsearch is also an open source search engine based on Lucene"}
{ "index" : { "_index" : "lsh_test", "_type" : "doc_type", "_id" : "2" }}
{"doc_lsh": "elasticsearch is also an open source enterprise search engine based on Lucene"}
{ "index" : { "_index" : "lsh_test", "_type" : "doc_type", "_id" : "3" }}
{"doc_lsh": "elasticsearch is also a popular open source enterprise search engine based on Lucene"}
{ "index" : { "_index" : "lsh_test", "_type" : "doc_type", "_id" : "4" }}
{"doc_lsh": "Apache Lucene is a high-performance, full-featured text search engine library written entirely in Java"}

mapping:

{
  "settings":{
    "analysis":{
      "filter":{
        "five_shingle":{
          "type":"shingle",
          "max_shingle_size":5,
          "min_shingle_size":3,
          "output_unigrams":false
        },
        "minhash":{
          "type":"min_hash",
          "with_rotation":"false"
        }
      },
      "analyzer":{
        "lsh_analyzer":{
          "type":"custom",
          "tokenizer":"whitespace",
          "filter":[
            "five_shingle",
            "minhash"
          ]
        }
      }
    }
  },
  "mappings":{
    "doc_type":{
      "properties":{
        "doc_lsh":{
          "type":"text",
          "analyzer":"lsh_analyzer",
          "store":"true"
        }
      }
    }
  }
}

Closes #20149.

[jpountz]

s/minhash/min_hash/

[clintongormley]

Hi @a2lin

Thanks for the PR. This needs documentation added too. Btw, why are you using shingles of 3..5 words? This will create many more tokens than you really need and isn't considered good practice. Instead I'd just use shingles of length 2.

[a2lin]

@clintongormley Thanks for the advice. I poked around with 5 because the function comment for MinHashFilter suggests that the expected incoming tokens are 5 word shingles.

Can you link me to an example of the documentation that this feature needs added? I could only find the shingled version of:

docs/reference/analysis/tokenfilters/minhash-tokenfilter.asciidoc

when I searched the code for the shingleTokenFilterFactory analogue.

[clintongormley]

@a2lin something like this https://www.elastic.co/guide/en/elasticsearch/reference/master/analysis-fingerprint-tokenfilter.html

[a2lin]

@clintongormley Oops, I thought that was generated from the file that @jpountz commented on. I'll look again.

[clintongormley]

Oh sorry @a2lin - I completely missed the asciidoc!

[s1monw]

ok to test

[s1monw]

@a2lin thanks for fixing this!!!!!

[a2lin]

@s1monw thanks for merging!