support field collapsing + rescore

[rpedela]

I have some searches that require field collapsing. I need the feature, and I also need the performance improvement over using parent/child or nested. I would like to start using the learning to rank plugin for these searches but rescore support is needed to have good performance, at least for my use case. Any chance of adding this?

[jimczi]

That's possible but that would mean rescoring the collapsed hits per shard and then doing the final collapsing of the rescored hits in the coordinating node. It would not be possible to select the top N uncollapsed and do the collapsing on the rescored docs only. Is it acceptable for your use case @rpedela ?

[rpedela]

The ideal for me would be to collapse first and then rescore the top N collapsed. I don't know if that is equivalent to rescoring the collapsed hits per shard, but it sounds close enough.

[jimczi]

It's not exactly equivalent to a global rescore of the top N collapsed because the rescoring would be per shard first but that's close enough. I'll mark this issue as adoptme because I don't have time to work on it right now. @rpedela would you like to contribute a patch for this ?

[fred84]

I want to take this issue. @jimczi could you please look at following example to verify that I'm correctly understand expected behaviour.

Example

Given we have following documents:

[
  {"name": "elasticsearch", "access": "public", "maintainers": 30 },
  {"name": "logstash"     , "access": "public", "maintainers": 20 },
  {"name": "kibana"       , "access": "public", "maintainers": 10 },
  {"name": "xpack"        , "access": "private", "maintainers": 20 },
  {"name": "beats"        , "access": "private", "maintainers": 5 },
  {"name": "security"     , "access": "private", "maintainers": 2 }
]

Query:

{
    "query": { "range": {"maintainers": {"gt": 3}}},

    "collapse" : {
        "field" : "access",
        "inner_hits": {
            "name": "most_maintainers", 
            "size": 2, 
            "sort": [{ "maintainers": "desc" }] 
        }
    },
    "rescore" : {
        "query" : {
            "rescore_query" : {
                "function_score" : {
                    "script_score": {
                        "script": {
                            "source": "doc.maintainers.value"
                        }
                    }
                }
            }
        }
    }
} 

Expected result:

{
  "hits" : {
    "total" : 5,
    "max_score" : 1.0, // default score
    "hits" : [
      {
        "_index" : "test",
        "_type" : "doc",
        "_id" : "3",
        "_score" : 1.0, // default score
        "_source" : {
          "name" : "kibana",
          "access" : "public",
          "maintainers" : 10
        },
        "fields" : {
          "access" : [
            "public"
          ]
        },
        "inner_hits" : {
          "most_maintainers" : {
            "hits" : {
              "total" : 3,
              "max_score" : 31, 
              "hits" : [
                {
                  "_index" : "test",
                  "_type" : "doc",
                  "_id" : "1",
                  "_score" : 31, // rescoring applied, 1 + 30
                  "_source" : {
                    "name" : "elasticsearch",
                    "access" : "public",
                    "maintainers" : 30
                  },
                  "sort" : [
                    30
                  ]
                },
                {
                  "_index" : "test",
                  "_type" : "doc",
                  "_id" : "2",
                  "_score" : 21, // rescoring applied
                  "_source" : {
                    "name" : "logstash",
                    "access" : "public",
                    "maintainers" : 20
                  },
                  "sort" : [
                    20
                  ]
                }
              ]
            }
          }
        }
      },
      {
        "_index" : "test",
        "_type" : "doc",
        "_id" : "5",
        "_score" : 1.0,
        "_source" : {
          "name" : "beats",
          "access" : "private",
          "maintainers" : 5
        },
        "fields" : {
          "access" : [
            "private"
          ]
        },
        "inner_hits" : {
          "most_maintainers" : {
            "hits" : {
              "total" : 2,
              "max_score" : 21,
              "hits" : [
                {
                  "_index" : "test",
                  "_type" : "doc",
                  "_id" : "4",
                  "_score" : 21,
                  "_source" : {
                    "name" : "xpack",
                    "access" : "private",
                    "maintainers" : 20
                  },
                  "sort" : [
                    20
                  ]
                },
                {
                  "_index" : "test",
                  "_type" : "doc",
                  "_id" : "5",
                  "_score" : 6,
                  "_source" : {
                    "name" : "beats",
                    "access" : "private",
                    "maintainers" : 5
                  },
                  "sort" : [
                    5
                  ]
                }
              ]
            }
          }
        }
      }
    ]
  }
}

[jimczi]

the rescoring should be applied to collapsed hits at the top level, not the inner_hits. Rescoring inner_hits should not be necessary, or at least considered as a different issue. The feature here would be to apply rescoring after collapsing but still on the top level documents. Looking at your example it would first collapse the top documents and then resort them using the rescore function.
This also means that the best documents after rescoring may not be the best document in the group after rescoring since the collapsing is done before the rescoring.

[fred84]

@jimczi, below is updated example. Is it correct now?

Query:

{
    "query": { "range": {"maintainers": {"gt": 3}}},
    "collapse" : {
        "field" : "access"
    },
    "rescore" : {
        "query" : {
            "rescore_query" : {
                "function_score" : {
                    "script_score": {
                        "script": {
                            "source": "doc.maintainers.value"
                        }
                    }
                }
            }
        }
    }
} 

Expected result:

{
  "hits" : {
    "total" : 5,
    "max_score" : 11.0, // smaller then possible best document score (31.0) because rescoring is done after collapsing
    "hits" : [
      {
        "_index" : "test",
        "_type" : "doc",
        "_id" : "3",
        "_score" : 11.0, // rescoring applied (10+1)
        "_source" : {
          "name" : "kibana",
          "access" : "public",
          "maintainers" : 10
        },
        "fields" : {
          "access" : [
            "public"
          ]
        }
      },
      {
        "_index" : "test",
        "_type" : "doc",
        "_id" : "5",
        "_score" : 6.0, // rescoring applied (5+1)
        "_source" : {
          "name" : "beats",
          "access" : "private",
          "maintainers" : 5
        },
        "fields" : {
          "access" : [
            "private"
          ]
        }
      }
    ]
  }
}

[rpedela]

@fred84 Thanks!

@jimczi Are inner_hits rescored for the nested and parent/child queries?

[jimczi]

Are inner_hits rescored for the nested and parent/child queries?

No

@fred84 , yes your example is correct, though the best document per group in your case will depend on the order of the documents in the index since all documents have the same score for the query (the range query uses a constant score).

[jasontedor]

Closed by #28521

[damitkwr]

The fix was reverted. Can we revisit this functionality? This limitation curtails the use of any LTR algorithms

[Hronom]

+1 Very needed

[qi20099]

+1 Very needed

[jimczi]

Reopening after discussing internally. We're actively working on this issue and plan to support this feature in the near future.

[Hronom]

After many years, can't believe about this.

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)