Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Meta] ReRankProcessor enhancement: ReRank by Field #926

Closed
brianf-aws opened this issue Oct 7, 2024 · 0 comments · Fixed by #932
Closed

[Meta] ReRankProcessor enhancement: ReRank by Field #926

brianf-aws opened this issue Oct 7, 2024 · 0 comments · Fixed by #932
Assignees

Comments

@brianf-aws
Copy link
Contributor

brianf-aws commented Oct 7, 2024

Is your feature request related to a problem?

Currently if a user has access to a field in their document or in a search response we would like to be able to re-rank by that metric. Currently the ReRank process supports ML_OpenSearch as a way to re rank. We would like to provide users with a way to perform a 2nd level re ranking, We would like to make it a ByFieldRerankProcessor

What solution would you like?

The rescoring logic (i.e update the _score field to reflect a new score) is already provided in the RescoringReRankProcessor. All that would be required is to implement ByFieldRerankProcessor to use the scores provided by the document or a previous search response.

Ideally the interface would look like this (My implementation is in the Neural Search repo)

{
    "response_processors": [
        {
            "rerank": {
                "by_field": {
                    "target_field": "ml_score",
                     "remove_target_field": true, ## Default false
                     "keep_previous_score" : true ## Default false
                }
            }
        }
    ]
}

It was discussed that previous scores should be kept as an option as this may hinder the expectation of the user, thus we added a field called keep_previous_score to allow this. We also want to give an option to be able delete the field (via remove_target_field) provided to perform the re ranking as this is redundant data.

What alternatives have you considered?

Creating a separate response processor, in open search core, that replaced and sorted the response. This was initially proposed in OpenSearch Core opensearch-project/OpenSearch#15631. But after offline discussion we decided that this functionality could be transferred to a processor that already does re ranking, in the Neural Search repo.

Do you have any additional context?

This functionality was brought up as a necessity to enhance the ML Inference Processor in the ML-Commons codebase

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: New
Development

Successfully merging a pull request may close this issue.

2 participants