Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds documentation about byField rerank processor #8593

Merged
merged 10 commits into from
Oct 23, 2024

Conversation

brianf-aws
Copy link
Contributor

Description

Adds information about the byField rerank processor. which applies a second level rerank to a search response based on a provided user field

Issues Resolved

N/A

Version

2.18

Frontend features

N/A

Related PRs

opensearch-project/neural-search#932

Checklist

  • By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
    For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Copy link

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review.

@kolchfa-aws kolchfa-aws added v2.18.0 release-notes PR: Include this PR in the automated release notes 3 - Tech review PR: Tech review in progress labels Oct 22, 2024
Copy link
Contributor

@mingshl mingshl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was saying rerank processor is a request processor. that is something we might want to backport and fix it

Copy link
Collaborator

@natebower natebower left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kolchfa-aws @brianf-aws Please see my comments and changes and let me know if you have any questions. Thanks!

_search-plugins/search-pipelines/rerank-processor.md Outdated Show resolved Hide resolved
_search-plugins/search-pipelines/rerank-processor.md Outdated Show resolved Hide resolved
_search-plugins/search-pipelines/rerank-processor.md Outdated Show resolved Hide resolved
_search-plugins/search-pipelines/rerank-processor.md Outdated Show resolved Hide resolved
_search-plugins/search-pipelines/rerank-processor.md Outdated Show resolved Hide resolved
_search-plugins/search-pipelines/rerank-processor.md Outdated Show resolved Hide resolved
_search-plugins/search-pipelines/rerank-processor.md Outdated Show resolved Hide resolved
_search-plugins/search-pipelines/rerank-processor.md Outdated Show resolved Hide resolved
{% include copy-curl.html %}

Which now yields the following. Observe that our documents are now sorted in a descending order based on the rating of each book, we kept the `previous_score` to help keep track of how we originally scored before reranking.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kolchfa-aws I had to make a couple of assumptions in my rewrites here. Please ensure that they retain technical accuracy.

kolchfa-aws and others added 2 commits October 23, 2024 09:01
Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: kolchfa-aws <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
Copy link
Collaborator

@natebower natebower left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kolchfa-aws Please see my comments and changes and let me know if you have any questions. Thanks!

_search-plugins/search-pipelines/rerank-processor.md Outdated Show resolved Hide resolved
_search-plugins/search-pipelines/rerank-processor.md Outdated Show resolved Hide resolved
_search-plugins/search-relevance/rerank-by-field.md Outdated Show resolved Hide resolved
_search-plugins/search-relevance/rerank-by-field.md Outdated Show resolved Hide resolved
_search-plugins/search-relevance/rerank-cross-encoder.md Outdated Show resolved Hide resolved
_search-plugins/search-relevance/rerank-cross-encoder.md Outdated Show resolved Hide resolved
Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: kolchfa-aws <[email protected]>
Configure a search pipeline with a [`rerank` processor]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/rerank-processor/) and specify the `by_field` rerank type. The pipeline sorts by the `reviews.stars` field (specified by a complete dot path to the field) and returns the original query scores for all documents along with their new scores:

```json
PUT /_search/pipeline/rerank_byfield_pipeline
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@brianf-aws Please verify this example. Thanks!

"genre": {
"type": "keyword"
},
"reviews": {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@brianf-aws: Does the reviews field need to be of an object type?

Copy link
Contributor Author

@brianf-aws brianf-aws Oct 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if reviews points to a numerical score it can be used for example

{
   reviews: 5
}

then the targetField would be just reviews. The target field depends on the knowledge of the user really and how well they know their documents

@kolchfa-aws kolchfa-aws added 6 - Done but waiting to merge PR: The work is done and ready to merge and removed 3 - Tech review PR: Tech review in progress labels Oct 23, 2024
If you use postman or dev tools it wont work since there are qoutes in the index this had to be changed. Also it had to be made clear where the search pipeline would be applied in doing a search

Signed-off-by: Brian Flores <[email protected]>
Copy link
Collaborator

@kolchfa-aws kolchfa-aws left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kolchfa-aws kolchfa-aws merged commit f43dcfa into opensearch-project:main Oct 23, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
6 - Done but waiting to merge PR: The work is done and ready to merge release-notes PR: Include this PR in the automated release notes v2.18.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants