Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix scann range search #315

Merged
merged 1 commit into from
Dec 25, 2023

Conversation

chasingegg
Copy link
Collaborator

@chasingegg chasingegg commented Dec 25, 2023

issue #171
Faiss is highly optimized for batch searching, and different queries will search the same ivf bucket by gathering different queries' nprobe list which could reorder the npobe sequence, this could cause bug since range search goes as a 'early termination' manner, it will go through all ivf buckets and early terminate when no results found in one bucket, we have to search buckets by its sorted distance to query. Also, knowhere will split nq to single query, no need to gather queries' nprobe list.

before this pr

SCANN {"metric_type": "L2", "radius": 65.0} 0.0088
SCANN {"metric_type": "IP", "radius": 8.699999809265137} 0.003
SCANN {"metric_type": "COSINE", "radius": 0.20000000298023224} 0.0018

range search recall for IVF_FLAT with same param:

IVF_FLAT {"metric_type": "L2", "radius": 65.0} 0.1613
IVF_FLAT {"metric_type": "IP", "radius": 8.699999809265137} 0.1643
IVF_FLAT {"metric_type": "COSINE", "radius": 0.20000000298023224} 0.1675

after this pr

SCANN {"metric_type": "L2", "radius": 65.0} 0.2464
SCANN {"metric_type": "IP", "radius": 8.699999809265137} 0.0886
SCANN {"metric_type": "COSINE", "radius": 0.20000000298023224} 0.1035

@sre-ci-robot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chasingegg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copy link

mergify bot commented Dec 25, 2023

@chasingegg 🔍 Important: PR Classification Needed!

For efficient project management and a seamless review process, it's essential to classify your PR correctly. Here's how:

  1. If you're fixing a bug, label it as kind/bug.
  2. For small tweaks (less than 20 lines without altering any functionality), please use kind/improvement.
  3. Significant changes that don't modify existing functionalities should be tagged as kind/enhancement.
  4. Adjusting APIs or changing functionality? Go with kind/feature.

For any PR outside the kind/improvement category, ensure you link to the associated issue using the format: “issue: #”.

Thanks for your efforts and contribution to the community!.

@chasingegg
Copy link
Collaborator Author

/kind bug

@sre-ci-robot sre-ci-robot added the kind/bug This PR is a bug fix label Dec 25, 2023
@chasingegg chasingegg force-pushed the fix-scann-range-search branch from 6eaef11 to 086f892 Compare December 25, 2023 03:11
@chasingegg
Copy link
Collaborator Author

/run-e2e

1 similar comment
@chasingegg
Copy link
Collaborator Author

/run-e2e

Signed-off-by: chasingegg <[email protected]>
@chasingegg
Copy link
Collaborator Author

/run-e2e

@mergify mergify bot added the ci-passed label Dec 25, 2023
@liliu-z
Copy link
Collaborator

liliu-z commented Dec 25, 2023

/lgtm

@sre-ci-robot sre-ci-robot merged commit aef7a3c into zilliztech:main Dec 25, 2023
9 checks passed
@chasingegg chasingegg deleted the fix-scann-range-search branch December 25, 2023 14:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants