-
Notifications
You must be signed in to change notification settings - Fork 197
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add CAGRA-Q build (compression) #2213
Add CAGRA-Q build (compression) #2213
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @achirkin for this PR! Here is a first batch of my comments.
Co-authored-by: Tamas Bela Feher <[email protected]>
…set from making a view
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Many thanks Artem for updating the PR, and also adding serialization methods. Overall it looks good, but I still have an issue with the index types.
…re explicit about arguments
…ccessible by the current device and document the api
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks Artem for integrating data compression into RAFT CAGRA! The PR looks good to me, just have two minor questions below.
Ideally we should go ahead and merge this, so that the follow up PR (#2206) is easier to review.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks Artem, the PR looks good to me!
Latest update:
|
/merge |
Add a
cagra::compress
function that implements CAGRA-Q (VQ + PQ) compression of a given dataset.The result,
compressed_dataset
, is supposed to complement the CAGRA graph duringcagra::search
in place of a raw dataset.Current state:
pq_bits
support ([4,5,6,7,8] - same as in IVF-PQ)pq_dim
values are accepted, but the dataset is not padded and thusdim
must be a multiple ofpq_dim
.half
to match the prototype implementation for now. This could be a runtime (build) parameter as well.uint8_t
,int8_t
,half
, andfloat
compile), but I tested onlyfloat
.