Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] CAGRA avoid unnecessary device memory copies of dataset / knn graph #1479

Closed
Tracked by #1392
tfeher opened this issue May 1, 2023 · 0 comments · Fixed by #1494
Closed
Tracked by #1392

[FEA] CAGRA avoid unnecessary device memory copies of dataset / knn graph #1479

tfeher opened this issue May 1, 2023 · 0 comments · Fixed by #1494
Labels
feature request New feature or request

Comments

@tfeher
Copy link
Contributor

tfeher commented May 1, 2023

Is your feature request related to a problem? Please describe.

When data / graph is in device memory, consider just storing a reference in the index for the data. See discussion #1375 (comment)

Tracker #1392
@tfeher tfeher added the feature request New feature or request label May 1, 2023
This was referenced May 1, 2023
Open
Open
Merged
@cjnolet cjnolet mentioned this issue Jun 22, 2023
Closed
9 tasks
rapids-bot bot pushed a commit that referenced this issue Jul 20, 2023
@tfeher
Cagra index construction without copying device mdarrays (#1494)
db4797f 
This PR aims to improve the workflow when dealing with large datasets. When experimenting with different versions of the knn-graph, we might want to construct indices with the same dataset (see #1435 for further discussion).

If the dataset is already in device memory (and rows are properly aligned / padded), then we only store a reference to the dataset, therefore multiple indices can refer to the same dataset. Similarly, when `knn_graph` is a device array, then store only a reference. 

Additionally, this PR adds `update_dataset` and `update_graph` methods to the index.
Closes #1479

Authors:
  - Tamas Bela Feher (https://github.com/tfeher)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #1494
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature request New feature or request
Projects
VS/ML/DM Primitives Release Board
Status: No status
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Cagra index construction without copying device mdarrays tfeher/raft
1 participant
@tfeher