Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make LERF compatible with DINOv2 #9

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Make LERF compatible with DINOv2 #9

wants to merge 3 commits into from

Conversation

chungmin99
Copy link
Collaborator

Added capability to use DINOv2 models

(Using implementation/torchhub as described by https://github.com/facebookresearch/dinov2)

TODOs:

  • Compare/contrast DINO vs. DINOv2 features
  • Add DINO feature visualization(?) -- for the compare/contrast. Initial implementation cause the colors to be inconsistent across renders, so it feels confusing/disorienting.
  • Add compatibility for ViT-B/L/G DINO versions (currently dino-field embedding out hardcoded to 384)

@chungmin99 chungmin99 requested a review from kerrj April 20, 2023 23:18
@linukc
Copy link

linukc commented Jun 5, 2023

i suggest to move KMeans and torch.pca_low_rank to the get_outputs_for_camera_ray_bundle function (aggregate all get_outputs from one image there and after perform dim reduction) to prevent different color palitre on a one image

@anas-zafar
Copy link

@chungmin99 , were you able to fully integrate Dinov2 with LERF?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants