target_embedding without training

[bkj]

Hello --

Really enjoyed the paper. One clarifying question: you add the target_embedding to the query point embedding here:
https://github.com/ZrrSkywalker/Personalize-SAM/blob/main/per_segment_anything/modeling/transformer.py#L94

but you don't fine-tune the model. Do you have an intuition for why that works? Is it basically that the TwoWayAttentionBlock is now computing attention based on the "average" of the similarity between points <-> image embeddings and target_embedding <-> image embeddings?

Thanks!