Similarity UDF #535
Similarity UDF #535
Conversation
| row.iloc[1], | ||
| ) | ||
|
|
||
| # Transform to 2D. |
There was a problem hiding this comment.
Why do we need to do this transformation?
There was a problem hiding this comment.
Faiss only accepts a 2D array, so I am doing this transformation.
There was a problem hiding this comment.
Okay, we should think about enforcing it in the binder. If it requires a lot of effort, we can skip it for now but add a comment.
There was a problem hiding this comment.
Ok. In this case, if the shape is not 2D (e.g., the feature vector output from the feature extractor), who will take the responsibility to convert it to 2D? The similarity UDF or something else?
There was a problem hiding this comment.
Is it standard practice to convert the feature vector to 2D for similarity? I was thinking of reporting it as an error to the user, and they will fix it. But if you think it is standard practise, I'm fine with it.
There was a problem hiding this comment.
Ok, I will leave a comment for this and we can think about improving this later.
For your approach, if we instead of asking users to take care the transformation, they will probably need another UDF just for reshaping?
jiashenC
force-pushed
the
similarity-fe
branch
from
4edcb2a to
21c069e
Compare
Similarity UDF