NIFI-12831: Add PutOpenSearchVector and QueryOpenSearchVector processors #8441
Conversation
There was a problem hiding this comment.
Thanks for working on these new components @mark-bathori. This highlights the need for moving Python Processors to a separate repository, but that doesn't need to prevent this from going forward.
On a cursory review, I noted one security concern related to certificate verification. We should not support disabling certificate verification as it provides a fundamental security check for TLS communication.
| VERIFY_CERTIFICATES = PropertyDescriptor( | ||
| name="Verify Certificates", | ||
| description="The password to use for authenticating to OpenSearch server", | ||
| allowable_values=["true", "false"], | ||
| default_value="false", | ||
| required=False, | ||
| validators=[StandardValidators.NON_EMPTY_VALIDATOR] | ||
| ) |
There was a problem hiding this comment.
In keeping with practices in other Processors, we should not support disabling certificate verification.
There was a problem hiding this comment.
Thanks for the comment @exceptionfactory, I'll remove this property.
...extensions/nifi-text-embeddings-module/src/main/python/vectorstores/OpenSearchVectorUtils.py
Outdated
Show resolved
Hide resolved
...extensions/nifi-text-embeddings-module/src/main/python/vectorstores/OpenSearchVectorUtils.py
Outdated
Show resolved
Hide resolved
...n-extensions/nifi-text-embeddings-module/src/main/python/vectorstores/PutOpenSearchVector.py
Outdated
Show resolved
Hide resolved
...n-extensions/nifi-text-embeddings-module/src/main/python/vectorstores/PutOpenSearchVector.py
Outdated
Show resolved
Hide resolved
...n-extensions/nifi-text-embeddings-module/src/main/python/vectorstores/PutOpenSearchVector.py
Outdated
Show resolved
Hide resolved
| ENGINE_VALUES = dict([NMSLIB, FAISS, LUCENE]) | ||
|
|
||
| # Space types | ||
| L2 = ("L2 (Euclidean distance)", "l2") |
There was a problem hiding this comment.
The space types (L2, L1, LINF, COSINESIMIL) seem to be the same for PutOpenSearchVector.py and QueryOpenSearchVector.py, those can be extracted to OpenSearchVectorUtils.py.
| ) | ||
| VECTOR_FIELD = PropertyDescriptor( | ||
| name="Vector Field Name", | ||
| description="The name of Document field where the embeddings are stored. This field need to be a 'knn_vector' typed field.", |
There was a problem hiding this comment.
Please use "document" here as well (as it is done in other descriptions)
...n-extensions/nifi-text-embeddings-module/src/main/python/vectorstores/PutOpenSearchVector.py
Outdated
Show resolved
Hide resolved
...n-extensions/nifi-text-embeddings-module/src/main/python/vectorstores/PutOpenSearchVector.py
Outdated
Show resolved
Hide resolved
...n-extensions/nifi-text-embeddings-module/src/main/python/vectorstores/PutOpenSearchVector.py
Outdated
Show resolved
Hide resolved
Signed-off-by: Pierre Villard <[email protected]> This closes apache#8441.
Signed-off-by: Pierre Villard <[email protected]> This closes apache#8441.
Summary
NIFI-12831
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking
Pull Request Tracking
NIFI-00000NIFI-00000Pull Request Formatting
mainbranchVerification
Please indicate the verification steps performed prior to pull request creation.
Build
mvn clean install -P contrib-checkLicensing
LICENSEandNOTICEfilesDocumentation