Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Multi-Vector support for HNSW search #13525

Open
wants to merge 135 commits into
base: main
Choose a base branch
from
Open

[WIP] Multi-Vector support for HNSW search #13525

Show file tree
Hide file tree
Changes from 47 commits
Commits
Show all changes
135 commits
Select commit Hold shift + click to select a range
96e09b3
tensor partial
vigyasharma May 15, 2024
50f3789
docstring edit
vigyasharma May 18, 2024
0050b30
define a KnnFloatTensorField
vigyasharma May 20, 2024
b543668
clean up tensor similarity function
vigyasharma May 20, 2024
c58a3fb
add to fieldIndo and indexing chain schema setup
vigyasharma Jun 13, 2024
b237cdb
started creating new tensor format
vigyasharma Jun 14, 2024
4324151
modified Lucene99FlatVectorsWriter to write tensors
vigyasharma Jun 16, 2024
774e91d
Tensor writer with changes to write tensorDataOffsets
vigyasharma Jun 20, 2024
d15cffc
write byte tensor values
vigyasharma Jun 20, 2024
ada3db1
read tensor metadata and create FieldEntry
vigyasharma Jun 20, 2024
e42a389
Added OffHeapFloatTensorValues
vigyasharma Jun 20, 2024
9bc98bc
support for OffHeapByteTensorValues with code to reuse the bytebuffer
vigyasharma Jun 21, 2024
5e11640
default flat tensor scorer impl
vigyasharma Jun 21, 2024
851a14b
Tensor reader impl. done
vigyasharma Jun 21, 2024
7c39b76
Flat tensor writer impl.
vigyasharma Jun 22, 2024
318455a
revert all tensor changes in FlatVectorsWriter
vigyasharma Jun 22, 2024
b24ae9a
remove unused imports
vigyasharma Jun 23, 2024
a35062f
modify hnswVectorsWriter to build graph with tensors
vigyasharma Jun 25, 2024
c94127d
reuse hnswvectorsreader for tensors
vigyasharma Jun 25, 2024
ced69a3
add knnTensorsformat with an hnsw impl
vigyasharma Jun 25, 2024
b20859f
plumb tensors into the indexing chain
vigyasharma Jun 25, 2024
a30e06e
update lucene94FieldInfosFormat constructor
vigyasharma Jun 25, 2024
13bd4c5
update FieldInfos ctor calls and bw codecs with new params
vigyasharma Jun 25, 2024
834db2b
get tensorReader from codec readers and segment readers
vigyasharma Jun 26, 2024
d214b1b
minor bug fix with readers
vigyasharma Jun 26, 2024
4adefed
add KnnFloatTensorQuery wrapper
vigyasharma Jun 26, 2024
b31e0be
add ByteTensorQuery
vigyasharma Jun 26, 2024
208b041
gradle tidy
vigyasharma Jun 26, 2024
4d173e7
remove commented code
vigyasharma Jun 26, 2024
b5551fc
tidy again
vigyasharma Jun 26, 2024
421b6d6
fix ctor bug
vigyasharma Jun 26, 2024
ee2e0d7
gradle check minus test files
vigyasharma Jun 26, 2024
f50e91a
add no commit tags
vigyasharma Jun 26, 2024
edde79a
make Tensor Similarity Fn a class with Agg enum
vigyasharma Jun 27, 2024
a2a91c4
changed tensor fields, fieldTypes and fieldInfos
vigyasharma Jun 27, 2024
0f7b1dd
modify FlatTensor reader and writers with new FI format
vigyasharma Jun 27, 2024
825e267
modify HnswVectors reader and writer for new FieldInfo
vigyasharma Jun 27, 2024
42475cc
update codecreader and tensor query classes
vigyasharma Jun 27, 2024
0316e02
fix FieldInfo in some files
vigyasharma Jun 27, 2024
0ccb40b
update FieldInfo ctor
vigyasharma Jun 27, 2024
ebc2518
add hashcode to TSF
vigyasharma Jun 27, 2024
21261af
gradle tidy
vigyasharma Jun 27, 2024
872874c
remove nocommits for builds
vigyasharma Jun 27, 2024
d52868f
linting, licenses, and compilation bugs
vigyasharma Jun 27, 2024
749a670
tidy
vigyasharma Jun 27, 2024
ece1728
add missing java docs
vigyasharma Jun 27, 2024
30bdb49
add no commits
vigyasharma Jun 28, 2024
b92d6eb
start single format multivectors change
vigyasharma Jun 30, 2024
1579aea
rename TensorSimilarity to MultiVectorSimilarity
vigyasharma Jul 1, 2024
4f5e494
Add interface to FlatVectorsScorer; del FlatTensorsScorer
vigyasharma Jul 1, 2024
f667e4d
Use DefaultFlatMultiVectorScorer that extends DefFlatVecScorer
vigyasharma Jul 1, 2024
6f13b2e
change OffHeapMVValues; change MVDataOffsetReader
vigyasharma Jul 1, 2024
429a106
change field values to Byte/FloatMultiVectorValue
vigyasharma Jul 1, 2024
104d802
changes to FlatVectorsWriter; del FlatTensorsWriter
vigyasharma Jul 1, 2024
5afa0d9
update FlatVectorsReader to handle multi-vectors
vigyasharma Jul 1, 2024
da32aae
minor import bug
vigyasharma Jul 2, 2024
3fb3808
del FlatTensorReader and FlatTensorFormat
vigyasharma Jul 2, 2024
d0a0b65
Hnsw reader, writer and format changes
vigyasharma Jul 2, 2024
001afd7
remove multivector metadata from hnsw format; not needed
vigyasharma Jul 2, 2024
ca41e85
remove codec change; use static default var for agg fn
vigyasharma Jul 2, 2024
db97c0d
change FieldInfo values; remove TensorQuery classes
vigyasharma Jul 2, 2024
7769e48
restore readers to main branch versions
vigyasharma Jul 2, 2024
1274a27
reword some occurrences of tensor in comments
vigyasharma Jul 2, 2024
05174ff
indexing change and vectorValConsumer changes
vigyasharma Jul 2, 2024
cd08506
some comment changes; use DEF_AGG for MultiVecSimFnAgg
vigyasharma Jul 2, 2024
79450dd
tidy
vigyasharma Jul 2, 2024
cd7e890
compile errors from FI ctor
vigyasharma Jul 2, 2024
0edb868
use ByteBuffer.clear instead of reset
vigyasharma Jul 2, 2024
19fe230
add logs for assert failures
vigyasharma Jul 2, 2024
bca27ff
missed a ;
vigyasharma Jul 2, 2024
93c5077
fix bug around multi-vector condition
vigyasharma Jul 3, 2024
bc0ebc1
bug fix for non-multi-vector case
vigyasharma Jul 3, 2024
1d80b28
tidy
vigyasharma Jul 3, 2024
a33d533
move changes to new format files
vigyasharma Jul 3, 2024
8a098f2
tidy; remove unused imports
vigyasharma Jul 3, 2024
c591b67
remove multi-vector specific check
vigyasharma Jul 3, 2024
51ca4dc
missing docstring
vigyasharma Jul 3, 2024
b5ada6f
tensor partial
vigyasharma May 15, 2024
bc68c8f
docstring edit
vigyasharma May 18, 2024
785c958
define a KnnFloatTensorField
vigyasharma May 20, 2024
d7197be
clean up tensor similarity function
vigyasharma May 20, 2024
9c6d4f5
add to fieldIndo and indexing chain schema setup
vigyasharma Jun 13, 2024
9ce6977
started creating new tensor format
vigyasharma Jun 14, 2024
462e150
modified Lucene99FlatVectorsWriter to write tensors
vigyasharma Jun 16, 2024
b2f3fad
Tensor writer with changes to write tensorDataOffsets
vigyasharma Jun 20, 2024
a88c4ba
write byte tensor values
vigyasharma Jun 20, 2024
8a3ec4f
read tensor metadata and create FieldEntry
vigyasharma Jun 20, 2024
9d6ffd5
Added OffHeapFloatTensorValues
vigyasharma Jun 20, 2024
cf4ca9c
support for OffHeapByteTensorValues with code to reuse the bytebuffer
vigyasharma Jun 21, 2024
a9a345b
default flat tensor scorer impl
vigyasharma Jun 21, 2024
555b055
Tensor reader impl. done
vigyasharma Jun 21, 2024
0abc881
Flat tensor writer impl.
vigyasharma Jun 22, 2024
eebf38e
revert all tensor changes in FlatVectorsWriter
vigyasharma Jun 22, 2024
5334fdb
remove unused imports
vigyasharma Jun 23, 2024
bc02b0b
modify hnswVectorsWriter to build graph with tensors
vigyasharma Jun 25, 2024
1a9ed21
reuse hnswvectorsreader for tensors
vigyasharma Jun 25, 2024
f567095
add knnTensorsformat with an hnsw impl
vigyasharma Jun 25, 2024
059a8ff
plumb tensors into the indexing chain
vigyasharma Jun 25, 2024
7b86353
update lucene94FieldInfosFormat constructor
vigyasharma Jun 25, 2024
1626195
update FieldInfos ctor calls and bw codecs with new params
vigyasharma Jun 25, 2024
27449f5
get tensorReader from codec readers and segment readers
vigyasharma Jun 26, 2024
34d4063
minor bug fix with readers
vigyasharma Jun 26, 2024
543565f
add KnnFloatTensorQuery wrapper
vigyasharma Jun 26, 2024
9a0633b
add ByteTensorQuery
vigyasharma Jun 26, 2024
90c42b2
gradle tidy
vigyasharma Jun 26, 2024
8c3b1c0
remove commented code
vigyasharma Jun 26, 2024
533c2bf
tidy again
vigyasharma Jun 26, 2024
9b7cb0b
fix ctor bug
vigyasharma Jun 26, 2024
c2ae83b
gradle check minus test files
vigyasharma Jun 26, 2024
0ecda0f
add no commit tags
vigyasharma Jun 26, 2024
24d231e
make Tensor Similarity Fn a class with Agg enum
vigyasharma Jun 27, 2024
d08bc68
changed tensor fields, fieldTypes and fieldInfos
vigyasharma Jun 27, 2024
419cc9e
modify FlatTensor reader and writers with new FI format
vigyasharma Jun 27, 2024
75ecb2c
modify HnswVectors reader and writer for new FieldInfo
vigyasharma Jun 27, 2024
e6e6bdf
update codecreader and tensor query classes
vigyasharma Jun 27, 2024
c0912d1
fix FieldInfo in some files
vigyasharma Jun 27, 2024
c24a99f
update FieldInfo ctor
vigyasharma Jun 27, 2024
04636b9
add hashcode to TSF
vigyasharma Jun 27, 2024
4b2bad4
gradle tidy
vigyasharma Jun 27, 2024
a98035c
remove nocommits for builds
vigyasharma Jun 27, 2024
56db6d0
linting, licenses, and compilation bugs
vigyasharma Jun 27, 2024
97d327d
tidy
vigyasharma Jun 27, 2024
acdd158
add missing java docs
vigyasharma Jun 27, 2024
c80cbcf
add no commits
vigyasharma Jun 28, 2024
6ca9e14
merge new changes
vigyasharma Jul 3, 2024
3365889
fix all merge conflicts
vigyasharma Jul 4, 2024
12f5b54
no commits
vigyasharma Jul 4, 2024
ffda63f
remove isMultiVector member in Lucene99FlatMultiVectorsWriter.FieldWr…
cpoerschke Jul 12, 2024
b2b95cd
remove FieldInfo.isMultiVector in favour of MultiVectorSimilarityFunc…
cpoerschke Jul 12, 2024
eacc63c
in IndexingChain.FieldSchema fold setMultiVectors into setVectors
cpoerschke Jul 12, 2024
1570690
remove metadata check for fixed vector lengths. add debug logs
vigyasharma Oct 21, 2024
b1e5568
fix hnsw vector to read MV scorer supplier; fix loop and array copy i…
vigyasharma Oct 22, 2024
ce2face
add Aggregate to FieldInfos format
vigyasharma Oct 22, 2024
2e52495
assert and logs on vector-data rw offsets
vigyasharma Oct 24, 2024
7aa9555
go to base vectorscorer for single valued use-case
vigyasharma Oct 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 3 additions & 9 deletions ...-codecs/src/java/org/apache/lucene/backward_codecs/lucene60/Lucene60FieldInfosFormat.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,15 +23,7 @@
import org.apache.lucene.codecs.CodecUtil;
import org.apache.lucene.codecs.DocValuesFormat;
import org.apache.lucene.codecs.FieldInfosFormat;
import org.apache.lucene.index.CorruptIndexException;
import org.apache.lucene.index.DocValuesType;
import org.apache.lucene.index.FieldInfo;
import org.apache.lucene.index.FieldInfos;
import org.apache.lucene.index.IndexFileNames;
import org.apache.lucene.index.IndexOptions;
import org.apache.lucene.index.SegmentInfo;
import org.apache.lucene.index.VectorEncoding;
import org.apache.lucene.index.VectorSimilarityFunction;
import org.apache.lucene.index.*;
import org.apache.lucene.store.ChecksumIndexInput;
import org.apache.lucene.store.DataOutput;
import org.apache.lucene.store.Directory;
Expand Down Expand Up @@ -218,6 +210,8 @@ private FieldInfo[] readFieldInfos(IndexInput input, int version) throws IOExcep
0,
VectorEncoding.FLOAT32,
VectorSimilarityFunction.EUCLIDEAN,
false,
TensorSimilarityFunction.Aggregation.SUM_MAX,
isSoftDeletesField,
false);
} catch (IllegalStateException e) {
Expand Down
18 changes: 6 additions & 12 deletions ...ne/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene80/Lucene80Codec.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,18 +23,7 @@
import org.apache.lucene.backward_codecs.lucene60.Lucene60FieldInfosFormat;
import org.apache.lucene.backward_codecs.lucene60.Lucene60PointsFormat;
import org.apache.lucene.backward_codecs.lucene70.Lucene70SegmentInfoFormat;
import org.apache.lucene.codecs.Codec;
import org.apache.lucene.codecs.CompoundFormat;
import org.apache.lucene.codecs.DocValuesFormat;
import org.apache.lucene.codecs.FieldInfosFormat;
import org.apache.lucene.codecs.KnnVectorsFormat;
import org.apache.lucene.codecs.LiveDocsFormat;
import org.apache.lucene.codecs.NormsFormat;
import org.apache.lucene.codecs.PointsFormat;
import org.apache.lucene.codecs.PostingsFormat;
import org.apache.lucene.codecs.SegmentInfoFormat;
import org.apache.lucene.codecs.StoredFieldsFormat;
import org.apache.lucene.codecs.TermVectorsFormat;
import org.apache.lucene.codecs.*;
import org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat;
import org.apache.lucene.codecs.perfield.PerFieldPostingsFormat;

Expand Down Expand Up @@ -132,4 +121,9 @@ public final NormsFormat normsFormat() {
public final KnnVectorsFormat knnVectorsFormat() {
return KnnVectorsFormat.EMPTY;
}

@Override
public KnnTensorsFormat knnTensorsFormat() {
return KnnTensorsFormat.EMPTY;
}
}
19 changes: 6 additions & 13 deletions ...ne/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene84/Lucene84Codec.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,19 +26,7 @@
import org.apache.lucene.backward_codecs.lucene60.Lucene60PointsFormat;
import org.apache.lucene.backward_codecs.lucene70.Lucene70SegmentInfoFormat;
import org.apache.lucene.backward_codecs.lucene80.Lucene80NormsFormat;
import org.apache.lucene.codecs.Codec;
import org.apache.lucene.codecs.CompoundFormat;
import org.apache.lucene.codecs.DocValuesFormat;
import org.apache.lucene.codecs.FieldInfosFormat;
import org.apache.lucene.codecs.FilterCodec;
import org.apache.lucene.codecs.KnnVectorsFormat;
import org.apache.lucene.codecs.LiveDocsFormat;
import org.apache.lucene.codecs.NormsFormat;
import org.apache.lucene.codecs.PointsFormat;
import org.apache.lucene.codecs.PostingsFormat;
import org.apache.lucene.codecs.SegmentInfoFormat;
import org.apache.lucene.codecs.StoredFieldsFormat;
import org.apache.lucene.codecs.TermVectorsFormat;
import org.apache.lucene.codecs.*;
import org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat;
import org.apache.lucene.codecs.perfield.PerFieldPostingsFormat;

Expand Down Expand Up @@ -138,6 +126,11 @@ public KnnVectorsFormat knnVectorsFormat() {
return KnnVectorsFormat.EMPTY;
}

@Override
public KnnTensorsFormat knnTensorsFormat() {
return KnnTensorsFormat.EMPTY;
}

/**
* Returns the postings format that should be used for writing new segments of <code>field</code>.
*
Expand Down
19 changes: 6 additions & 13 deletions ...ne/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene86/Lucene86Codec.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,19 +25,7 @@
import org.apache.lucene.backward_codecs.lucene60.Lucene60FieldInfosFormat;
import org.apache.lucene.backward_codecs.lucene80.Lucene80NormsFormat;
import org.apache.lucene.backward_codecs.lucene84.Lucene84PostingsFormat;
import org.apache.lucene.codecs.Codec;
import org.apache.lucene.codecs.CompoundFormat;
import org.apache.lucene.codecs.DocValuesFormat;
import org.apache.lucene.codecs.FieldInfosFormat;
import org.apache.lucene.codecs.FilterCodec;
import org.apache.lucene.codecs.KnnVectorsFormat;
import org.apache.lucene.codecs.LiveDocsFormat;
import org.apache.lucene.codecs.NormsFormat;
import org.apache.lucene.codecs.PointsFormat;
import org.apache.lucene.codecs.PostingsFormat;
import org.apache.lucene.codecs.SegmentInfoFormat;
import org.apache.lucene.codecs.StoredFieldsFormat;
import org.apache.lucene.codecs.TermVectorsFormat;
import org.apache.lucene.codecs.*;
import org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat;
import org.apache.lucene.codecs.perfield.PerFieldPostingsFormat;

Expand Down Expand Up @@ -137,6 +125,11 @@ public final KnnVectorsFormat knnVectorsFormat() {
return KnnVectorsFormat.EMPTY;
}

@Override
public KnnTensorsFormat knnTensorsFormat() {
return KnnTensorsFormat.EMPTY;
}

/**
* Returns the postings format that should be used for writing new segments of <code>field</code>.
*
Expand Down
19 changes: 6 additions & 13 deletions ...ne/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene87/Lucene87Codec.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,19 +27,7 @@
import org.apache.lucene.backward_codecs.lucene84.Lucene84PostingsFormat;
import org.apache.lucene.backward_codecs.lucene86.Lucene86PointsFormat;
import org.apache.lucene.backward_codecs.lucene86.Lucene86SegmentInfoFormat;
import org.apache.lucene.codecs.Codec;
import org.apache.lucene.codecs.CompoundFormat;
import org.apache.lucene.codecs.DocValuesFormat;
import org.apache.lucene.codecs.FieldInfosFormat;
import org.apache.lucene.codecs.FilterCodec;
import org.apache.lucene.codecs.KnnVectorsFormat;
import org.apache.lucene.codecs.LiveDocsFormat;
import org.apache.lucene.codecs.NormsFormat;
import org.apache.lucene.codecs.PointsFormat;
import org.apache.lucene.codecs.PostingsFormat;
import org.apache.lucene.codecs.SegmentInfoFormat;
import org.apache.lucene.codecs.StoredFieldsFormat;
import org.apache.lucene.codecs.TermVectorsFormat;
import org.apache.lucene.codecs.*;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it will help with churn, etc. and we prefer not having * imports.

import org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat;
import org.apache.lucene.codecs.perfield.PerFieldPostingsFormat;

Expand Down Expand Up @@ -162,6 +150,11 @@ public final KnnVectorsFormat knnVectorsFormat() {
return KnnVectorsFormat.EMPTY;
}

@Override
public KnnTensorsFormat knnTensorsFormat() {
return KnnTensorsFormat.EMPTY;
}

/**
* Returns the postings format that should be used for writing new segments of <code>field</code>.
*
Expand Down
19 changes: 6 additions & 13 deletions ...ne/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene90/Lucene90Codec.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,19 +17,7 @@
package org.apache.lucene.backward_codecs.lucene90;

import java.util.Objects;
import org.apache.lucene.codecs.Codec;
import org.apache.lucene.codecs.CompoundFormat;
import org.apache.lucene.codecs.DocValuesFormat;
import org.apache.lucene.codecs.FieldInfosFormat;
import org.apache.lucene.codecs.FilterCodec;
import org.apache.lucene.codecs.KnnVectorsFormat;
import org.apache.lucene.codecs.LiveDocsFormat;
import org.apache.lucene.codecs.NormsFormat;
import org.apache.lucene.codecs.PointsFormat;
import org.apache.lucene.codecs.PostingsFormat;
import org.apache.lucene.codecs.SegmentInfoFormat;
import org.apache.lucene.codecs.StoredFieldsFormat;
import org.apache.lucene.codecs.TermVectorsFormat;
import org.apache.lucene.codecs.*;
import org.apache.lucene.codecs.lucene90.Lucene90CompoundFormat;
import org.apache.lucene.codecs.lucene90.Lucene90DocValuesFormat;
import org.apache.lucene.codecs.lucene90.Lucene90LiveDocsFormat;
Expand Down Expand Up @@ -165,6 +153,11 @@ public KnnVectorsFormat knnVectorsFormat() {
return knnVectorsFormat;
}

@Override
public KnnTensorsFormat knnTensorsFormat() {
return KnnTensorsFormat.EMPTY;
}

/**
* Returns the postings format that should be used for writing new segments of <code>field</code>.
*
Expand Down
12 changes: 3 additions & 9 deletions ...-codecs/src/java/org/apache/lucene/backward_codecs/lucene90/Lucene90FieldInfosFormat.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,15 +22,7 @@
import org.apache.lucene.codecs.CodecUtil;
import org.apache.lucene.codecs.DocValuesFormat;
import org.apache.lucene.codecs.FieldInfosFormat;
import org.apache.lucene.index.CorruptIndexException;
import org.apache.lucene.index.DocValuesType;
import org.apache.lucene.index.FieldInfo;
import org.apache.lucene.index.FieldInfos;
import org.apache.lucene.index.IndexFileNames;
import org.apache.lucene.index.IndexOptions;
import org.apache.lucene.index.SegmentInfo;
import org.apache.lucene.index.VectorEncoding;
import org.apache.lucene.index.VectorSimilarityFunction;
import org.apache.lucene.index.*;
import org.apache.lucene.store.ChecksumIndexInput;
import org.apache.lucene.store.DataOutput;
import org.apache.lucene.store.Directory;
Expand Down Expand Up @@ -195,6 +187,8 @@ public FieldInfos read(
vectorDimension,
VectorEncoding.FLOAT32,
vectorDistFunc,
false,
TensorSimilarityFunction.Aggregation.SUM_MAX,
isSoftDeletesField,
false);
infos[i].checkConsistency();
Expand Down
19 changes: 6 additions & 13 deletions ...ne/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene91/Lucene91Codec.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,19 +20,7 @@
import org.apache.lucene.backward_codecs.lucene90.Lucene90FieldInfosFormat;
import org.apache.lucene.backward_codecs.lucene90.Lucene90PostingsFormat;
import org.apache.lucene.backward_codecs.lucene90.Lucene90SegmentInfoFormat;
import org.apache.lucene.codecs.Codec;
import org.apache.lucene.codecs.CompoundFormat;
import org.apache.lucene.codecs.DocValuesFormat;
import org.apache.lucene.codecs.FieldInfosFormat;
import org.apache.lucene.codecs.FilterCodec;
import org.apache.lucene.codecs.KnnVectorsFormat;
import org.apache.lucene.codecs.LiveDocsFormat;
import org.apache.lucene.codecs.NormsFormat;
import org.apache.lucene.codecs.PointsFormat;
import org.apache.lucene.codecs.PostingsFormat;
import org.apache.lucene.codecs.SegmentInfoFormat;
import org.apache.lucene.codecs.StoredFieldsFormat;
import org.apache.lucene.codecs.TermVectorsFormat;
import org.apache.lucene.codecs.*;
import org.apache.lucene.codecs.lucene90.Lucene90CompoundFormat;
import org.apache.lucene.codecs.lucene90.Lucene90DocValuesFormat;
import org.apache.lucene.codecs.lucene90.Lucene90LiveDocsFormat;
Expand Down Expand Up @@ -165,6 +153,11 @@ public KnnVectorsFormat knnVectorsFormat() {
return knnVectorsFormat;
}

@Override
public KnnTensorsFormat knnTensorsFormat() {
return KnnTensorsFormat.EMPTY;
}

/**
* Returns the postings format that should be used for writing new segments of <code>field</code>.
*
Expand Down
19 changes: 6 additions & 13 deletions ...ne/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene92/Lucene92Codec.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,19 +20,7 @@
import org.apache.lucene.backward_codecs.lucene90.Lucene90FieldInfosFormat;
import org.apache.lucene.backward_codecs.lucene90.Lucene90PostingsFormat;
import org.apache.lucene.backward_codecs.lucene90.Lucene90SegmentInfoFormat;
import org.apache.lucene.codecs.Codec;
import org.apache.lucene.codecs.CompoundFormat;
import org.apache.lucene.codecs.DocValuesFormat;
import org.apache.lucene.codecs.FieldInfosFormat;
import org.apache.lucene.codecs.FilterCodec;
import org.apache.lucene.codecs.KnnVectorsFormat;
import org.apache.lucene.codecs.LiveDocsFormat;
import org.apache.lucene.codecs.NormsFormat;
import org.apache.lucene.codecs.PointsFormat;
import org.apache.lucene.codecs.PostingsFormat;
import org.apache.lucene.codecs.SegmentInfoFormat;
import org.apache.lucene.codecs.StoredFieldsFormat;
import org.apache.lucene.codecs.TermVectorsFormat;
import org.apache.lucene.codecs.*;
import org.apache.lucene.codecs.lucene90.Lucene90CompoundFormat;
import org.apache.lucene.codecs.lucene90.Lucene90DocValuesFormat;
import org.apache.lucene.codecs.lucene90.Lucene90LiveDocsFormat;
Expand Down Expand Up @@ -168,6 +156,11 @@ public KnnVectorsFormat knnVectorsFormat() {
return knnVectorsFormat;
}

@Override
public KnnTensorsFormat knnTensorsFormat() {
return KnnTensorsFormat.EMPTY;
}

/**
* Returns the postings format that should be used for writing new segments of <code>field</code>.
*
Expand Down
19 changes: 6 additions & 13 deletions ...ne/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene94/Lucene94Codec.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,19 +19,7 @@
import java.util.Objects;
import org.apache.lucene.backward_codecs.lucene90.Lucene90PostingsFormat;
import org.apache.lucene.backward_codecs.lucene90.Lucene90SegmentInfoFormat;
import org.apache.lucene.codecs.Codec;
import org.apache.lucene.codecs.CompoundFormat;
import org.apache.lucene.codecs.DocValuesFormat;
import org.apache.lucene.codecs.FieldInfosFormat;
import org.apache.lucene.codecs.FilterCodec;
import org.apache.lucene.codecs.KnnVectorsFormat;
import org.apache.lucene.codecs.LiveDocsFormat;
import org.apache.lucene.codecs.NormsFormat;
import org.apache.lucene.codecs.PointsFormat;
import org.apache.lucene.codecs.PostingsFormat;
import org.apache.lucene.codecs.SegmentInfoFormat;
import org.apache.lucene.codecs.StoredFieldsFormat;
import org.apache.lucene.codecs.TermVectorsFormat;
import org.apache.lucene.codecs.*;
import org.apache.lucene.codecs.lucene90.Lucene90CompoundFormat;
import org.apache.lucene.codecs.lucene90.Lucene90DocValuesFormat;
import org.apache.lucene.codecs.lucene90.Lucene90LiveDocsFormat;
Expand Down Expand Up @@ -168,6 +156,11 @@ public KnnVectorsFormat knnVectorsFormat() {
return knnVectorsFormat;
}

@Override
public KnnTensorsFormat knnTensorsFormat() {
return KnnTensorsFormat.EMPTY;
}

/**
* Returns the postings format that should be used for writing new segments of <code>field</code>.
*
Expand Down
19 changes: 6 additions & 13 deletions ...ne/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene95/Lucene95Codec.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,19 +19,7 @@
import java.util.Objects;
import org.apache.lucene.backward_codecs.lucene90.Lucene90PostingsFormat;
import org.apache.lucene.backward_codecs.lucene90.Lucene90SegmentInfoFormat;
import org.apache.lucene.codecs.Codec;
import org.apache.lucene.codecs.CompoundFormat;
import org.apache.lucene.codecs.DocValuesFormat;
import org.apache.lucene.codecs.FieldInfosFormat;
import org.apache.lucene.codecs.FilterCodec;
import org.apache.lucene.codecs.KnnVectorsFormat;
import org.apache.lucene.codecs.LiveDocsFormat;
import org.apache.lucene.codecs.NormsFormat;
import org.apache.lucene.codecs.PointsFormat;
import org.apache.lucene.codecs.PostingsFormat;
import org.apache.lucene.codecs.SegmentInfoFormat;
import org.apache.lucene.codecs.StoredFieldsFormat;
import org.apache.lucene.codecs.TermVectorsFormat;
import org.apache.lucene.codecs.*;
import org.apache.lucene.codecs.lucene90.Lucene90CompoundFormat;
import org.apache.lucene.codecs.lucene90.Lucene90DocValuesFormat;
import org.apache.lucene.codecs.lucene90.Lucene90LiveDocsFormat;
Expand Down Expand Up @@ -168,6 +156,11 @@ public KnnVectorsFormat knnVectorsFormat() {
return knnVectorsFormat;
}

@Override
public KnnTensorsFormat knnTensorsFormat() {
return KnnTensorsFormat.EMPTY;
}

/**
* Returns the postings format that should be used for writing new segments of <code>field</code>.
*
Expand Down
19 changes: 7 additions & 12 deletions lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextCodec.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,18 +16,7 @@
*/
package org.apache.lucene.codecs.simpletext;

import org.apache.lucene.codecs.Codec;
import org.apache.lucene.codecs.CompoundFormat;
import org.apache.lucene.codecs.DocValuesFormat;
import org.apache.lucene.codecs.FieldInfosFormat;
import org.apache.lucene.codecs.KnnVectorsFormat;
import org.apache.lucene.codecs.LiveDocsFormat;
import org.apache.lucene.codecs.NormsFormat;
import org.apache.lucene.codecs.PointsFormat;
import org.apache.lucene.codecs.PostingsFormat;
import org.apache.lucene.codecs.SegmentInfoFormat;
import org.apache.lucene.codecs.StoredFieldsFormat;
import org.apache.lucene.codecs.TermVectorsFormat;
import org.apache.lucene.codecs.*;

/**
* plain text index format.
Expand Down Expand Up @@ -107,4 +96,10 @@ public PointsFormat pointsFormat() {
public KnnVectorsFormat knnVectorsFormat() {
return knnVectorsFormat;
}

@Override
public KnnTensorsFormat knnTensorsFormat() {
// TODO: Implement a SimpleTextFormat
return KnnTensorsFormat.EMPTY;
}
}
3 changes: 3 additions & 0 deletions lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextFieldInfosFormat.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
import org.apache.lucene.index.IndexFileNames;
import org.apache.lucene.index.IndexOptions;
import org.apache.lucene.index.SegmentInfo;
import org.apache.lucene.index.TensorSimilarityFunction;
import org.apache.lucene.index.VectorEncoding;
import org.apache.lucene.index.VectorSimilarityFunction;
import org.apache.lucene.store.ChecksumIndexInput;
Expand Down Expand Up @@ -199,6 +200,8 @@ public FieldInfos read(
vectorNumDimensions,
vectorEncoding,
vectorDistFunc,
false,
TensorSimilarityFunction.Aggregation.SUM_MAX,
isSoftDeletesField,
isParentField);
}
Expand Down
Loading