Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/learned sparse main #3

Open
wants to merge 70 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
5971c93
sparse mapper field and query builder
zhichao-aws Sep 5, 2023
0386a95
fix typo
zhichao-aws Sep 6, 2023
489b6e8
Add map result support in neural search for non text embedding models
zane-neo Aug 22, 2023
18022b8
Fix compilation failure issue
zane-neo Sep 1, 2023
886cdeb
Add more UTs
zane-neo Sep 1, 2023
d336f88
add sparse encoding processor
xinyual Sep 6, 2023
bb14947
add sparse encoding processor
xinyual Sep 7, 2023
5bb409b
remove guava in gradle
xinyual Sep 7, 2023
8e8df7e
modify access control
xinyual Sep 7, 2023
5e81ee5
Add map result support in neural search for non text embedding models
zane-neo Aug 22, 2023
20dd78e
Fix compilation failure issue
zane-neo Sep 1, 2023
05c3be8
change output logic
xinyual Sep 8, 2023
734fd50
create abstract
xinyual Sep 11, 2023
c00a4cf
create abstract proccesor
xinyual Sep 11, 2023
a973b42
add abstract class
xinyual Sep 11, 2023
30fc444
remove duplicate code
xinyual Sep 11, 2023
e2a30de
remove duplicate code
xinyual Sep 11, 2023
6b94a17
remove dl process
xinyual Sep 11, 2023
a3d09bd
move static to abstract class
xinyual Sep 11, 2023
f10c94d
update query rewrite logic
zhichao-aws Sep 8, 2023
589b1c0
modify header
zhichao-aws Sep 11, 2023
ec3f426
merge conflict
xinyual Sep 22, 2023
a8520d3
delete index mapper, change to rank_features
zhichao-aws Sep 13, 2023
b964d6c
remove unused import
zhichao-aws Sep 13, 2023
be45f86
list return result
zhichao-aws Sep 13, 2023
dbe00fd
refactor type and listTypeNestedMapKey, tidy
zhichao-aws Sep 14, 2023
c109666
forbid nested input. tidy.
zhichao-aws Sep 14, 2023
90516b2
tidy
zhichao-aws Sep 14, 2023
4d79cc4
enable nested
zhichao-aws Sep 15, 2023
79d861e
fix test
zhichao-aws Sep 15, 2023
9ab2e74
Add ut it to sparse encoding processor (#6)
xinyual Sep 18, 2023
84915f0
utils, tidy
zhichao-aws Sep 15, 2023
3bb95e3
rename to sparse_encoding query
zhichao-aws Sep 15, 2023
b4156f0
add validation and ut
zhichao-aws Sep 18, 2023
4771cd1
sparse encoding query builder ut
zhichao-aws Sep 18, 2023
5d12758
rename
zhichao-aws Sep 19, 2023
51a9ef3
UT for utils
zhichao-aws Sep 19, 2023
77eb300
enrich sparse encoding IT mappings
zhichao-aws Sep 19, 2023
854e9c4
add it
zhichao-aws Sep 19, 2023
03ff8b8
add it
zhichao-aws Sep 19, 2023
e791807
add integ test
zhichao-aws Sep 20, 2023
10e599a
rename resource file
zhichao-aws Sep 20, 2023
1e14a26
tidy
zhichao-aws Sep 20, 2023
473c68d
remove BoundedLinearQuery and TokenScoreUpperBound
zhichao-aws Sep 20, 2023
fa11056
tidy
zhichao-aws Sep 20, 2023
439d628
add delta to loose the equal
zhichao-aws Sep 20, 2023
99a739d
move SparseEncodingQueryBuilder to upper level path
zhichao-aws Sep 20, 2023
65b1e4f
tidy
zhichao-aws Sep 20, 2023
916b3cf
add it
zhichao-aws Sep 20, 2023
283a7a3
Update src/main/java/org/opensearch/neuralsearch/ml/MLCommonsClientAc…
zhichao-aws Sep 25, 2023
c3d9fd3
Update src/main/java/org/opensearch/neuralsearch/util/TokenWeightUtil…
zhichao-aws Sep 25, 2023
11cf97d
restore gradle.propeties
zhichao-aws Sep 25, 2023
057f435
add release notes
zhichao-aws Sep 25, 2023
351bae9
change field modifier to private for NLPProcessor
zhichao-aws Sep 25, 2023
f58a073
add comments
zhichao-aws Sep 25, 2023
791c6ca
use StringUtils to check
zhichao-aws Sep 25, 2023
6878c01
null check
zhichao-aws Sep 25, 2023
c6c631e
modify changelog
zhichao-aws Sep 26, 2023
ec70c34
nit
zhichao-aws Sep 26, 2023
9223e31
nit
zhichao-aws Sep 26, 2023
ba10e27
remove query tokens from user interface
zhichao-aws Sep 26, 2023
9647ac9
fix test
zhichao-aws Sep 26, 2023
169934a
tidy
zhichao-aws Sep 26, 2023
a47c8b6
update function name
zhichao-aws Sep 26, 2023
b48091f
add javadoc
zhichao-aws Sep 26, 2023
cfc847d
remove debug log including inference result
zhichao-aws Sep 27, 2023
508b462
make query text and model id required
zhichao-aws Sep 27, 2023
aae62d4
minor changes based on comments
zhichao-aws Sep 27, 2023
2d51bb9
add locale to String.format
zhichao-aws Sep 27, 2023
9611411
update mock model url
zhichao-aws Sep 27, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
change output logic
Signed-off-by: xinyual <[email protected]>
xinyual authored and zhichao-aws committed Sep 22, 2023
commit 05c3be826e9a497eb5d4f57d4b34aecca72f5fa7
Original file line number Diff line number Diff line change
@@ -96,7 +96,13 @@ public void execute(IngestDocument ingestDocument, BiConsumer<IngestDocument, Ex
if (inferenceList.size() == 0) {
handler.accept(ingestDocument, null);
} else {
mlCommonsClientAccessor.inferenceSentencesWithMapResult(this.modelId, inferenceList, ActionListener.wrap(resultTokenWeights -> {
mlCommonsClientAccessor.inferenceSentencesWithMapResult(this.modelId, inferenceList, ActionListener.wrap(resultMaps -> {
List<Map<String, ?> > resultTokenWeights = new ArrayList<>();
for (Map<String, ?> map: resultMaps)
{
resultTokenWeights.addAll((List<Map<String, ?>>)map.get("response") );
}
log.info(resultTokenWeights);
setVectorFieldsToDocument(ingestDocument, ProcessMap, resultTokenWeights);
handler.accept(ingestDocument, null);
}, e -> { handler.accept(null, e); }));