Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Backport 2.x Manually][Fix] add non-null check for queryBuilder in NeuralQueryEnricherProcessor #618

Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
f1dfaed
Fix minor build.gradle issues (#28) (#29)
opensearch-trigger-bot[bot] Oct 24, 2022
e797931
Fix group id for ml-commons dependency (#32) (#33)
opensearch-trigger-bot[bot] Oct 25, 2022
d404336
Add opensearch prefix to plugin name (#42)
opensearch-trigger-bot[bot] Oct 29, 2022
b7ad60d
Fixing the locale changes from locale.default to locale.ROOT to fix t…
navneet1v Oct 29, 2022
f153ced
Change text embedding processor to async mode for better isolation (#…
navneet1v Oct 29, 2022
1342a07
Added windows support for CI. (#40) (#41)
opensearch-trigger-bot[bot] Oct 29, 2022
e861cc1
Add integration tests for neural query (#47)
opensearch-trigger-bot[bot] Oct 31, 2022
10a7ee7
Switch processor IT to use Lucene (#49)
opensearch-trigger-bot[bot] Nov 1, 2022
9828869
Add release note draft automation (#53)
opensearch-trigger-bot[bot] Nov 3, 2022
0765c8b
Refactor project package structure (#56)
opensearch-trigger-bot[bot] Nov 3, 2022
7de489d
Add 2.4.0.0 releases notes (#59)
opensearch-trigger-bot[bot] Nov 4, 2022
8cdaf5f
Wrap PR links in parentheses (#62)
opensearch-trigger-bot[bot] Nov 4, 2022
d088a45
Update release to experimental (#65)
opensearch-trigger-bot[bot] Nov 7, 2022
5634ada
Fix error caching models in integ test (#68)
opensearch-trigger-bot[bot] Nov 11, 2022
ce55c1a
Change the behavior when embedding fields not present (#72) (#77)
opensearch-trigger-bot[bot] Dec 6, 2022
b7d4e34
Added release notes for 2.4.1 version (#79) (#80)
opensearch-trigger-bot[bot] Dec 6, 2022
2bfa8a8
Add filter option for query type (#89)
opensearch-trigger-bot[bot] Dec 21, 2022
93af62f
Increment version to 2.5.0-SNAPSHOT (#76)
opensearch-trigger-bot[bot] Dec 22, 2022
8174048
Updated the neural search plugin to implement the Extensible Plugin i…
navneet1v Dec 22, 2022
0e03e3e
Remove unused MLPredict Transport action from src. (#94) (#97)
opensearch-trigger-bot[bot] Jan 9, 2023
054629f
Add retry mechanism for neural search inference (#98)
opensearch-trigger-bot[bot] Jan 10, 2023
9313413
Use short-form MAINTAINERS.md (#85)
opensearch-trigger-bot[bot] Jan 10, 2023
ecede3d
Add release notes for 25 (#101)
opensearch-trigger-bot[bot] Jan 11, 2023
c6721b3
[Backport 2.x] Update default setting for integ tests (#107)
jmazanec15 Jan 17, 2023
619fb94
Update snapshot to 2.6.0 (#117) (#121)
opensearch-trigger-bot[bot] Feb 22, 2023
7c25309
Add release notes (#118) (#120)
opensearch-trigger-bot[bot] Feb 28, 2023
9aa9461
Increment version to 2.7.0-SNAPSHOT (#115)
opensearch-trigger-bot[bot] Mar 8, 2023
081e42d
Add GHA to publish to maven repository (#130) (#132)
opensearch-trigger-bot[bot] Mar 8, 2023
4c8c865
Add reflection dependency (#136) (#137)
opensearch-trigger-bot[bot] Mar 15, 2023
698f850
Add CHANGELOG (#135) (#138)
opensearch-trigger-bot[bot] Mar 15, 2023
7d5ebd0
Modify SetOnce to fix forbidden API (#122)
jmazanec15 Mar 21, 2023
a128750
Created untriaged issue workflow. (#112) (#140)
opensearch-trigger-bot[bot] Mar 21, 2023
a0b7a07
Fixed compilation errors after recent changes in core (#129) (#147)
naveentatikonda Apr 3, 2023
415bf23
Add 2.7.0 Release Notes (#153) (#155)
opensearch-trigger-bot[bot] Apr 17, 2023
e8c732f
Added badges and daily CI runs. (#164) (#165)
opensearch-trigger-bot[bot] Apr 24, 2023
acdb7f2
[Backport 2.x] Baseline owners and maintainers (#170) (#171)
martin-gaievski May 12, 2023
6ad0099
Bump gradle version to 8.1.1 (#169) (#173)
opensearch-trigger-bot[bot] May 17, 2023
0be2195
Add Auto Release Workflow (#167) (#177)
opensearch-trigger-bot[bot] May 24, 2023
aca5f60
Adding 2.8 release notes (#178) (#179)
opensearch-trigger-bot[bot] May 26, 2023
3547d05
Update version to 2.8-SNAPSHOT (#181)
martin-gaievski May 26, 2023
5228fc7
Adding model group id to integ tests (#189)
martin-gaievski Jun 1, 2023
7ee1eb3
Increment version to 2.9.0-SNAPSHOT (#191)
opensearch-trigger-bot[bot] Jun 6, 2023
9599371
Fix update document with knnn_vector size not matching issue (#208)
zane-neo Jul 11, 2023
f9d64d8
Add 2.9.0 release note (#209) (#211)
opensearch-trigger-bot[bot] Jul 11, 2023
f2dd286
Increment version to 2.10.0-SNAPSHOT (#215)
opensearch-trigger-bot[bot] Jul 31, 2023
8a7ab9b
Fix compilation errors, change core package name in import (#219) (#221)
opensearch-trigger-bot[bot] Jul 31, 2023
adf3925
Porting different Backward incompatible changes happened in Core to 2…
navneet1v Aug 17, 2023
185050a
[Backport 2.x] Added Score Normalization and Combination feature, man…
martin-gaievski Aug 25, 2023
e564d4d
Changed format for hybrid query results to a single list of scores wi…
opensearch-trigger-bot[bot] Aug 29, 2023
40479e9
Added validations for score combination weights in Hybrid Search (#26…
opensearch-trigger-bot[bot] Aug 30, 2023
d0ed52e
Made hybrid search active by default by flipping feature flag default…
martin-gaievski Sep 2, 2023
adecea5
Added release notes for 2.10 release (#277) (#287)
navneet1v Sep 7, 2023
415082e
[Backport 2.x] Supporting sparse semantic retrieval in neural search …
opensearch-trigger-bot[bot] Sep 27, 2023
848ec21
[Backport to 2.x] Fix flaky integ test caused by latency of undeploy …
zane-neo Sep 28, 2023
9e12de8
Made hybrid QPS extending QueryPhaseSearcherWrapper (#356) (#364)
opensearch-trigger-bot[bot] Sep 29, 2023
7ae48df
Add max token score for SparseEncodingQueryBuilder and do renaming (#…
zhichao-aws Oct 1, 2023
610b5ef
Increment version to 2.11.0-SNAPSHOT (#289)
opensearch-trigger-bot[bot] Oct 3, 2023
8d749b8
[Backport 2.x] [Feature] Support for Default Model Id (#365)
vibrantvarun Oct 3, 2023
841f280
Added Multimodal semantic search feature (#359)
martin-gaievski Oct 3, 2023
bc6ac7a
Fixed exception in Hybrid Query for one shard and multiple node (#396…
opensearch-trigger-bot[bot] Oct 6, 2023
47d1aee
Fixed error for case when mltensor has data as null (#404)
martin-gaievski Oct 6, 2023
0ed523b
Add 2.11 release note (#392) (#407)
heemin32 Oct 6, 2023
87d6e5e
[bug fix] Fix async actions are left in neural_sparse query (#438) (#…
opensearch-trigger-bot[bot] Oct 11, 2023
0517a91
Adding developer guide (#471) (#472)
opensearch-trigger-bot[bot] Oct 25, 2023
5442f84
Onboard jenkins prod docker images on github actions (#483) (#484)
opensearch-trigger-bot[bot] Nov 3, 2023
2fe7ed2
Update dependency org.json:json to v20231013 (#481) (#489)
opensearch-trigger-bot[bot] Nov 10, 2023
a430c6d
[AUTO] Increment version to 2.12.0-SNAPSHOT (#414)
opensearch-trigger-bot[bot] Nov 21, 2023
ef19ffa
Fixed exception for case when Hybrid query being wrapped into bool qu…
martin-gaievski Nov 21, 2023
04bf2a4
Deprecate max_token_score field of neural_sparse query (#478)
zhichao-aws Nov 22, 2023
27a38cb
Fixed Hybrid query for cases when it's wrapped into other compound qu…
martin-gaievski Dec 1, 2023
a72b394
Added the github action to copy the attached issues label to PR. (#50…
opensearch-trigger-bot[bot] Dec 5, 2023
63fe67f
[Backport 2.x] Added support for jdk-21 (#513)
martin-gaievski Dec 6, 2023
31b3f66
Fixing multiple issues reported in #497 (#524) (#527)
martin-gaievski Jan 2, 2024
77a4464
[Backport 2.x] Backport 515 to 2.x (#528)
vibrantvarun Jan 3, 2024
bb2d59e
Update Developer guide with steps to run BWC Test Suite (#531) (#532)
opensearch-trigger-bot[bot] Jan 8, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
# This should match the owning team set up in https://github.com/orgs/opensearch-project/teams
* @opensearch-project/neural-search
* @heemin32 @navneet1v @VijayanB @vamshin @jmazanec15 @naveentatikonda @junqiu-lei @martin-gaievski @sean-zheng-amazon @model-collapse @wujunshen @zane-neo @ylwu-amzn @jngz-es
2 changes: 1 addition & 1 deletion .github/draft-release-notes-config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,4 +37,4 @@ categories:
- 'Maintenance'
- title: 'Refactoring'
labels:
- 'Refactoring'
- 'Refactoring'
95 changes: 87 additions & 8 deletions .github/workflows/CI.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
name: Build and Test Neural Search
on:
schedule:
- cron: '0 0 * * *' # every night
push:
branches:
- "*"
Expand All @@ -10,13 +12,54 @@ on:
- "feature/**"

jobs:
Check-neural-search:
Get-CI-Image-Tag:
uses: opensearch-project/opensearch-build/.github/workflows/get-ci-image-tag.yml@main
with:
product: opensearch

Check-neural-search-linux:
needs: Get-CI-Image-Tag
strategy:
matrix:
java: [11, 17, 21]
os: [ubuntu-latest]

name: Gradle Check Linux
runs-on: ${{ matrix.os }}
container:
# using the same image which is used by opensearch-build team to build the OpenSearch Distribution
# this image tag is subject to change as more dependencies and updates will arrive over time
image: ${{ needs.Get-CI-Image-Tag.outputs.ci-image-version-linux }}
# need to switch to root so that github actions can install runner binary on container without permission issues.
options: --user root

steps:
- name: Checkout neural-search
uses: actions/checkout@v1

- name: Setup Java ${{ matrix.java }}
uses: actions/setup-java@v1
with:
java-version: ${{ matrix.java }}

- name: Run build
run: |
chown -R 1000:1000 `pwd`
su `id -un 1000` -c "./gradlew check"

- name: Upload Coverage Report
uses: codecov/codecov-action@v1
with:
token: ${{ secrets.CODECOV_TOKEN }}

Check-neural-search-windows:
strategy:
matrix:
java: [11, 17]
java: [11, 17, 21]
os: [windows-latest]

name: Run Gradle Check on neural-search Plugin
runs-on: ubuntu-latest
name: Gradle Check Windows
runs-on: ${{ matrix.os }}

steps:
- name: Checkout neural-search
Expand All @@ -36,13 +79,49 @@ jobs:
with:
token: ${{ secrets.CODECOV_TOKEN }}

Precommit-neural-search:
Precommit-neural-search-linux:
needs: Get-CI-Image-Tag
strategy:
matrix:
java: [11, 17, 21]
os: [ubuntu-latest]

name: Pre-commit Linux
runs-on: ${{ matrix.os }}
container:
# using the same image which is used by opensearch-build team to build the OpenSearch Distribution
# this image tag is subject to change as more dependencies and updates will arrive over time
image: ${{ needs.Get-CI-Image-Tag.outputs.ci-image-version-linux }}
# need to switch to root so that github actions can install runner binary on container without permission issues.
options: --user root

steps:
- name: Checkout neural-search
uses: actions/checkout@v1

- name: Setup Java ${{ matrix.java }}
uses: actions/setup-java@v1
with:
java-version: ${{ matrix.java }}

- name: Run build
run: |
chown -R 1000:1000 `pwd`
su `id -un 1000` -c "./gradlew precommit --parallel"

- name: Upload Coverage Report
uses: codecov/codecov-action@v1
with:
token: ${{ secrets.CODECOV_TOKEN }}

Precommit-neural-search-windows:
strategy:
matrix:
java: [11, 17]
java: [11, 17, 21]
os: [windows-latest]

name: Pre-commit neural-search Plugin
runs-on: ubuntu-latest
name: Pre-commit Windows
runs-on: ${{ matrix.os }}

steps:
- name: Checkout neural-search
Expand Down
19 changes: 19 additions & 0 deletions .github/workflows/add-untriaged.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
name: Apply 'untriaged' label during issue lifecycle

on:
issues:
types: [opened, reopened, transferred]

jobs:
apply-label:
runs-on: ubuntu-latest
steps:
- uses: actions/github-script@v6
with:
script: |
github.rest.issues.addLabels({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
labels: ['untriaged']
})
28 changes: 28 additions & 0 deletions .github/workflows/auto-release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
name: Releases

on:
push:
tags:
- '*'

jobs:
build:
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- name: GitHub App token
id: github_app_token
uses: tibdex/[email protected]
with:
app_id: ${{ secrets.APP_ID }}
private_key: ${{ secrets.APP_PRIVATE_KEY }}
installation_id: 22958780
- name: Get tag
id: tag
uses: dawidd6/action-get-tag@v1
- uses: actions/checkout@v2
- uses: ncipollo/release-action@v1
with:
github_token: ${{ steps.github_app_token.outputs.token }}
bodyFile: release-notes/opensearch-neural-search.release-notes-${{steps.tag.outputs.tag}}.md
65 changes: 65 additions & 0 deletions .github/workflows/backwards_compatibility_tests_workflow.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
name: Backwards Compatibility Tests NeuralSearch
on:
push:
branches:
- "*"
- "feature/**"
pull_request:
branches:
- "*"
- "feature/**"

jobs:
Restart-Upgrade-BWCTests-NeuralSearch:
strategy:
matrix:
java: [ 11, 17, 21 ]
os: [ubuntu-latest,windows-latest]
bwc_version : ["2.9.0","2.10.0","2.11.0"]
opensearch_version : [ "2.12.0-SNAPSHOT" ]

name: NeuralSearch Restart-Upgrade BWC Tests
runs-on: ${{ matrix.os }}
env:
BWC_VERSION_RESTART_UPGRADE: ${{ matrix.bwc_version }}

steps:
- name: Checkout NeuralSearch
uses: actions/checkout@v1

- name: Setup Java ${{ matrix.java }}
uses: actions/setup-java@v1
with:
java-version: ${{ matrix.java }}

- name: Run NeuralSearch Restart-Upgrade BWC Tests from BWCVersion-${{ matrix.bwc_version }} to OpenSearch Version-${{ matrix.opensearch_version }} on ${{matrix.os}}
run: |
echo "Running restart-upgrade backwards compatibility tests ..."
./gradlew :qa:restart-upgrade:testAgainstNewCluster -D'tests.bwc.version=${{ matrix.bwc_version }}'

Rolling-Upgrade-BWCTests-NeuralSearch:
strategy:
matrix:
java: [ 11, 17, 21 ]
os: [ubuntu-latest,windows-latest]
bwc_version: [ "2.11.0" ]
opensearch_version: [ "2.12.0-SNAPSHOT" ]

name: NeuralSearch Rolling-Upgrade BWC Tests
runs-on: ${{ matrix.os }}
env:
BWC_VERSION_ROLLING_UPGRADE: ${{ matrix.bwc_version }}

steps:
- name: Checkout NeuralSearch
uses: actions/checkout@v1

- name: Setup Java ${{ matrix.java }}
uses: actions/setup-java@v1
with:
java-version: ${{ matrix.java }}

- name: Run NeuralSearch Rolling-Upgrade BWC Tests from BWCVersion-${{ matrix.bwc_version }} to OpenSearch Version-${{ matrix.opensearch_version }} on ${{matrix.os}}
run: |
echo "Running rolling-upgrade backwards compatibility tests ..."
./gradlew :qa:rolling-upgrade:testRollingUpgrade -D'tests.bwc.version=${{ matrix.bwc_version }}'
18 changes: 18 additions & 0 deletions .github/workflows/changelog_verifier.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
name: "Changelog Verifier"
on:
pull_request:
types: [opened, edited, review_requested, synchronize, reopened, ready_for_review, labeled, unlabeled]

jobs:
# Enforces the update of a changelog file on every pull request
verify-changelog:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
token: ${{ secrets.GITHUB_TOKEN }}
ref: ${{ github.event.pull_request.head.sha }}

- uses: dangoslen/changelog-enforcer@v3
with:
skipLabels: "autocut, skip-changelog"
21 changes: 21 additions & 0 deletions .github/workflows/copy-linked-issue-labels.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
name: Copy labels from linked issues
on:
pull_request_target:
types: [opened, edited, review_requested, synchronize, reopened, ready_for_review]

jobs:
copy-issue-labels:
if: github.repository == 'opensearch-project/neural-search'
runs-on: ubuntu-latest
permissions:
issues: read
contents: read
pull-requests: write
steps:
- name: copy-issue-labels
uses: michalvankodev/[email protected]
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
labels-to-exclude: |
untriaged
triaged
18 changes: 0 additions & 18 deletions .github/workflows/dco.yml

This file was deleted.

6 changes: 3 additions & 3 deletions .github/workflows/delete_backport_branch.yml
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
name: Delete merged branch of the backport PRs
on:
on:
pull_request:
types:
- closed

jobs:
delete-branch:
runs-on: ubuntu-latest
Expand All @@ -12,4 +12,4 @@ jobs:
- name: Delete merged branch
uses: SvanBoxel/delete-merged-branch@main
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
20 changes: 20 additions & 0 deletions .github/workflows/draft-release-notes-workflow.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
name: Release Drafter

on:
push:
branches:
- main

jobs:
update_release_draft:
name: Update draft release notes
runs-on: ubuntu-latest
steps:
- name: Update draft release notes
uses: release-drafter/release-drafter@v5
with:
config-name: draft-release-notes-config.yml
name: Version (set here)
tag: (None)
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
2 changes: 1 addition & 1 deletion .github/workflows/links.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,4 @@ jobs:
env:
GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}
- name: Fail if there were link errors
run: exit ${{ steps.lychee.outputs.exit_code }}
run: exit ${{ steps.lychee.outputs.exit_code }}
35 changes: 35 additions & 0 deletions .github/workflows/maven-publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: Publish snapshots to maven

on:
workflow_dispatch:
push:
branches:
- 'main'
- '1.*'
- '2.*'

jobs:
build-and-publish-snapshots:
runs-on: ubuntu-latest

permissions:
id-token: write
contents: write

steps:
- uses: actions/setup-java@v3
with:
distribution: temurin # Temurin is a distribution of adoptium
java-version: 11
- uses: actions/checkout@v3
- uses: aws-actions/configure-aws-credentials@v1
with:
role-to-assume: ${{ secrets.PUBLISH_SNAPSHOTS_ROLE }}
aws-region: us-east-1
- name: publish snapshots to maven
run: |
export SONATYPE_USERNAME=$(aws secretsmanager get-secret-value --secret-id maven-snapshots-username --query SecretString --output text)
export SONATYPE_PASSWORD=$(aws secretsmanager get-secret-value --secret-id maven-snapshots-password --query SecretString --output text)
echo "::add-mask::$SONATYPE_USERNAME"
echo "::add-mask::$SONATYPE_PASSWORD"
./gradlew publishPluginZipPublicationToSnapshotsRepository
Loading
Loading