Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Upgrade] Lucene 9.0.0 release #1109

Merged
merged 11 commits into from
Mar 15, 2022
Merged

Conversation

nknize
Copy link
Collaborator

@nknize nknize commented Aug 18, 2021

This PR upgrades the core codebase from Lucene 8.10.1 to lucene-9.0.0 release. It includes all necessary refactoring of features and API changes when upgrading to the new major Lucene release.

relates #1067
depends on #2452

@nknize nknize added v2.0.0 Version 2.0.0 WIP Work in progress >upgrade Label used when upgrading library dependencies (e.g., Lucene) labels Aug 18, 2021
@nknize nknize marked this pull request as draft August 18, 2021 17:12
@nknize
Copy link
Collaborator Author

nknize commented Aug 18, 2021

TODO:

  • remove all deprecated features from Legacy version 6.x (will open a meta issue to track)

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Wrapper Validation success 52d7adf3cf035003e921e35de7deb72c03c6b9bc

@opensearch-ci-bot
Copy link
Collaborator

✅   DCO Check Passed 52d7adf3cf035003e921e35de7deb72c03c6b9bc

@opensearch-ci-bot
Copy link
Collaborator

❌   Gradle Precommit failure 52d7adf3cf035003e921e35de7deb72c03c6b9bc
Log 960

@opensearch-ci-bot
Copy link
Collaborator

Can one of the admins verify this patch?

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Wrapper Validation success 52d7adf3cf035003e921e35de7deb72c03c6b9bc

@opensearch-ci-bot
Copy link
Collaborator

✅   DCO Check Passed 52d7adf3cf035003e921e35de7deb72c03c6b9bc

@opensearch-ci-bot
Copy link
Collaborator

❌   Gradle Precommit failure 52d7adf3cf035003e921e35de7deb72c03c6b9bc
Log 1411

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Wrapper Validation success c40161ecbcc40b9ed213028438e42f5c91d38113

@opensearch-ci-bot
Copy link
Collaborator

❌   Gradle Check failure c40161ecbcc40b9ed213028438e42f5c91d38113
Log 1280

Reports 1280

@opensearch-ci-bot
Copy link
Collaborator

❌   Gradle Precommit failure c40161ecbcc40b9ed213028438e42f5c91d38113
Log 1669

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Wrapper Validation success 9a897b4ff762b4e92ba38b9baa33854f0414fda2

@opensearch-ci-bot
Copy link
Collaborator

❌   Gradle Precommit failure 9a897b4ff762b4e92ba38b9baa33854f0414fda2
Log 1677

@opensearch-ci-bot
Copy link
Collaborator

❌   Gradle Check failure 9a897b4ff762b4e92ba38b9baa33854f0414fda2
Log 1288

Reports 1288

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Wrapper Validation success f94754a4c79c8b01a12a30ea1b5142971b9d887b

@opensearch-ci-bot
Copy link
Collaborator

❌   Gradle Check failure f94754a4c79c8b01a12a30ea1b5142971b9d887b
Log 1304

Reports 1304

@opensearch-ci-bot
Copy link
Collaborator

❌   Gradle Precommit failure f94754a4c79c8b01a12a30ea1b5142971b9d887b
Log 1685

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Wrapper Validation success 65dae2e281d749ccb38e1f8aaf5f970b242a37c5

@opensearch-ci-bot
Copy link
Collaborator

❌   Gradle Precommit failure 65dae2e281d749ccb38e1f8aaf5f970b242a37c5
Log 1686

@opensearch-ci-bot
Copy link
Collaborator

❌   Gradle Check failure 65dae2e281d749ccb38e1f8aaf5f970b242a37c5
Log 1305

Reports 1305

@rishabhmaurya rishabhmaurya mentioned this pull request Jan 5, 2022
7 tasks
@nknize nknize changed the title [Upgrade] Lucene 9.0.0-snapshot-803d131fd08 [Upgrade] Lucene 9.0.0 release Jan 10, 2022
@nknize nknize marked this pull request as ready for review January 10, 2022 21:05
@nknize nknize removed the WIP Work in progress label Mar 14, 2022
Copy link
Member

@dblock dblock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did a read, but it's a bit over my head given I know little about Lucene. I think we should merge it as is (feel free to update the small stuff before/after) and iterate for anything that falls out.

@@ -154,13 +154,6 @@ protected void addCustomXContentFields(XContentBuilder builder, Params params) t
if (segment.getSegmentSort() != null) {
toXContent(builder, segment.getSegmentSort());
}
if (segment.ramTree != null) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Explain?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ramEstimator was removed from Lucene 9.. so we remove it from the segment stats.

@reta
Copy link
Collaborator

reta commented Mar 15, 2022

@nknize sorry, DCO check failed for some reasons :( could you please amend the last commit? thank you

@opensearch-ci-bot
Copy link
Collaborator

❌   Gradle Check failure 636db0caafa49a4b9317831b33795cce149696ea
Log 3414

Reports 3414

nknize and others added 9 commits March 15, 2022 14:21
Removes types from UID class along with cleaning up all obsolete MapperService
dependendices. This includes removing UID from the Translog Delete operation
which is no longer needed due to type dependency removal.

Signed-off-by: Nicholas Walter Knize <[email protected]>
Signed-off-by: Nicholas Walter Knize <[email protected]>
This commit upgrades the core codebase from Lucene 8.9 to
lucene-9.0.0. It includes all necessary refactoring of features and
API changes when upgrading to a new major Lucene release.

Signed-off-by: Nicholas Walter Knize <[email protected]>
Signed-off-by: Andriy Redko <[email protected]>
Signed-off-by: Nicholas Walter Knize <[email protected]>
Signed-off-by: Andriy Redko <[email protected]>
Signed-off-by: Nicholas Walter Knize <[email protected]>
Signed-off-by: Nicholas Walter Knize <[email protected]>
Signed-off-by: Andriy Redko <[email protected]>
Signed-off-by: Nicholas Walter Knize <[email protected]>
Signed-off-by: Nicholas Walter Knize <[email protected]>
This reverts commit e7af399.

Signed-off-by: Nicholas Walter Knize <[email protected]>
Signed-off-by: Nicholas Walter Knize <[email protected]>
@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Check success 0add86fc1c87a2876266c6b63e5e7be2f7197b95
Log 3415

Reports 3415

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Check success f080199
Log 3416

Reports 3416

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Check success 451a409
Log 3417

Reports 3417

@nknize nknize merged commit 006c832 into opensearch-project:main Mar 15, 2022
@nknize nknize mentioned this pull request Mar 21, 2022
2 tasks
Yury-Fridlyand added a commit to Bit-Quill/opensearch-net that referenced this pull request Jun 9, 2022
`OpenSearch` 2.0 uses newer version of `Lucene` (9.0) which doesn't provide segments stats info.

Ref: opensearch-project/OpenSearch#2029 opensearch-project/OpenSearch#1109
See also history for `server/src/main/java/org/opensearch/index/engine/SegmentsStats.java` in `OpenSearch` repo.

Signed-off-by: Yury Fridlyand <[email protected]>
Yury-Fridlyand added a commit to Bit-Quill/opensearch-net that referenced this pull request Aug 29, 2022
* Update nuget packages (opensearch-project#88)
* Cleanup - removing stale dependencies.
* Fix 2 typos in scripts project
* Update `scripts` project file to include all relevant objects.
* Rename `master` node role to `cluster_manager` as it was done in OpenSearch.
    Ref: opensearch-project/OpenSearch#2480
* Remove validation for indices segments stats.
    `OpenSearch` 2.0 uses newer version of `Lucene` (9.0) which doesn't provide segments stats info.
    Ref: opensearch-project/OpenSearch#2029 opensearch-project/OpenSearch#1109
    See also history for `server/src/main/java/org/opensearch/index/engine/SegmentsStats.java` in `OpenSearch` repo.
* Remove tests for `_type` validation in mapping APIs as it was removed from `OpenSearch`.
    Ref: opensearch-project/OpenSearch#2238 opensearch-project/OpenSearch#2480
* Remove usage of deprecated `search.remote` settings.
    Ref: opensearch-project/OpenSearch#1870
* Update abstractions package - patch to support OpenSearch 2.0. Update integration workflow to run tests on OpenSearch 2.0.
* Rename `master_timeout` to `cluster_manager_timeout` in all APIs where it is used.
* Enrich comments to already renamed `CatMaster`/`CatClusterManager` API.
* Rename in `/_cluster/stats`/`cluster.stats` and `/_cluster/state`/`cluster.state`.
* Add deprecation info.
* Rename in comments.
* Rename in test data.
* Renamings in tests including `MasterEligible`, but mark it is obsolete.
* Rename branch reference in scripting.
* Mark `indices.exists_type`/`TypeExists` APIs as deprecated.
* Update compatibility matrix and include it into `sln` file.
* Add deprecation notice to all reference of `include_type_name`/`IncludeTypeName`.
* Update compatibility matrix.
* Remove `OpenDistro` compatibility notice.
* Update repo link.
* Add small README for each project being released.
* Address PR opensearch-project#51 feedback.

Signed-off-by: Yury-Fridlyand <[email protected]>
wbeckler pushed a commit to opensearch-project/opensearch-net that referenced this pull request Aug 31, 2022
* Update nuget packages (#88)
* Cleanup - removing stale dependencies.
* Fix 2 typos in scripts project
* Update `scripts` project file to include all relevant objects.
* Rename `master` node role to `cluster_manager` as it was done in OpenSearch.
    Ref: opensearch-project/OpenSearch#2480
* Remove validation for indices segments stats.
    `OpenSearch` 2.0 uses newer version of `Lucene` (9.0) which doesn't provide segments stats info.
    Ref: opensearch-project/OpenSearch#2029 opensearch-project/OpenSearch#1109
    See also history for `server/src/main/java/org/opensearch/index/engine/SegmentsStats.java` in `OpenSearch` repo.
* Remove tests for `_type` validation in mapping APIs as it was removed from `OpenSearch`.
    Ref: opensearch-project/OpenSearch#2238 opensearch-project/OpenSearch#2480
* Remove usage of deprecated `search.remote` settings.
    Ref: opensearch-project/OpenSearch#1870
* Update abstractions package - patch to support OpenSearch 2.0. Update integration workflow to run tests on OpenSearch 2.0.
* Rename `master_timeout` to `cluster_manager_timeout` in all APIs where it is used.
* Enrich comments to already renamed `CatMaster`/`CatClusterManager` API.
* Rename in `/_cluster/stats`/`cluster.stats` and `/_cluster/state`/`cluster.state`.
* Add deprecation info.
* Rename in comments.
* Rename in test data.
* Renamings in tests including `MasterEligible`, but mark it is obsolete.
* Rename branch reference in scripting.
* Mark `indices.exists_type`/`TypeExists` APIs as deprecated.
* Update compatibility matrix and include it into `sln` file.
* Add deprecation notice to all reference of `include_type_name`/`IncludeTypeName`.
* Update compatibility matrix.
* Remove `OpenDistro` compatibility notice.
* Update repo link.
* Add small README for each project being released.
* Address PR #51 feedback.

Signed-off-by: Yury-Fridlyand <[email protected]>

Signed-off-by: Yury-Fridlyand <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>upgrade Label used when upgrading library dependencies (e.g., Lucene) v2.0.0 Version 2.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants