[Refactor] remaining ImmutableOpenMap usage to j.u.Map and remove class #7309

nknize · 2023-04-26T20:07:12Z

Refactors all remaining usage of HPPC backed ImmutableOpenMap to j.u.Map. This is the last usage of the class so ImmutableOpenMap is also completely removed in favor of java.util.Map usage.

relates #5910

Refactors all remaining usage of HPPC backed ImmutableOpenMap to j.u.Map. This is the last usage of the class so ImmutableOpenMap is completely removed in favor of java maps. Signed-off-by: Nicholas Walter Knize <[email protected]>

github-actions · 2023-04-26T20:38:46Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards

URL: https://build.ci.opensearch.org/job/gradle-check/14576/
CommitID: 353c025
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov-commenter · 2023-04-26T20:44:30Z

Codecov Report

Merging #7309 (353c025) into main (7b8796a) will increase coverage by 0.21%.
The diff coverage is 92.19%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main    #7309      +/-   ##
============================================
+ Coverage     70.45%   70.67%   +0.21%     
- Complexity    59381    59513     +132     
============================================
  Files          4862     4859       -3     
  Lines        285489   285339     -150     
  Branches      41144    41133      -11     
============================================
+ Hits         201143   201653     +510     
+ Misses        67777    67166     -611     
+ Partials      16569    16520      -49

Impacted Files	Coverage Δ
...cluster/reroute/TransportClusterRerouteAction.java	`46.05% <0.00%> (+0.59%)`	⬆️
...ices/shards/TransportIndicesShardStoresAction.java	`4.90% <0.00%> (ø)`
...a/org/opensearch/common/io/stream/StreamInput.java	`88.80% <ø> (+0.42%)`	⬆️
.../org/opensearch/common/io/stream/StreamOutput.java	`94.80% <ø> (+0.71%)`	⬆️
.../src/main/java/org/opensearch/gateway/Gateway.java	`8.33% <0.00%> (ø)`
.../opensearch/rest/action/cat/RestIndicesAction.java	`46.03% <0.00%> (-0.12%)`	⬇️
...min/indices/shards/IndicesShardStoresResponse.java	`55.55% <71.42%> (ø)`
...dmin/indices/settings/get/GetSettingsResponse.java	`73.17% <91.66%> (-0.64%)`	⬇️
.../java/org/opensearch/index/mapper/FieldMapper.java	`80.75% <94.11%> (+0.01%)`	⬆️
...rch/action/admin/indices/get/GetIndexResponse.java	`74.84% <100.00%> (+2.45%)`	⬆️
... and 18 more

... and 469 files with indirect coverage changes

andrross

This refactoring has caused some issues for the plugins, but since we've already started it will be good to finish.

I'm wondering if it'll be less impactful on downstream dependencies to do the backport in a single commit that combines all the ImmutableOpenMap removal changes.

nknize · 2023-04-27T03:11:56Z

I'm wondering if it'll be less impactful on downstream dependencies to do the backport in a single commit that combines all the ImmutableOpenMap removal changes.

I'm not opposed that. Or I can take care of all of these backports pretty quickly myself.

….com/opensearch-project/OpenSearch/pull/7309 Signed-off-by: Joshua Palis <[email protected]>

…y and multi-entity detectors (#880) * Initial historical analysis commit. Registers ForwardADTaskTransportAction, ADBatchAnomalyResultTransportAction, replaces Internal Aggregation classes with corresponding Parsed Aggregation class. removes hashring from historical analysis workflow Signed-off-by: Joshua Palis <[email protected]> * Reverts changes to AnomalyResultBulkIndexHandler and uses the SDKRestClient to bulk index anomaly results, replaces transport service calls with corresponding client execute calls. Adds check for request content for job details registration, this prevents historical task requests from triggering job details registration Signed-off-by: Joshua Palis <[email protected]> * Adds another check for job details registration for start path only, adds transport action for stop historical detector Signed-off-by: Joshua Palis <[email protected]> * Fixing import error, test classes Signed-off-by: Joshua Palis <[email protected]> * spotlessApply Signed-off-by: Joshua Palis <[email protected]> * Commenting out parse bucket tests since ParsedAggregation classes have no constructor to access via reflection Signed-off-by: Joshua Palis <[email protected]> * Addressing PR comments Signed-off-by: Joshua Palis <[email protected]> * replaceing .valuesIt() with .values().iterator() due tohttps://github.com/opensearch-project/OpenSearch/pull/7309 Signed-off-by: Joshua Palis <[email protected]> * removing ImmutableOpenMap Signed-off-by: Joshua Palis <[email protected]> * Reverting dataGeneration script changes Signed-off-by: Joshua Palis <[email protected]> * fixing iterator Signed-off-by: Joshua Palis <[email protected]> * Fixing data nodes iterator Signed-off-by: Joshua Palis <[email protected]> * removing value from iterator Signed-off-by: Joshua Palis <[email protected]> --------- Signed-off-by: Joshua Palis <[email protected]>

…nsearch-project#7309) Refactors all remaining usage of HPPC backed ImmutableOpenMap to j.u.Map. This is the last usage of the class so ImmutableOpenMap is completely removed in favor of java maps. Signed-off-by: Nicholas Walter Knize <[email protected]>

soosinha · 2023-05-22T04:28:43Z

Due to this refactoring, we have seen improvement in heap memory usage. The MultiFields object in the FieldMapper class was modified to use Collections.UnmodifiableMap instead of ImmutableOpenMap. This reduced the retained memory usage of a multiFields object from 200 bytes to 64 bytes. When multiplying this gain by the number of fields and the number of indices in the cluster, we are observing considerable reduction in heap memory usage.

opensearch-trigger-bot · 2023-06-27T00:38:31Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-7309-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 d984f5028e77e281ed08386d43de4e64e06f2bf8
# Push it to GitHub
git push --set-upstream origin backport/backport-7309-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-7309-to-2.x.

…nsearch-project#7309) Refactors all remaining usage of HPPC backed ImmutableOpenMap to j.u.Map. This is the last usage of the class so ImmutableOpenMap is completely removed in favor of java maps. Signed-off-by: Nicholas Walter Knize <[email protected]> (cherry picked from commit d984f50)

…) (#8316) Refactors all remaining usage of HPPC backed ImmutableOpenMap to j.u.Map. This is the last usage of the class so ImmutableOpenMap is completely removed in favor of java maps. Signed-off-by: Nicholas Walter Knize <[email protected]> (cherry picked from commit d984f50)

…nsearch-project#7309) Refactors all remaining usage of HPPC backed ImmutableOpenMap to j.u.Map. This is the last usage of the class so ImmutableOpenMap is completely removed in favor of java maps. Signed-off-by: Nicholas Walter Knize <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

nknize added enhancement Enhancement or improvement to existing feature or request skip-changelog v2.8.0 'Issues and PRs related to version v2.8.0' labels Apr 26, 2023

nknize requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, owaiskazi19, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89, tlfeng, VachaShah and xuezhou25 as code owners April 26, 2023 20:07

nknize changed the title ~~[Refactor] Refactors remaining ImmutableOpenMap usage to j.u.Map~~ [Refactor] remaining ImmutableOpenMap usage to j.u.Map and remove class Apr 26, 2023

andrross approved these changes Apr 26, 2023

View reviewed changes

andrross merged commit d984f50 into opensearch-project:main Apr 26, 2023

MaxKsyunz mentioned this pull request Apr 27, 2023

Keep up with refactoring in OpenSearch. opensearch-project/sql#1588

Merged

joshpalis added a commit to joshpalis/anomaly-detection that referenced this pull request Apr 27, 2023

replaceing .valuesIt() with .values().iterator() due tohttps://github…

6bfaad5

….com/opensearch-project/OpenSearch/pull/7309 Signed-off-by: Joshua Palis <[email protected]>

Tjofil mentioned this pull request Apr 28, 2023

Upgrade Gradle, Gradle test-retry and change ImmutableOpenMap functions to J U Map's opensearch-project/performance-analyzer#438

Merged

5 tasks

This was referenced May 2, 2023

Move ImmutableOpenMap usage to j.u.Map and upgrade gradle to 7.6.1 opensearch-project/k-NN#885

Merged

Move old manifests to legacy opensearch-project/opensearch-build#3473

Merged

b4sjoo mentioned this pull request May 11, 2023

[BUG] ML-Commons Plugin build failure due to OpenSearch Core breaking changes opensearch-project/ml-commons#891

Closed

nknize added the backport 2.x Backport to 2.x branch label Jun 27, 2023

nknize mentioned this pull request Jun 28, 2023

[Backport 2.x] [Refactor] Refactors remaining ImmutableOpenMap usage to j.u.Map (#7309) #8316

Merged

nknize added v2.9.0 'Issues and PRs related to version v2.9.0' and removed v2.8.0 'Issues and PRs related to version v2.8.0' labels Jun 28, 2023

b4sjoo mentioned this pull request Jul 7, 2023

Backport core String lib refactor fix PR and migrate all ImmutableMap with j.u.map opensearch-project/ml-commons#1045

Closed

5 tasks

monusingh-1 mentioned this pull request Jul 11, 2023

Increment version to 2.9 and refactor opensearch-project/cross-cluster-replication#1061

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Refactor] remaining ImmutableOpenMap usage to j.u.Map and remove class #7309

[Refactor] remaining ImmutableOpenMap usage to j.u.Map and remove class #7309

nknize commented Apr 26, 2023 •

edited

Loading

github-actions bot commented Apr 26, 2023

codecov-commenter commented Apr 26, 2023

andrross left a comment

nknize commented Apr 27, 2023

soosinha commented May 22, 2023

opensearch-trigger-bot bot commented Jun 27, 2023

[Refactor] remaining ImmutableOpenMap usage to j.u.Map and remove class #7309

[Refactor] remaining ImmutableOpenMap usage to j.u.Map and remove class #7309

Conversation

nknize commented Apr 26, 2023 • edited Loading

github-actions bot commented Apr 26, 2023

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented Apr 26, 2023

Codecov Report

andrross left a comment

Choose a reason for hiding this comment

nknize commented Apr 27, 2023

soosinha commented May 22, 2023

opensearch-trigger-bot bot commented Jun 27, 2023

nknize commented Apr 26, 2023 •

edited

Loading