Remove commons-collections 3.2.2 #2924

willyborankin · 2023-07-02T09:43:00Z

Description

Removed commons-collections 3.2.2
Reasons:

it is not supported anymore
there were only 2 places where it used:
- Class LogCapturingAppender - replaced CircularFifoBuffer with guava EvictingQueue
- ResolvedIndicesProvider - replace MultyKey with the custom one

Issues Resolved

Testing

[Please provide details of testing done: unit testing, integration testing and manual testing]

Check List

New functionality includes testing
New functionality has been documented
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov · 2023-07-02T09:45:24Z

Codecov Report

Merging #2924 (59e721b) into main (092e8f5) will decrease coverage by 3.46%.
The diff coverage is 75.00%.

@@             Coverage Diff              @@
##               main    #2924      +/-   ##
============================================
- Coverage     62.29%   58.83%   -3.46%     
+ Complexity     3337     3088     -249     
============================================
  Files           266      266              
  Lines         19650    19668      +18     
  Branches       3329     3334       +5     
============================================
- Hits          12240    11571     -669     
- Misses         5783     6437     +654     
- Partials       1627     1660      +33

Impacted Files	Coverage Δ
...earch/security/resolver/IndexResolverReplacer.java	`68.29% <75.00%> (+0.20%)`	⬆️

... and 47 files with indirect coverage changes

willyborankin · 2023-07-02T11:04:37Z

src/main/java/org/opensearch/security/resolver/IndexResolverReplacer.java

@@ -158,9 +158,49 @@ private class ResolvedIndicesProvider implements IndicesProvider {
        private final ImmutableSet.Builder<String> remoteIndices;
        // set of previously resolved index requests to avoid resolving
        // the same index more than once while processing bulk requests
-        private final Set<MultiKey> alreadyResolved;
+        private final Set<AlreadyResolvedKey> alreadyResolved;


Not sure. But using unbounded collection could lead to a memory leak. So a question to those who knows this part better than me. Is it possible?

If we wanted to prevent the chance of a memory leak entirely, I believe we would need to cap the size of the set. The problem is determining an appropriate cap for the size. I believe the set should be cleared with each request so it should not persist for too long, so the issue would only be that we ran into a memory issue in a single bulk request. For determining the size to cap it at, we could look at the expected size of each AlreadyResolvedKey object and then determine how much memory we want to allow the task to allocate. Obviously, we will reach issues at certain extremes unless we can start to calculate based off of node resources.

stephen-crawford

Overall looks good to me. I just left one question about the use of the CCR parameter.

stephen-crawford · 2023-07-03T03:03:39Z

src/main/java/org/opensearch/security/resolver/IndexResolverReplacer.java

@@ -158,9 +158,49 @@ private class ResolvedIndicesProvider implements IndicesProvider {
        private final ImmutableSet.Builder<String> remoteIndices;
        // set of previously resolved index requests to avoid resolving
        // the same index more than once while processing bulk requests
-        private final Set<MultiKey> alreadyResolved;
+        private final Set<AlreadyResolvedKey> alreadyResolved;


If we wanted to prevent the chance of a memory leak entirely, I believe we would need to cap the size of the set. The problem is determining an appropriate cap for the size. I believe the set should be cleared with each request so it should not persist for too long, so the issue would only be that we ran into a memory issue in a single bulk request. For determining the size to cap it at, we could look at the expected size of each AlreadyResolvedKey object and then determine how much memory we want to allow the task to allocate. Obviously, we will reach issues at certain extremes unless we can start to calculate based off of node resources.

stephen-crawford · 2023-07-03T03:07:49Z

src/main/java/org/opensearch/security/resolver/IndexResolverReplacer.java

+
+            private final IndicesOptions indicesOptions;
+
+            private final boolean enableCrossClusterResolution;


Sorry if I am missing something, but how is this field being used? Is this part of the bulk request logic in some way? I am not particularly familiar with this--are we using this field to differentiate between two instances of the same index operation but with different CCR permissions?

enableCrossClusterResolution determines here:

security/src/main/java/org/opensearch/security/resolver/IndexResolverReplacer.java

Line 375 in 1089bd6

final boolean enableCrossClusterResolution = localRequest instanceof FieldCapabilitiesRequest

.
I played with it a bit. If I remove it all tests pass except PIT tests. TBH did not investigate why.

No worries.

cwperks

This change looks good to me.

cwperks · 2023-07-03T12:54:17Z

src/main/java/org/opensearch/security/resolver/IndexResolverReplacer.java

@@ -336,9 +376,13 @@ public String[] provide(String[] original, Object localRequest, boolean supports
                || localRequest instanceof SearchRequest
                || localRequest instanceof ResolveIndexAction.Request;
            // skip the whole thing if we have seen this exact resolveIndexPatterns request
-            if (alreadyResolved.add(


This change looks fine to me. I was also looking at where alreadyResolved was used and it looks like this line is primarily why a MultiKey was chosen. IMO the logic utilizing MultiKey was hard to follow and this is clearer.

cwperks · 2023-07-07T14:57:22Z

@willyborankin Can you resolve the conflict in this PR?

willyborankin · 2023-07-07T15:08:46Z

@willyborankin Can you resolve the conflict in this PR?

sure

Signed-off-by: Andrey Pleskach <[email protected]>

opensearch-trigger-bot · 2023-07-07T16:21:43Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-2924-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 49cbf520a9f45ab5e975dfe0b7c9ff489a57c338
# Push it to GitHub
git push --set-upstream origin backport/backport-2924-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-2924-to-2.x.

Signed-off-by: Andrey Pleskach <[email protected]> (cherry picked from commit 49cbf52) Signed-off-by: Andrey Pleskach <[email protected]>

(cherry picked from commit 49cbf52) Signed-off-by: Andrey Pleskach <[email protected]>

This change combines the many updates from the following commits: * 5f62e8a dependabot: bump commons-io:commons-io from 2.11.0 to 2.13.0 (opensearch-project#3074) * 2f69a10 bump com.github.wnameless.json:json-base from 2.4.0 to 2.4.1 (opensearch-project#3062) * c0e50da dependabot: bump org.cryptacular:cryptacular from 1.2.4 to 1.2.5 (opensearch-project#3071) * d3488e8 dependabot: bump kafka_version from 3.5.0 to 3.5.1 (opensearch-project#3041) * ab6778d Update ospackage, checker-qual, zcxvbn and error_prone_annotations, camel-xmlsecurity (opensearch-project#3023) * 0e6608d Bump JSON libs (opensearch-project#2926) * df07bea SAML 4.3.0 addition persmission (opensearch-project#2987) * e5348eb Change maven repo location for compatibility check (opensearch-project#2980) * 4a1ec53 Bump jaxb to 2.3.8 (opensearch-project#2977) * 9599155 Bump guava to 32.1.1-jre (opensearch-project#2976) * 06eed60 dependabot: bump org.glassfish.jaxb:jaxb-runtime from 2.3.4 to 4.0.3 (opensearch-project#2970) * 1113244 Bump eventbus to 3.3.1 (opensearch-project#2965) * 99ff7b3 dependabot: bump org.apache.bcel:bcel from 6.6.0 to 6.7.0 (opensearch-project#2969) * 0794c3f dependabot: bump jakarta.xml.bind:jakarta.xml.bind-api (opensearch-project#2968) * 9e6aab3 dependabot: bump com.google.j2objc:j2objc-annotations from 1.3 to 2.8 (opensearch-project#2963) * 8227f64 dependabot: bump com.sun.istack:istack-commons-runtime (opensearch-project#2960) * 8e044a6 dependabot: bump org.apiguardian:apiguardian-api from 1.0.0 to 1.1.2 (opensearch-project#2964) * 49cbf52 Remove commons-collections 3.2.2 (opensearch-project#2924) * 092e8f5 Bump SAML libs (opensearch-project#2927) * 8ab7cb4 Resolve CVE-2023-2976 by forcing use of Guava 32.0.1 (opensearch-project#2937) * 4eef662 Clean up and bump Apache libs (opensearch-project#2925) * 9a72355 Bump BouncyCastle from jdk15on to jdk15to18 (opensearch-project#2901) * e4f4817 [Enhancement] Parallel test jobs for CI (opensearch-project#2861) * d871af3 Update snappy to 1.1.10.1 and guava to 32.0.1-jre (opensearch-project#2886) * c808692 Format everything (opensearch-project#2866) Signed-off-by: Peter Nied <[email protected]>

willyborankin force-pushed the exclude-collections branch from e3b5f98 to 4353b02 Compare July 2, 2023 11:02

willyborankin commented Jul 2, 2023

View reviewed changes

willyborankin marked this pull request as ready for review July 2, 2023 11:04

willyborankin requested review from cliu123, cwperks, DarshitChanpura, davidlago, peternied, RyanL1997, stephen-crawford and reta as code owners July 2, 2023 11:04

stephen-crawford reviewed Jul 3, 2023

View reviewed changes

cwperks previously approved these changes Jul 3, 2023

View reviewed changes

cwperks added the backport 2.x backport to 2.x branch label Jul 3, 2023

willyborankin dismissed cwperks’s stale review via 1089bd6 July 3, 2023 19:06

willyborankin force-pushed the exclude-collections branch from 4353b02 to 1089bd6 Compare July 3, 2023 19:06

stephen-crawford previously approved these changes Jul 5, 2023

View reviewed changes

cwperks previously approved these changes Jul 7, 2023

View reviewed changes

Remove commons-collections 3.2.2

59e721b

Signed-off-by: Andrey Pleskach <[email protected]>

willyborankin dismissed stale reviews from cwperks and stephen-crawford via 59e721b July 7, 2023 15:14

willyborankin force-pushed the exclude-collections branch from 1089bd6 to 59e721b Compare July 7, 2023 15:14

cwperks approved these changes Jul 7, 2023

View reviewed changes

stephen-crawford approved these changes Jul 7, 2023

View reviewed changes

stephen-crawford merged commit 49cbf52 into opensearch-project:main Jul 7, 2023

willyborankin mentioned this pull request Jul 8, 2023

[Backport 2.x] Remove commons-collections 3.2.2 (#2924) #2957

Merged

cwperks pushed a commit that referenced this pull request Jul 10, 2023

Remove commons-collections 3.2.2 (#2924) (#2957)

58a7e20

(cherry picked from commit 49cbf52) Signed-off-by: Andrey Pleskach <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove commons-collections 3.2.2 #2924

Remove commons-collections 3.2.2 #2924

willyborankin commented Jul 2, 2023 •

edited

Loading

codecov bot commented Jul 2, 2023 •

edited

Loading

willyborankin Jul 2, 2023 •

edited

Loading

stephen-crawford Jul 3, 2023

stephen-crawford left a comment

stephen-crawford Jul 3, 2023

stephen-crawford Jul 3, 2023

willyborankin Jul 4, 2023 •

edited

Loading

stephen-crawford Jul 5, 2023

cwperks left a comment

cwperks Jul 3, 2023

cwperks commented Jul 7, 2023

willyborankin commented Jul 7, 2023

opensearch-trigger-bot bot commented Jul 7, 2023


		private final IndicesOptions indicesOptions;

		private final boolean enableCrossClusterResolution;

Remove commons-collections 3.2.2 #2924

Remove commons-collections 3.2.2 #2924

Conversation

willyborankin commented Jul 2, 2023 • edited Loading

Description

Issues Resolved

Testing

Check List

codecov bot commented Jul 2, 2023 • edited Loading

Codecov Report

willyborankin Jul 2, 2023 • edited Loading

Choose a reason for hiding this comment

stephen-crawford Jul 3, 2023

Choose a reason for hiding this comment

stephen-crawford left a comment

Choose a reason for hiding this comment

stephen-crawford Jul 3, 2023

Choose a reason for hiding this comment

stephen-crawford Jul 3, 2023

Choose a reason for hiding this comment

willyborankin Jul 4, 2023 • edited Loading

Choose a reason for hiding this comment

stephen-crawford Jul 5, 2023

Choose a reason for hiding this comment

cwperks left a comment

Choose a reason for hiding this comment

cwperks Jul 3, 2023

Choose a reason for hiding this comment

cwperks commented Jul 7, 2023

willyborankin commented Jul 7, 2023

opensearch-trigger-bot bot commented Jul 7, 2023

willyborankin commented Jul 2, 2023 •

edited

Loading

codecov bot commented Jul 2, 2023 •

edited

Loading

willyborankin Jul 2, 2023 •

edited

Loading

willyborankin Jul 4, 2023 •

edited

Loading