[BUG] java.util.ConcurrentModificationException with plugins.security.cache.ttl_minutes=0 #2263
Assignees
Comments
olddanmer
added
bug
|
Thanks for filing @olddanmer thanks for the great details on this issue. |
RyanL1997
added
triaged
|
[Triaged] Thanks for filing this issue. We will take look into it. |
RyanL1997
added
untriaged
|
[Triage] Hi @RyanL1997, thank you for bringing this back to our collective attention. If you could try to reproduce this issue on the newer versions, could you please follow-up with what you find? Thank you. |
stephen-crawford
added
triaged
|
Hi @olddanmer, sorry for the late reply. I have tried to reproduce this issue and here are some findings from my end: For this one:
I think this original issue runs into that error because of the mis-configuring of Here are some specs:
@cwperks , could you please help me to verify that if you can/cannot reproduce this issue? Thanks. |
|
@RyanL1997 I was looking at this this morning and was just about to give up trying to reproduce, but eventually I was able to reproduce this locally. I reproduced this using the release candidate build for 2.8 so I have confirmed that it affects the latest version. My setup locally is the demo configuration with the I was able to reproduce this by writing a python script and using the bulk API: It took a few minutes after running to see the error. |
|
I haven't tested it out, but one thing I would try is checking if the set of security roles already contains a role before adding it to the set in this block: https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/user/User.java#L275-L284 It would be helpful to understand why the |
|
I have the same issue with opensearch 2.6.0. It occurs on all our environments, with irregular timing, but about every 3 minutes on average. Logs are similar to the v2.3.0 logs above, but the exception is thrown from a different place in PriviiegesEvaluator, if that offers you any further insight (stack trace below) Our /usr/share/opensearch/config/opensearch.yml is: |
|
Hi guys, thank you for providing all the details. I have successfully reproduce the error by following @cwperks's procedures: 
|
|
In testing, one of the primary places this issue arises is here in A potential solution could be to wrap the |
|
Attach to @cwperks's comment and also move some of offline discussions over here: The line of |
|
@RyanL1997 Exactly. With the concurrent modification imagine this: This is the scenario where concurrent modification can happen. |
3 tasks
3 tasks
opensearch-trigger-bot bot
pushed a commit
that referenced
this issue
Aug 7, 2023
…esolve ConcurrentModificationException on bulk request (#3094) ### Description This PR changes the order of `setUserInfoInThreadContext` and `addSecurityRoles` to ensure that `addSecurityRoles` is called before `setUserInfoInThreadContext`. The ConcurrentModificationException would arise in scenarios where the first thread was done with `setUserInfoInThreadContext` and had moved onto `addSecurityRoles` to add the mapped roles into the user's set of security roles. On the first call to `addSecurityRoles` it may be populating the set with data, any subsequent call would be a noop in other threads. If simultaneously there is another thread executing `setUserInfoInThreadContext` while the first thread is in `addSecurityRoles` then a ConcurrentModificationException is thrown inside the `Sets.union(...)` call. By calling `addSecurityRoles` before `setUserInfoInThreadContext`, it can be guaranteed that no ConcurrentModificationException could be thrown because the user's security roles will already be set and any thread that attempts another call will be a noop. * Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation) Bug fix ### Issues Resolved #2263 ### Testing Tested by running a python script to bulk insert 100k documents and using tmux to run the script in multiple shells at once. Before the change the bug is reproducible regularly, but after the change the bug cannot be reproduced. ### Check List - [ ] New functionality includes testing - [ ] New functionality has been documented - [ ] Commits are signed per the DCO using --signoff By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin). --------- Signed-off-by: Craig Perkins <[email protected]> Signed-off-by: Craig Perkins <[email protected]> (cherry picked from commit cd699bb) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
stephen-crawford
pushed a commit
to stephen-crawford/security
that referenced
this issue
Aug 11, 2023
…esolve ConcurrentModificationException on bulk request (opensearch-project#3094) ### Description This PR changes the order of `setUserInfoInThreadContext` and `addSecurityRoles` to ensure that `addSecurityRoles` is called before `setUserInfoInThreadContext`. The ConcurrentModificationException would arise in scenarios where the first thread was done with `setUserInfoInThreadContext` and had moved onto `addSecurityRoles` to add the mapped roles into the user's set of security roles. On the first call to `addSecurityRoles` it may be populating the set with data, any subsequent call would be a noop in other threads. If simultaneously there is another thread executing `setUserInfoInThreadContext` while the first thread is in `addSecurityRoles` then a ConcurrentModificationException is thrown inside the `Sets.union(...)` call. By calling `addSecurityRoles` before `setUserInfoInThreadContext`, it can be guaranteed that no ConcurrentModificationException could be thrown because the user's security roles will already be set and any thread that attempts another call will be a noop. * Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation) Bug fix ### Issues Resolved opensearch-project#2263 ### Testing Tested by running a python script to bulk insert 100k documents and using tmux to run the script in multiple shells at once. Before the change the bug is reproducible regularly, but after the change the bug cannot be reproduced. ### Check List - [ ] New functionality includes testing - [ ] New functionality has been documented - [ ] Commits are signed per the DCO using --signoff By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin). --------- Signed-off-by: Craig Perkins <[email protected]> Signed-off-by: Craig Perkins <[email protected]>
opensearch-trigger-bot bot
pushed a commit
that referenced
this issue
Aug 14, 2023
…esolve ConcurrentModificationException on bulk request (#3094) ### Description This PR changes the order of `setUserInfoInThreadContext` and `addSecurityRoles` to ensure that `addSecurityRoles` is called before `setUserInfoInThreadContext`. The ConcurrentModificationException would arise in scenarios where the first thread was done with `setUserInfoInThreadContext` and had moved onto `addSecurityRoles` to add the mapped roles into the user's set of security roles. On the first call to `addSecurityRoles` it may be populating the set with data, any subsequent call would be a noop in other threads. If simultaneously there is another thread executing `setUserInfoInThreadContext` while the first thread is in `addSecurityRoles` then a ConcurrentModificationException is thrown inside the `Sets.union(...)` call. By calling `addSecurityRoles` before `setUserInfoInThreadContext`, it can be guaranteed that no ConcurrentModificationException could be thrown because the user's security roles will already be set and any thread that attempts another call will be a noop. * Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation) Bug fix ### Issues Resolved #2263 ### Testing Tested by running a python script to bulk insert 100k documents and using tmux to run the script in multiple shells at once. Before the change the bug is reproducible regularly, but after the change the bug cannot be reproduced. ### Check List - [ ] New functionality includes testing - [ ] New functionality has been documented - [ ] Commits are signed per the DCO using --signoff By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin). --------- Signed-off-by: Craig Perkins <[email protected]> Signed-off-by: Craig Perkins <[email protected]> (cherry picked from commit cd699bb) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
cwperks
added a commit
to cwperks/security
that referenced
this issue
Aug 16, 2023
…esolve ConcurrentModificationException on bulk request (opensearch-project#3094) This PR changes the order of `setUserInfoInThreadContext` and `addSecurityRoles` to ensure that `addSecurityRoles` is called before `setUserInfoInThreadContext`. The ConcurrentModificationException would arise in scenarios where the first thread was done with `setUserInfoInThreadContext` and had moved onto `addSecurityRoles` to add the mapped roles into the user's set of security roles. On the first call to `addSecurityRoles` it may be populating the set with data, any subsequent call would be a noop in other threads. If simultaneously there is another thread executing `setUserInfoInThreadContext` while the first thread is in `addSecurityRoles` then a ConcurrentModificationException is thrown inside the `Sets.union(...)` call. By calling `addSecurityRoles` before `setUserInfoInThreadContext`, it can be guaranteed that no ConcurrentModificationException could be thrown because the user's security roles will already be set and any thread that attempts another call will be a noop. * Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation) Bug fix opensearch-project#2263 Tested by running a python script to bulk insert 100k documents and using tmux to run the script in multiple shells at once. Before the change the bug is reproducible regularly, but after the change the bug cannot be reproduced. - [ ] New functionality includes testing - [ ] New functionality has been documented - [ ] Commits are signed per the DCO using --signoff By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin). --------- Signed-off-by: Craig Perkins <[email protected]> Signed-off-by: Craig Perkins <[email protected]> (cherry picked from commit cd699bb)
cwperks
added a commit
to cwperks/security
that referenced
this issue
Aug 16, 2023
…esolve ConcurrentModificationException on bulk request (opensearch-project#3094) This PR changes the order of `setUserInfoInThreadContext` and `addSecurityRoles` to ensure that `addSecurityRoles` is called before `setUserInfoInThreadContext`. The ConcurrentModificationException would arise in scenarios where the first thread was done with `setUserInfoInThreadContext` and had moved onto `addSecurityRoles` to add the mapped roles into the user's set of security roles. On the first call to `addSecurityRoles` it may be populating the set with data, any subsequent call would be a noop in other threads. If simultaneously there is another thread executing `setUserInfoInThreadContext` while the first thread is in `addSecurityRoles` then a ConcurrentModificationException is thrown inside the `Sets.union(...)` call. By calling `addSecurityRoles` before `setUserInfoInThreadContext`, it can be guaranteed that no ConcurrentModificationException could be thrown because the user's security roles will already be set and any thread that attempts another call will be a noop. * Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation) Bug fix opensearch-project#2263 Tested by running a python script to bulk insert 100k documents and using tmux to run the script in multiple shells at once. Before the change the bug is reproducible regularly, but after the change the bug cannot be reproduced. - [ ] New functionality includes testing - [ ] New functionality has been documented - [ ] Commits are signed per the DCO using --signoff By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin). --------- Signed-off-by: Craig Perkins <[email protected]> Signed-off-by: Craig Perkins <[email protected]> (cherry picked from commit cd699bb)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What is the bug?
With option
plugins.security.cache.ttl_minutes: 0opensearch sometimes raises java.util.ConcurrentModificationException error and logstash get 500 error at that moment.I use dashboards with OpenID integration, so option
plugins.security.cache.ttl_minutes: 0is recommended by documentation. Withoutplugins.security.cache.ttl_minutes: 0dashboards doesn't work properly with OpenID (see bug opensearch-project/security-dashboards-plugin#1138 (comment))How can one reproduce the bug?
Steps to reproduce the behavior:
plugins.security.cache.ttl_minutes=0opensearch logs
Logstash logs:
What is the expected behavior?
opensearch indexes data from logstash without errors
What is your host/environment?
opensearch.yml
opensearch-security/config.yml
The text was updated successfully, but these errors were encountered: