CI: SmokeTestWatcherWithSecurityIT.testSearchTransformInsufficientPermissions fails #35361
Assignees
Labels
Comments
alpar-t
added
>test-failure
|
Pinging @elastic/es-core-features |
|
It seems like #33291 fixed the original assertion but resulted in a new failure. |
jasontedor
added a commit
to jasontedor/elasticsearch
that referenced
this issue
Nov 8, 2018
* elastic/master: (25 commits) Fixes fast vector highlighter docs per issue 24318. (elastic#34190) [ML] Prevent notifications on deletion of a non existent job (elastic#35337) [CCR] Auto follow Coordinator fetch cluster state in system context (elastic#35120) Mute test for elastic#35361 Preserve `date_histogram` format when aggregating on unmapped fields (elastic#35254) Test: Mute failing SSL test Allow unmapped fields in composite aggregations (elastic#35331) [RCI] Add IndexShardOperationPermits.asyncBlockOperations(ActionListener<Releasable>) (elastic#34902) HLRC: reindex API with wait_for_completion false (elastic#35202) Add docs on JNA temp directory not being noexec (elastic#35355) [CCR] Adjust list of dynamic index settings that should be replicated (elastic#35195) Replicate index settings to followers (elastic#35089) Rename RealmConfig.globalSettings() to settings() (elastic#35330) [TEST] Cleanup FileUserPasswdStoreTests (elastic#35329) Scripting: Add back lookup vars in score script (elastic#34833) watcher: Fix integration tests to ensure correct start/stop of Watcher (elastic#35271) Remove ALL shard check in CheckShrinkReadyStep (elastic#35346) Use soft-deleted docs to resolve strategy for engine operation (elastic#35230) [ILM] Check shard and relocation status in AllocationRoutedStep (elastic#35316) Ignore date ranges containing 'now' when pre-processing a percolator query (elastic#35160) ...
|
@atorok Latest commit in the build : e685cfe That build was 15 commits too early. Can we close this and un-mute the test. FWIW, I am not 100% positive that the fix is a complete, but it does eliminate a race condition which should reduce some (hopefully all) of the noise from this test. |
|
I didn't mean to imply that your changes caused new issues, I was thinking they uncovered it by going past the assert and running into But the last build on master that failed like this was before your changes. |
|
Build https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-darwin-compatibility/55/consoleFull seems to have similar failures |
|
The latest build failure includes the the "fix" commit, which didn't seem to fix this test. Here are the relevant bits : Doesn't repro for me. Feel free to mute this test while we get to a resolution. |
|
Build https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+periodic/255/console
Still doesn't reproduce locally. |
|
I muted SmokeTestWatcherWithSecurityIT in master and 6.x |
|
And some more on 6.5: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.5+multijob-darwin-compatibility/70/console Probably worth muting there as well. |
|
I cherry-picked 2791e81 to 6.5, muting all of SmokeTestWatcherWithSecurityIT there as well. |
talevy
added a commit
to talevy/elasticsearch
that referenced
this issue
Feb 12, 2019
talevy
added a commit
that referenced
this issue
Feb 14, 2019
|
this popped up again after I unmuted it. I'll spend some time debugging. If nothing promising, I will re-mute |
|
@tvernum you can take a look at it if you'd like. I intended on returning to the investigation tomorrow, but am not married to it |
jakelandis
added a commit
to jakelandis/elasticsearch
that referenced
this issue
Apr 29, 2019
This commit un-mutes SmokeTestWatcherWithSecurityIT by ensuring that between test invocations a new WatchId is used. It is believed that since the same WatchId is used between tests, and that Watch is never deleted, the test failures may be due to a race condition between the current Watch execution, the new Watch (with the same ID), and the busy wait for the Watcher history document. This race condtion can be avoided by simply using a new WatchId and removing the Watch between test invocations. This commit also adds a minor change (XPackRestTestConstants.TEMPLATE_NAMES_NO_ILM) that is needed since the test was muted. Fixes elastic#35361
jakelandis
added a commit
to jakelandis/elasticsearch
that referenced
this issue
May 2, 2019
This commit unmutes the org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT test suite, fixes a bug [1] that was introduced while the test was muted, and adds some additional debug logging, and enables debug for the ES instance used in this Watcher test. The bug fixed here is minor and unlikely to happen. It requires ES to be started with ILM disabled, Watcher enabled, and Watcher explicitly stopped and restarted. Due to validation Watcher does not fully start and can result in a partially started state. This is an unlikely scenerio outside of the testing framework. Optimistically closing the following Fixes elastic#35361 Fixes elastic#30777 Fixes elastic#35361 Fixes elastic#33291 Fixes elastic#29893 If this does not fully fix the issue, there will now be better debug logging.
jakelandis
added a commit
to jakelandis/elasticsearch
that referenced
this issue
May 10, 2019
There are likely multiple root causes to the seemingly random failures generated by SmokeTestWatcherWithSecurityIT. This commit un-mutes this this test, address one known cause and adds debug logging for this test. The known root cause for one failure is that we can have a Watch running that is reading data from an index. Before we stop Watcher we delete that index. If Watcher happens to execute after deletion of the index but before the stop of Watcher the test can fail. The fix here is to simply move the index deletion after the stop of Watcher. Related elastic#35361 Related elastic#30777 Related elastic#35361 Related elastic#33291 Related elastic#29893
|
Un-muted this test on PR #42409 to obtain additional logs. If (when?) this test fails again please obtain the following information before muting the test:
|
jakelandis
added a commit
that referenced
this issue
May 24, 2019
* Address test failures for SmokeTestWatcherWithSecurityIT There are likely multiple root causes to the seemingly random failures generated by SmokeTestWatcherWithSecurityIT. This commit un-mutes this this test, address one known cause and adds debug logging for this test. The known root cause for one failure is that we can have a Watch running that is reading data from an index. Before we stop Watcher we delete that index. If Watcher happens to execute after deletion of the index but before the stop of Watcher the test can fail. The fix here is to simply move the index deletion after the stop of Watcher. Related #35361 Related #30777 Related #33291 Related #29893
gurkankaymak
pushed a commit
to gurkankaymak/elasticsearch
that referenced
this issue
May 27, 2019
* Address test failures for SmokeTestWatcherWithSecurityIT There are likely multiple root causes to the seemingly random failures generated by SmokeTestWatcherWithSecurityIT. This commit un-mutes this this test, address one known cause and adds debug logging for this test. The known root cause for one failure is that we can have a Watch running that is reading data from an index. Before we stop Watcher we delete that index. If Watcher happens to execute after deletion of the index but before the stop of Watcher the test can fail. The fix here is to simply move the index deletion after the stop of Watcher. Related elastic#35361 Related elastic#30777 Related elastic#33291 Related elastic#29893
|
This failed today. Links
Failure Reproduction Cluster Logs |
|
Urgh. Wrong test. The above failure is from |
|
Closing in favour of #30777 as this test fails in a similar way as the mentioned issue. PR #50931 will add more logging in case this fails again. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Example build failure
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/162/
There are other issues for this tetst: #30777 and #29893
Reproduction line
does not reproduce locally
Example relevant log:
test log
17:22:27 > Throwable #1: org.elasticsearch.client.ResponseException: method [POST], host [http://[::1]:43210], URI [/.watcher-history-*/_search], status line [HTTP/1.1 503 Service Unavailable] 17:22:27 > {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]},"status":503} 17:22:27 > at __randomizedtesting.SeedInfo.seed([98BA944A70941ADF:5CF90B4519F1D32]:0) 17:22:27 2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+elasticsearch+master+intake/x-pack/qa/smoke-test-watcher-with-security/build/testrun/integTestRunner/J0/temp/org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT_98BA944A70941ADF-001 17:22:27 2> NOTE: test params are: codec=Asserting(Lucene80): {}, docValues:{}, maxPointsInLeafNode=1435, maxMBSortInHeap=7.554326604207141, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@39e7681e), locale=de-AT, timezone=Etc/Universal 17:22:27 > at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:687) 17:22:27 > at org.elasticsearch.client.RestClient.performRequest(RestClient.java:218) 17:22:27 > at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$3(SmokeTestWatcherWithSecurityIT.java:325) 17:22:27 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836) 17:22:27 2> NOTE: Linux 3.10.0-862.14.4.el7.x86_64 amd64/Oracle Corporation 1.8.0_192 (64-bit)/cpus=16,threads=1,free=435688080,total=514850816 17:22:27 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:822) 17:22:27 2> NOTE: All tests run in this JVM: [SmokeTestWatcherWithSecurityIT] 17:22:27 > at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306) 17:22:27 > at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:301) 17:22:27 > at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformInsufficientPermissions(SmokeTestWatcherWithSecurityIT.java:237) 17:22:27 > at java.lang.Thread.run(Thread.java:748) 17:22:27 > Caused by: org.elasticsearch.client.ResponseException: method [POST], host [http://[::1]:43210], URI [/.watcher-history-*/_search], status line [HTTP/1.1 503 Service Unavailable] 17:22:27 > {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]},"status":503} 17:22:27 > at org.elasticsearch.client.RestClient$1.completed(RestClient.java:307) 17:22:27 > at org.elasticsearch.client.RestClient$1.completed(RestClient.java:292) 17:22:27 > at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119) 17:22:27 > at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177) 17:22:27 > at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436) 17:22:27 > at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326) 17:22:27 > at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) 17:22:27 > at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) 17:22:27 > at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) 17:22:27 > at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) 17:22:27 > at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) 17:22:27 > at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) 17:22:27 > at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) 17:22:27 > at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) 17:22:27 > at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) 17:22:27 > at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) 17:22:27 > ... 1 more 17:22:27 Completed [1/2] in 18.03s, 6 tests, 1 error <<< FAILURES!server
17:22:45 [2018-11-07T15:22:23,953][ERROR][o.e.x.w.a.i.IndexActionFactory] [node-0] failed to execute action [mWqFTVwuOPGWaKGubYkz/index] 17:22:45 org.elasticsearch.ElasticsearchSecurityException: action [indices:data/write/bulk[s]] is unauthorized for user [watcher_manager] 17:22:45 at org.elasticsearch.xpack.core.security.support.Exceptions.authorizationError(Exceptions.java:30) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:580) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationService.denial(AuthorizationService.java:555) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:299) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:169) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:175) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:169) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:157) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:181) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:159) ~[?:?] 17:22:45 at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:178) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:211) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:222) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:176) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:136) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:106) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:158) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:111) ~[?:?] 17:22:45 at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:139) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:117) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:386) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:499) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportBulkAction.executeIngestAndBulk(TransportBulkAction.java:238) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:164) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:88) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:141) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:90) ~[?:?] 17:22:45 at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:179) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:175) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:169) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:157) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:181) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:159) ~[?:?] 17:22:45 at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:178) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:211) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:222) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:176) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:136) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:106) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:158) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:111) ~[?:?] 17:22:45 at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:139) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:117) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction.doExecute(TransportSingleItemBulkWriteAction.java:69) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction.doExecute(TransportSingleItemBulkWriteAction.java:44) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:141) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:90) ~[?:?] 17:22:45 at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:179) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:175) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:169) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:157) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:181) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:159) ~[?:?] 17:22:45 at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:178) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:211) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:222) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:176) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:136) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:106) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:158) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:111) ~[?:?] 17:22:45 at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:139) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:117) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:389) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:378) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.client.support.AbstractClient.index(AbstractClient.java:397) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.actions.index.ExecutableIndexAction.lambda$execute$0(ExecutableIndexAction.java:100) ~[?:?] 17:22:45 at org.elasticsearch.xpack.core.ClientHelper.executeWithHeaders(ClientHelper.java:122) ~[x-pack-core-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.actions.index.ExecutableIndexAction.execute(ExecutableIndexAction.java:99) ~[?:?] 17:22:45 at org.elasticsearch.xpack.core.watcher.actions.ActionWrapper.execute(ActionWrapper.java:144) [x-pack-core-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:455) [x-pack-watcher-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:295) [x-pack-watcher-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$5(ExecutionService.java:395) [x-pack-watcher-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:544) [x-pack-watcher-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:627) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] 17:22:45 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] 17:22:45 at java.lang.Thread.run(Thread.java:834) [?:?] 17:22:45 ... SKIPPED 18 LINES ... 17:22:45 [2018-11-07T15:22:26,492][WARN ][r.suppressed ] [node-0] path: /.watcher-history-*/_search, params: {index=.watcher-history-*} 17:22:45 org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed 17:22:45 at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:106) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.search.InitialSearchPhase.access$200(InitialSearchPhase.java:50) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:277) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:53) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:440) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1127) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1244) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1218) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:66) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:102) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:348) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:342) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:336) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1075) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:726) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] 17:22:45 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] 17:22:45 at java.lang.Thread.run(Thread.java:834) [?:?] 17:22:45 ... SKIPPED 11 LINES ... 17:22:45 [2018-11-07T15:22:27,434][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [test_watch_9a9dc0c0-31f0-4ba8-97cd-fc2b896c6e1a-2018-11-07T15:22:27.430Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 65 LINES ... 17:22:45 [2018-11-07T15:22:30,649][ERROR][o.e.x.w.t.s.ExecutableSearchTransform] [node-0] failed to execute [search] transform for [_inlined__0dd49d6b-5069-41e4-bc95-7c9f3dbd5154-2018-11-07T15:22:30.646Z] 17:22:45 org.elasticsearch.common.ParsingException: no [query] registered for [does_not_exist] 17:22:45 at org.elasticsearch.index.query.AbstractQueryBuilder.parseInnerQueryBuilder(AbstractQueryBuilder.java:326) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.search.builder.SearchSourceBuilder.parseXContent(SearchSourceBuilder.java:1044) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.search.builder.SearchSourceBuilder.parseXContent(SearchSourceBuilder.java:989) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.support.search.WatcherSearchTemplateService.toSearchRequest(WatcherSearchTemplateService.java:70) ~[x-pack-watcher-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.transform.search.ExecutableSearchTransform.execute(ExecutableSearchTransform.java:51) [x-pack-watcher-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.transform.search.ExecutableSearchTransform.execute(ExecutableSearchTransform.java:27) [x-pack-watcher-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:444) [x-pack-watcher-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:295) [x-pack-watcher-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction$1.doRun(TransportExecuteWatchAction.java:149) [x-pack-watcher-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] 17:22:45 at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] 17:22:45 at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:627) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] 17:22:45 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] 17:22:45 at java.lang.Thread.run(Thread.java:834) [?:?] 17:22:45 ... SKIPPED 51 LINES ... 17:22:45 [2018-11-07T15:22:32,861][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_be0e7718-f474-43e4-b303-27c4809b48b5-2018-11-07T15:22:32.857Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 12 LINES ... 17:22:45 [2018-11-07T15:22:33,320][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_c791e726-58ee-4d44-8444-e26449d5a826-2018-11-07T15:22:33.133Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 [2018-11-07T15:22:33,399][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_12ce4688-7a30-4944-84ae-2061df2a7a49-2018-11-07T15:22:33.380Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 [2018-11-07T15:22:33,444][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_8d8bc3f3-94c0-45a9-a0b0-ba841ac17243-2018-11-07T15:22:33.419Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 15 LINES ... 17:22:45 [2018-11-07T15:22:33,999][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_ba8356d3-b059-4893-894f-d32f61a8b147-2018-11-07T15:22:33.927Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 11 LINES ... 17:22:45 [2018-11-07T15:22:34,389][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_7e085d94-cfba-4eae-adae-2fde7c41b79d-2018-11-07T15:22:34.364Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 21 LINES ... 17:22:45 [2018-11-07T15:22:35,171][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_9cf2df9c-5279-45cd-a6b6-cb5d2960ae56-2018-11-07T15:22:35.084Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 10 LINES ... 17:22:45 [2018-11-07T15:22:35,523][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_f1417293-e22c-4f2f-b9b4-0beae3ba2858-2018-11-07T15:22:35.521Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 21 LINES ... 17:22:45 [2018-11-07T15:22:36,291][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [test_watch_44475776-4c29-4e35-9d32-bccfb3cb725d-2018-11-07T15:22:36.214Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 11 LINES ... 17:22:45 [2018-11-07T15:22:36,659][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_a1d347a5-7f2a-4af4-ba7f-af60a08e7c21-2018-11-07T15:22:36.656Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 17 LINES ... 17:22:45 [2018-11-07T15:22:37,251][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_5c6fb0f8-3856-472c-83bc-a2eef30b5eb2-2018-11-07T15:22:37.181Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 14 LINES ... 17:22:45 [2018-11-07T15:22:38,680][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [test_watch_ee71191e-4d90-4349-931a-d5f927417296-2018-11-07T15:22:37.659Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 42 LINES ... 17:22:45 [2018-11-07T15:22:40,236][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [test_watch_b9d98c2e-20ea-4363-9dbc-aff7cf00a985-2018-11-07T15:22:40.235Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 [2018-11-07T15:22:40,242][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [test_watch_c87d0280-3af9-4551-bcd8-4897a313476c-2018-11-07T15:22:40.241Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 10 LINES ... 17:22:45 [2018-11-07T15:22:40,611][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_c33163fe-7ff5-426a-b602-41d70f92da42-2018-11-07T15:22:40.585Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 27 LINES ... 17:22:45 [2018-11-07T15:22:41,584][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_b4833751-f2c7-46cc-a1da-c99af56f0745-2018-11-07T15:22:41.416Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 [2018-11-07T15:22:41,648][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_b5b0be44-b074-4d11-a961-9a07165d5f69-2018-11-07T15:22:41.630Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 [2018-11-07T15:22:41,691][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_10d43a66-61cb-4f02-9573-d891d419831f-2018-11-07T15:22:41.668Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 18 LINES ... 17:22:45 [2018-11-07T15:22:42,327][ERROR][o.e.x.w.a.i.IndexActionFactory] [node-0] failed to execute action [my_watch/index] 17:22:45 org.elasticsearch.ElasticsearchSecurityException: action [indices:data/write/bulk[s]] is unauthorized for user [watcher_manager] 17:22:45 at org.elasticsearch.xpack.core.security.support.Exceptions.authorizationError(Exceptions.java:30) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:580) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationService.denial(AuthorizationService.java:555) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:299) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:169) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:175) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:169) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:157) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:181) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:159) ~[?:?] 17:22:45 at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:178) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:211) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:222) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:176) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:136) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:106) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:158) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:111) ~[?:?] 17:22:45 at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:139) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:117) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:386) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:499) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportBulkAction.executeIngestAndBulk(TransportBulkAction.java:238) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:164) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:88) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:141) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:90) ~[?:?] 17:22:45 at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:179) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:175) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:169) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:157) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:181) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:159) ~[?:?] 17:22:45 at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:178) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:211) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:222) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:176) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:136) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:106) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:158) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:111) ~[?:?] 17:22:45 at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:139) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:117) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction.doExecute(TransportSingleItemBulkWriteAction.java:69) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction.doExecute(TransportSingleItemBulkWriteAction.java:44) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:141) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:90) ~[?:?] 17:22:45 at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:179) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:175) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:169) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:157) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:181) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:159) ~[?:?] 17:22:45 at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:178) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:211) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:222) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:176) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:136) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:106) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:158) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:111) ~[?:?] 17:22:45 at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:139) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:117) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:389) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:378) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.client.support.AbstractClient.index(AbstractClient.java:397) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.actions.index.ExecutableIndexAction.lambda$execute$0(ExecutableIndexAction.java:100) ~[?:?] 17:22:45 at org.elasticsearch.xpack.core.ClientHelper.executeWithHeaders(ClientHelper.java:122) ~[x-pack-core-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.actions.index.ExecutableIndexAction.execute(ExecutableIndexAction.java:99) ~[?:?] 17:22:45 at org.elasticsearch.xpack.core.watcher.actions.ActionWrapper.execute(ActionWrapper.java:144) [x-pack-core-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:455) [x-pack-watcher-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:295) [x-pack-watcher-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction$1.doRun(TransportExecuteWatchAction.java:149) [x-pack-watcher-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] 17:22:45 at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] 17:22:45 at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:627) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] 17:22:45 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] 17:22:45 at java.lang.Thread.run(Thread.java:834) [?:?] 17:22:45 [2018-11-07T15:22:42,330][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_3063ff1f-3667-41eb-b1c6-e6bf66ddee50-2018-11-07T15:22:42.326Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 8 LINES ... 17:22:45 [2018-11-07T15:22:42,583][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [test_watch_2492079c-a958-46e5-b15e-d44347cabab7-2018-11-07T15:22:42.583Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 3 LINES ... 17:22:45 [2018-11-07T15:22:42,720][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [test_watch_1ac97e9e-d648-41d7-b19c-34f6beb8262c-2018-11-07T15:22:42.611Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 [2018-11-07T15:22:42,766][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [test_watch_9264c270-5da1-4e3c-9d75-46132f5c192b-2018-11-07T15:22:42.748Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 10 LINES ... 17:22:45 [2018-11-07T15:22:43,095][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_0a38706d-af18-4cd9-8393-a04d29f30381-2018-11-07T15:22:43.026Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 15 LINES ... 17:22:45 [2018-11-07T15:22:43,671][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_36d41cd5-379d-4bb9-b7a3-4babe24cd2de-2018-11-07T15:22:43.646Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 =========================================The test also seems to fail with:
Similar to #33291 /cc @jakelandis
Frequency
1-2 times / week, increased to 4 this week.
Going to mute the test.
The text was updated successfully, but these errors were encountered: