Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] MultiClusterSearchWithSecurityYamlTestSuiteIT #31462

Closed
albertzaharovits opened this issue Jun 20, 2018 · 4 comments
Closed

[CI] MultiClusterSearchWithSecurityYamlTestSuiteIT #31462

albertzaharovits opened this issue Jun 20, 2018 · 4 comments
Assignees
Labels
:Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) >test-failure Triaged test failures from CI

Comments

@albertzaharovits
Copy link
Contributor

albertzaharovits commented Jun 20, 2018

The following Rest IntegTest is failing:

./gradlew :x-pack:qa:multi-cluster-search-security:mixedClusterTestRunner -Dtests.seed=986EC309467F6543 -Dtests.class=org.elasticsearch.xpack.security.MultiClusterSearchWithSecurityYamlTestSuiteIT -Dtests.method="test {yaml=multi_cluster/20_info/Add transient remote cluster based on the preset cluster and check remote info}" -Dtests.security.manager=true -Dtests.locale=gu -Dtests.timezone=Pacific/Fiji -Dtests.rest.suite=multi_cluster

with

MultiClusterSearchWithSecurityYamlTestSuiteIT.test {yaml=multi_cluster/20_info/Add transient remote cluster based on the preset cluster and check remote info} <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: Failure at [multi_cluster/20_info:64]: expected [2xx] status code but api [search] returned [403 Forbidden] [{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/shards/search_shards] is unauthorized for user [_system]","stack_trace":"ElasticsearchSecurityException[action [indices:admin/shards/search_shards] is unauthorized for user [_system]]\n\tat org.elasticsearch.xpack.core.security.support.Exceptions.authorizationError(Exceptions.java:30)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:574)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.denial(AuthorizationService.java:552)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:157)\n\tat org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$2(ServerTransportFilter.java:147)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:173)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:167)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:149)\n\tat org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$3(ServerTransportFilter.java:150)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60)\n\tat 

and does not reproduce locally, looks like a race.

Indeed _system user somehow gets to handle a shard search which is not something he has privileges for and is rightfully rejected.

The problem is to see how the _system user assigned to a search request without any Authorization header: x-pack/qa/multi-cluster-search-security/src/test/resources/rest-api-spec/test/multi_cluster/20_info.yml:L64

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=java10,ES_RUNTIME_JAVA=java10,nodes=virtual&&linux/124/consoleFull

Note: looks similar to #30565

@albertzaharovits albertzaharovits added >test-failure Triaged test failures from CI :Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) labels Jun 20, 2018
@albertzaharovits albertzaharovits self-assigned this Jun 20, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-security

@albertzaharovits
Copy link
Contributor Author

Another identical failure, this time on 6.x:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=amazon/1132/console

17:43:08   1> [2018-06-20T05:43:05,769][INFO ][o.e.x.s.MultiClusterSearchWithSecurityYamlTestSuiteIT] Stash dump on test failure [{
17:43:08   1>   "stash" : {
17:43:08   1>     "remote_ip" : "[::1]:46125",
17:43:08   1>     "body" : {
17:43:08   1>       "error" : {
17:43:08   1>         "root_cause" : [
17:43:08   1>           {
17:43:08   1>             "type" : "security_exception",
17:43:08   1>             "reason" : "action [indices:admin/shards/search_shards] is unauthorized for user [_system]",
17:43:08   1>             "stack_trace" : "ElasticsearchSecurityException[action [indices:admin/shards/search_shards] is unauthorized for user [_system]]
17:43:08   1> 	at org.elasticsearch.xpack.core.security.support.Exceptions.authorizationError(Exceptions.java:30)
17:43:08   1> 	at org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:574)
17:43:08   1> 	at org.elasticsearch.xpack.security.authz.AuthorizationService.denial(AuthorizationService.java:552)
17:43:08   1> 	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:157)
17:43:08   1> 	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$2(ServerTransportFilter.java:147)
17:43:08   1> 	at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:173)
17:43:08   1> 	at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:167)
17:43:08   1> 	at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:149)
17:43:08   1> 	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$3(ServerTransportFilter.java:150)
17:43:08   1> 	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60)
17:43:08   1> 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:172)
17:43:08   1> 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:205)
17:43:08   1> 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:216)
17:43:08   1> 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:170)
17:43:08   1> 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:131)
17:43:08   1> 	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:101)
17:43:08   1> 	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:129)
17:43:08   1> 	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:302)
17:43:08   1> 	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
17:43:08   1> 	at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1663)
17:43:08   1> 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
17:43:08   1> 	at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:135)
17:43:08   1> 	at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1621)
17:43:08   1> 	at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1485)
17:43:08   1> 	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)
17:43:08   1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
17:43:08   1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
17:43:08   1> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
17:43:08   1> 	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
17:43:08   1> 	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
17:43:08   1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
17:43:08   1> 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
17:43:08   1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
17:43:08   1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
17:43:08   1> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
17:43:08   1> 	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)
17:43:08   1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
17:43:08   1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
17:43:08   1> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
17:43:08   1> 	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
17:43:08   1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
17:43:08   1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
17:43:08   1> 	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
17:43:08   1> 	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
17:43:08   1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
17:43:08   1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545)
17:43:08   1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499)
17:43:08   1> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
17:43:08   1> 	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
17:43:08   1> 	at java.lang.Thread.run(Thread.java:844)
17:43:08   1> "
17:43:08   1>           }
17:43:08   1>         ],
17:43:08   1>         "type" : "security_exception",
17:43:08   1>         "reason" : "action [indices:admin/shards/search_shards] is unauthorized for user [_system]",
17:43:08   1>         "stack_trace" : "ElasticsearchSecurityException[action [indices:admin/shards/search_shards] is unauthorized for user [_system]]
17:43:08   1> 	at org.elasticsearch.xpack.core.security.support.Exceptions.authorizationError(Exceptions.java:30)
17:43:08   1> 	at org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:574)
17:43:08   1> 	at org.elasticsearch.xpack.security.authz.AuthorizationService.denial(AuthorizationService.java:552)
17:43:08   1> 	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:157)
17:43:08   1> 	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$2(ServerTransportFilter.java:147)
17:43:08   1> 	at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:173)
17:43:08   1> 	at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:167)
17:43:08   1> 	at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:149)
17:43:08   1> 	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$3(ServerTransportFilter.java:150)
17:43:08   1> 	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60)
17:43:08   1> 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:172)
17:43:08   1> 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:205)
17:43:08   1> 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:216)
17:43:08   1> 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:170)
17:43:08   1> 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:131)
17:43:08   1> 	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:101)
17:43:08   1> 	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:129)

@jasontedor
Copy link
Member

Maybe this was caused by #31241?

@jasontedor
Copy link
Member

@ywelsch Would you take a look at this one?

ywelsch added a commit that referenced this issue Jun 27, 2018
Establishing remote cluster connections uses a queue to coordinate multiple concurrent connect
attempts. Connect attempts can be initiated by user triggered searches as well as by system events 
(e.g. when nodes disconnect). Multiple such concurrent events can lead to the connectListener of
one event to be called under the thread context of another connect attempt. This can lead to the
situation as seen in #31462 where the connect listener is executed under the system context, which 
breaks when fetching the search shards from the remote cluster.

Closes #31462
ywelsch added a commit that referenced this issue Jun 27, 2018
Establishing remote cluster connections uses a queue to coordinate multiple concurrent connect
attempts. Connect attempts can be initiated by user triggered searches as well as by system events 
(e.g. when nodes disconnect). Multiple such concurrent events can lead to the connectListener of
one event to be called under the thread context of another connect attempt. This can lead to the
situation as seen in #31462 where the connect listener is executed under the system context, which 
breaks when fetching the search shards from the remote cluster.

Closes #31462
ywelsch added a commit that referenced this issue Jun 27, 2018
Establishing remote cluster connections uses a queue to coordinate multiple concurrent connect
attempts. Connect attempts can be initiated by user triggered searches as well as by system events 
(e.g. when nodes disconnect). Multiple such concurrent events can lead to the connectListener of
one event to be called under the thread context of another connect attempt. This can lead to the
situation as seen in #31462 where the connect listener is executed under the system context, which 
breaks when fetching the search shards from the remote cluster.

Closes #31462
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) >test-failure Triaged test failures from CI
Projects
None yet
Development

No branches or pull requests

4 participants