Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SimpleKdcLdapServerTests.testPrincipalCreationAndSearchOnLdap fails sporadically #32739

Closed
alpar-t opened this issue Aug 9, 2018 · 5 comments
Assignees
Labels
:Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) >test-failure Triaged test failures from CI v7.0.0-beta1

Comments

@alpar-t
Copy link
Contributor

alpar-t commented Aug 9, 2018

Logs: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=java10,ES_RUNTIME_JAVA=java8,nodes=virtual&&linux/226/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+periodic/2514/console

reproduction line (does not reproduce locally), failed once before on August 3 on the ccr branhc in CI.

REPRODUCE WITH: ./gradlew :x-pack:plugin:security:test \
  -Dtests.seed=D3ABF0085A9FA9D7 \
  -Dtests.class=org.elasticsearch.xpack.security.authc.kerberos.SimpleKdcLdapServerTests \
  -Dtests.method="testPrincipalCreationAndSearchOnLdap" \
  -Dtests.security.manager=true \
  -Dtests.locale=en \
  -Dtests.timezone=America/Argentina/Cordoba

Relevant logs:

19:18:51   1> [2018-08-08T14:18:39,944][INFO ][o.e.x.s.a.k.KerberosTicketValidatorTests] [testValidKebrerosTicket] before test
19:18:51   2> REPRODUCE WITH: ./gradlew :x-pack:plugin:security:test -Dtests.seed=199D7AB2E045185D -Dtests.class=org.elasticsearch.xpack.security.authc.kerberos.KerberosTicketValidatorTests -Dtests.method="testValidKebrerosTicket" -Dtests.security.manager=true -Dtests.locale=de-AT -Dtests.timezone=America/Rankin_Inlet
19:18:51   1> [2018-08-08T14:18:40,249][INFO ][o.a.k.k.k.i.LdapIdentityBackend] [testValidKebrerosTicket] Initializing the Ldap identity backend.
19:18:51   1> [2018-08-08T14:18:40,402][INFO ][o.a.d.a.l.c.o.DefaultLdapCodecService] [testValidKebrerosTicket] Registered pre-bundled control factory: 1.3.6.1.4.1.18060.0.0.1
19:18:51   1> [2018-08-08T14:18:40,405][INFO ][o.a.d.a.l.c.o.DefaultLdapCodecService] [testValidKebrerosTicket] Registered pre-bundled control factory: 2.16.840.1.113730.3.4.7
19:18:51   1> [2018-08-08T14:18:40,409][INFO ][o.a.d.a.l.c.o.DefaultLdapCodecService] [testValidKebrerosTicket] Registered pre-bundled control factory: 2.16.840.1.113730.3.4.2
19:18:51   2> REPRODUCE WITH: ./gradlew :x-pack:plugin:security:test -Dtests.seed=199D7AB2E045185D -Dtests.class=org.elasticsearch.xpack.security.authc.kerberos.KerberosTicketValidatorTests -Dtests.method="testValidKebrerosTicket" -Dtests.security.manager=true -Dtests.locale=de-AT -Dtests.timezone=America/Rankin_Inlet
19:18:51   1> [2018-08-08T14:18:40,412][INFO ][o.a.d.a.l.c.o.DefaultLdapCodecService] [testValidKebrerosTicket] Registered pre-bundled control factory: 2.16.840.1.113730.3.4.18
19:18:51   1> [2018-08-08T14:18:40,414][INFO ][o.a.d.a.l.c.o.DefaultLdapCodecService] [testValidKebrerosTicket] Registered pre-bundled control factory: 1.2.840.113556.1.4.319
19:18:51   1> [2018-08-08T14:18:40,416][INFO ][o.a.d.a.l.c.o.DefaultLdapCodecService] [testValidKebrerosTicket] Registered pre-bundled control factory: 2.16.840.1.113730.3.4.3
19:18:51   1> [2018-08-08T14:18:40,418][INFO ][o.a.d.a.l.c.o.DefaultLdapCodecService] [testValidKebrerosTicket] Registered pre-bundled control factory: 1.3.6.1.4.1.4203.1.10.1
19:18:51   1> [2018-08-08T14:18:40,420][INFO ][o.a.d.a.l.c.o.DefaultLdapCodecService] [testValidKebrerosTicket] Registered pre-bundled control factory: 1.2.840.113556.1.4.473
19:18:51   1> [2018-08-08T14:18:40,421][INFO ][o.a.d.a.l.c.o.DefaultLdapCodecService] [testValidKebrerosTicket] Registered pre-bundled control factory: 1.2.840.113556.1.4.474
19:18:51   1> [2018-08-08T14:18:40,423][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled control factory: 1.3.6.1.4.1.18060.0.0.1
19:18:51   1> [2018-08-08T14:18:40,424][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled control factory: 2.16.840.1.113730.3.4.7
19:18:51   1> [2018-08-08T14:18:40,424][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled control factory: 2.16.840.1.113730.3.4.2
19:18:51   1> [2018-08-08T14:18:40,424][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled control factory: 2.16.840.1.113730.3.4.18
19:18:51   1> [2018-08-08T14:18:40,424][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled control factory: 1.2.840.113556.1.4.319
19:18:51   1> [2018-08-08T14:18:40,424][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled control factory: 2.16.840.1.113730.3.4.3
19:18:51   1> [2018-08-08T14:18:40,424][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled control factory: 1.3.6.1.4.1.4203.1.10.1
19:18:51   1> [2018-08-08T14:18:40,426][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled control factory: 1.3.6.1.4.1.42.2.27.8.5.1
19:18:51   1> [2018-08-08T14:18:40,427][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled control factory: 2.16.840.1.113730.3.4.9
19:18:51   1> [2018-08-08T14:18:40,429][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled control factory: 2.16.840.1.113730.3.4.10
19:18:51   1> [2018-08-08T14:18:40,430][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled control factory: 1.3.6.1.4.1.4203.1.9.1.3
19:18:51   1> [2018-08-08T14:18:40,432][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled control factory: 1.3.6.1.4.1.4203.1.9.1.4
19:18:51   1> [2018-08-08T14:18:40,434][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled control factory: 1.3.6.1.4.1.4203.1.9.1.1
19:18:51   1> [2018-08-08T14:18:40,435][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled control factory: 1.3.6.1.4.1.4203.1.9.1.2
19:18:51   1> [2018-08-08T14:18:40,435][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled control factory: 1.2.840.113556.1.4.473
19:18:51   1> [2018-08-08T14:18:40,435][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled control factory: 1.2.840.113556.1.4.474
19:18:51   1> [2018-08-08T14:18:40,437][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled control factory: 1.2.840.113556.1.4.841
19:18:51   1> [2018-08-08T14:18:40,442][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled control factory: 1.2.840.113556.1.4.417
19:18:51   1> [2018-08-08T14:18:40,446][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled control factory: 1.2.840.113556.1.4.1413
19:18:51   1> [2018-08-08T14:18:40,450][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled control factory: 1.2.840.113556.1.4.528
19:18:51   1> [2018-08-08T14:18:40,458][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled extended operation factory: 1.3.6.1.1.8
19:18:51   1> [2018-08-08T14:18:40,464][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled extended operation factory: 1.3.6.1.4.1.18060.0.1.8
19:18:51   1> [2018-08-08T14:18:40,472][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled extended operation factory: 1.3.6.1.4.1.18060.0.1.3
19:18:51   1> [2018-08-08T14:18:40,479][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled extended operation factory: 1.3.6.1.4.1.18060.0.1.6
19:18:51   1> [2018-08-08T14:18:40,483][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled extended operation factory: 1.3.6.1.4.1.18060.0.1.5
19:18:51   1> [2018-08-08T14:18:40,491][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled extended operation factory: 1.3.6.1.4.1.4203.1.11.1
19:18:51   1> [2018-08-08T14:18:40,498][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled extended operation factory: 1.3.6.1.4.1.4203.1.11.3
19:18:51   1> [2018-08-08T14:18:40,505][INFO ][o.a.d.a.l.c.s.CodecFactoryUtil] [testValidKebrerosTicket] Registered pre-bundled extended operation factory: 1.3.6.1.4.1.1466.20037
19:18:51   1> [2018-08-08T14:18:41,366][INFO ][o.e.x.s.a.k.KerberosTicketValidatorTests] [testValidKebrerosTicket] after test
19:18:51 ERROR   1.45s J3 | KerberosTicketValidatorTests.testValidKebrerosTicket <<< FAILURES!
19:18:51    > Throwable #1: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessClassInPackage.sun.reflect")
19:18:51    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
19:18:51    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
19:18:51    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
19:18:51    > 	at java.lang.SecurityManager.checkPackageAccess(SecurityManager.java:1564)
19:18:51    > 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:329)
19:18:51    > 	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
19:18:51    > 	at org.apache.mina.core.future.DefaultIoFuture.checkDeadLock(DefaultIoFuture.java:279)
19:18:51    > 	at org.apache.mina.core.future.DefaultIoFuture.await0(DefaultIoFuture.java:241)
19:18:51    > 	at org.apache.mina.core.future.DefaultIoFuture.awaitUninterruptibly(DefaultIoFuture.java:174)
19:18:51    > 	at org.apache.directory.ldap.client.api.LdapNetworkConnection.writeRequest(LdapNetworkConnection.java:4252)
19:18:51    > 	at org.apache.directory.ldap.client.api.LdapNetworkConnection.bindAsync(LdapNetworkConnection.java:1389)
19:18:51    > 	at org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1292)
19:18:51    > 	at org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:130)
19:18:51    > 	at org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:114)
19:18:51    > 	at org.apache.kerby.kerberos.kdc.identitybackend.LdapIdentityBackend.startConnection(LdapIdentityBackend.java:100)
19:18:51    > 	at org.apache.kerby.kerberos.kdc.identitybackend.LdapIdentityBackend.doInitialize(LdapIdentityBackend.java:111)
19:18:51    > 	at org.apache.kerby.kerberos.kerb.identity.backend.AbstractIdentityBackend.initialize(AbstractIdentityBackend.java:67)
19:18:51    > 	at org.apache.kerby.kerberos.kerb.server.KdcUtil.getBackend(KdcUtil.java:115)
19:18:51    > 	at org.apache.kerby.kerberos.kerb.server.impl.AbstractInternalKdcServer.init(AbstractInternalKdcServer.java:65)
19:18:51    > 	at org.apache.kerby.kerberos.kerb.server.KdcServer.init(KdcServer.java:256)
19:18:51    > 	at org.apache.kerby.kerberos.kerb.server.SimpleKdcServer.init(SimpleKdcServer.java:155)
19:18:51    > 	at org.elasticsearch.xpack.security.authc.kerberos.SimpleKdcLdapServer.prepareKdcServerAndStart(SimpleKdcLdapServer.java:150)
19:18:51    > 	at org.elasticsearch.xpack.security.authc.kerberos.SimpleKdcLdapServer.init(SimpleKdcLdapServer.java:104)
19:18:51    > 	at org.elasticsearch.xpack.security.authc.kerberos.SimpleKdcLdapServer.access$000(SimpleKdcLdapServer.java:39)
19:18:51    > 	at org.elasticsearch.xpack.security.authc.kerberos.SimpleKdcLdapServer$2.run(SimpleKdcLdapServer.java:89)
19:18:51    > 	at org.elasticsearch.xpack.security.authc.kerberos.SimpleKdcLdapServer$2.run(SimpleKdcLdapServer.java:86)
19:18:51    > 	at java.security.AccessController.doPrivileged(Native Method)
19:18:51    > 	at org.elasticsearch.xpack.security.authc.kerberos.SimpleKdcLdapServer.<init>(SimpleKdcLdapServer.java:86)
19:18:51    > 	at org.elasticsearch.xpack.security.authc.kerberos.KerberosTestCase.startSimpleKdcLdapServer(KerberosTestCase.java:105)
19:18:51    > 	at java.lang.Thread.run(Thread.java:748)Throwable #2: java.lang.NullPointerException
19:18:51    > 	at org.elasticsearch.xpack.security.authc.kerberos.KerberosTestCase.tearDownMiniKdc(KerberosTestCase.java:128)
19:18:51    > 	at java.lang.Thread.run(Thread.java:748)
@alpar-t alpar-t added >test-failure Triaged test failures from CI :Security/Security Security issues without another label v7.0.0 v6.5.0 and removed v6.5.0 labels Aug 9, 2018
@jaymode
Copy link
Member

jaymode commented Aug 24, 2018

@bizybot can you please track down this failure?

@jaymode jaymode added the :Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) label Aug 24, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-security

@jaymode jaymode removed the :Security/Security Security issues without another label label Aug 24, 2018
bizybot pushed a commit to bizybot/elasticsearch that referenced this issue Aug 27, 2018
The test SimpleKdcLdapServerTests#testPrincipalCreationAndSearchOnLdap
fails intermittently and I could not reproduce this locally.

There were two exceptions from the console logs of which one might be
the reason for the failure. When simple kdc ldap server starts,
internally it starts kdc server and ldap server. Kdc server then tries
to connect to configured ldap backend and during this process it
waits for the connection to succeed, meanwhile checking for deadlocks
in between. During this deadlock check it needs permission to
`accessClassInPackage.sun.reflect`, the fix here is to add the
required permission so that the check does not throw exception.

I think once we fix this issue, we may have something to look forward
if there is indeed a deadlock or its just waiting for the process
to complete on slow test runs.
Added a null check in after test method.

See elastic#32739
@bizybot
Copy link
Contributor

bizybot commented Sep 4, 2018

I am in process of moving these tests to evil-tests as it needs some permission which we do not want to enable on production.

bizybot referenced this issue in bizybot/elasticsearch Sep 7, 2018
We have a test dependency on Apache Mina when using SimpleKdcServer
for testing Kerberos. When checking for ldap backend connectivity,
the code checks for deadlocks which require additional security
permissions `accessClassInPackage.sun.reflect`. As this is only for
test and we do not want to add security permissions to production,
this commit moves these tests and related classes to
x-pack evil-tests where they can run with security manager disabled.
The plan is to handle the security manager exception in the upstream
and then once the release is available run these tests with security
manager enabled.

Closes#32739
bizybot added a commit that referenced this issue Sep 14, 2018
We have a test dependency on Apache Mina when using SimpleKdcServer
for testing Kerberos. When checking for LDAP backend connectivity,
the code checks for deadlocks which require additional security
permissions accessClassInPackage.sun.reflect. As this is only for
test and we do not want to add security permissions to production,
this commit moves these tests and related classes to
x-pack evil-tests where they can run with security manager disabled.
The plan is to handle the security manager exception in the upstream issue
DIRMINA-1093
and then once the release is available to run these tests with security
manager enabled.

Closes #32739
bizybot added a commit to bizybot/elasticsearch that referenced this issue Sep 14, 2018
We have a test dependency on Apache Mina when using SimpleKdcServer
for testing Kerberos. When checking for LDAP backend connectivity,
the code checks for deadlocks which require additional security
permissions accessClassInPackage.sun.reflect. As this is only for
test and we do not want to add security permissions to production,
this commit moves these tests and related classes to
x-pack evil-tests where they can run with security manager disabled.
The plan is to handle the security manager exception in the upstream issue
DIRMINA-1093
and then once the release is available to run these tests with security
manager enabled.

Closes elastic#32739
@danielmitterdorfer
Copy link
Member

We have another test failure on the 6.4 branch in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.4+matrix-java-periodic/ES_BUILD_JAVA=java10,ES_RUNTIME_JAVA=java8fips,nodes=virtual&&linux/57/consoleFull and I think this is the same error:

23:32:28 ERROR   1.58s J0 | SimpleKdcLdapServerTests.testPrincipalCreationAndSearchOnLdap <<< FAILURES!
23:32:28    > Throwable #1: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessClassInPackage.sun.reflect")
23:32:28   2>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
23:32:28   2>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
23:32:28   2>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
23:32:28    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
23:32:28    > 	at java.security.AccessController.checkPermission(AccessController.java:884)
23:32:28    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
23:32:28    > 	at java.lang.SecurityManager.checkPackageAccess(SecurityManager.java:1564)
23:32:28   2>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
23:32:28    > 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:329)
23:32:28   2>         at org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:112)
[...]

I attempted to cherry-pick your changes to 6.4 but it does not apply cleanly. Can you please backport this to 6.4 @bizybot as you are probably more familiar with the changes than me? Thank you!

bizybot added a commit that referenced this issue Oct 16, 2018
We have a test dependency on Apache Mina when using SimpleKdcServer
for testing Kerberos. When checking for LDAP backend connectivity,
the code checks for deadlocks which require additional security
permissions accessClassInPackage.sun.reflect. As this is only for
test and we do not want to add security permissions to production,
this commit moves these tests and related classes to
x-pack evil-tests where they can run with security manager disabled.
The plan is to handle the security manager exception in the upstream issue
DIRMINA-1093
and then once the release is available to run these tests with security
manager enabled.

Closes #32739
@bizybot
Copy link
Contributor

bizybot commented Oct 16, 2018

Backported change to 6.4 (move SimpleKdc to evil-tests), closing this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) >test-failure Triaged test failures from CI v7.0.0-beta1
Projects
None yet
Development

No branches or pull requests

6 participants