SimpleKdcLdapServerTests testClientServiceMutualAuthentication fails on JDK15-ea

[jaymode]

Build scan: https://gradle-enterprise.elastic.co/s/h2zdsinitnabi

Repro line:

./gradlew ':x-pack:qa:evil-tests:unitTest' -Dtests.seed=81536A3F3B7CCE98 -Dtests.class=org.elasticsearch.xpack.security.authc.kerberos.SimpleKdcLdapServerTests -Dtests.method="testClientServiceMutualAuthentication" -Dtests.security.manager=false -Dtests.locale=mni-Beng-IN -Dtests.timezone=Asia/Calcutta -Dcompiler.java=12 -Druntime.java=15

Reproduces locally?: Yes

Applicable branches: 6.8

Failure history: https://build-stats.elastic.co/app/kibana#/discover?_g=(refreshInterval:(pause:!t,value:0),time:(from:now%2Fy,mode:quick,to:now))&_a=(columns:!(test.failed-testcases),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:b646ed00-7efc-11e8-bf69-63c8ef516157,key:build.project,negate:!f,params:(query:elasticsearch,type:phrase),type:phrase,value:elasticsearch),query:(match:(build.project:(query:elasticsearch,type:phrase))))),index:b646ed00-7efc-11e8-bf69-63c8ef516157,interval:auto,query:(language:lucene,query:'test.failed-testcases.test:testClientServiceMutualAuthentication'),sort:!(process.time-start,desc))

Failure excerpt:

java.security.PrivilegedActionException: java.security.PrivilegedActionException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
   > 	at __randomizedtesting.SeedInfo.seed([81536A3F3B7CCE98:FBA7DB287B5CD03D]:0)
   > 	at java.base/java.security.AccessController.doPrivileged(AccessController.java:558)
   > 	at org.elasticsearch.xpack.security.authc.kerberos.KerberosTestCase.doAsWrapper(KerberosTestCase.java:186)
   > 	at org.elasticsearch.xpack.security.authc.kerberos.SpnegoClient.<init>(SpnegoClient.java:86)
   > 	at org.elasticsearch.xpack.security.authc.kerberos.SimpleKdcLdapServerTests.testClientServiceMutualAuthentication(SimpleKdcLdapServerTests.java:57)
   > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
   > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   > 	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
   > 	at java.base/java.lang.Thread.run(Thread.java:832)
   > Caused by: java.security.PrivilegedActionException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
   > 	at java.base/java.security.AccessController.doPrivileged(AccessController.java:695)
   > 	at java.base/javax.security.auth.Subject.doAs(Subject.java:425)
   > 	at org.elasticsearch.xpack.security.authc.kerberos.KerberosTestCase.lambda$doAsWrapper$2(KerberosTestCase.java:186)
   > 	at java.base/java.security.AccessController.doPrivileged(AccessController.java:554)
   > 	... 39 more
   > Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
   > 	at java.security.jgss/sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:164)
   > 	at java.security.jgss/sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:126)
   > 	at java.security.jgss/sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:174)
   > 	at java.security.jgss/sun.security.jgss.spnego.SpNegoMechFactory.getCredentialElement(SpNegoMechFactory.java:146)
   > 	at java.security.jgss/sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:174)
   > 	at java.security.jgss/sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:439)
   > 	at java.security.jgss/sun.security.jgss.GSSCredentialImpl.<init>(GSSCredentialImpl.java:74)
   > 	at java.security.jgss/sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:135)
   > 	at org.elasticsearch.xpack.security.authc.kerberos.SpnegoClient.lambda$new$1(SpnegoClient.java:87)
   > 	at java.base/java.security.AccessController.doPrivileged(AccessController.java:691)
   > 	... 42 more

This fails reproducibly for me on the 6.8 branch once I installed JDK15. Note, I tested with b26 of JDK15 and this is not the JDK issue that causes #56507.

[elasticmachine]

Pinging @elastic/es-security (:Security/Authentication)

[jaymode]

Some possible causes could be https://bugs.openjdk.java.net/browse/JDK-8160818 and/or https://bugs.openjdk.java.net/browse/JDK-8239385.

[ywangd]

TLDR: This is not really a Kerberos issue, but mainly an issue about localised DateFormat. It manifests itself in this particular test and version due to interactions among Java 15, Apache Kerby library and our test setup.

I spent quite some time digging into this and here are the details:

1. JDK 9 introduced i18n enhancements which makes CLRD the default locale data. It was previously JRE. This broke some of our date related code and was addressed by passing java.locale.providers=COMPAT (Pass java.locale.providers=COMPAT to Java 9 onwards #28080). This is in effect since v6.2. But it does not affect tests as it was added into jvm.options file.
2. [Java.time] Calculate week of a year with ISO rules #48209 added the above setting to tests and this change is for v7.6+, so v6.8 is left out. (The PR also added SPI as a provider but it does not impact the behaviour we are discussing here)
3. But so far the test still worked becausejava.time.format.DateTimeFormatter did not honor the default locale values. But JDK 15 b23 fixed it with JDK-8244245. This means if numbers have different representations other than ASCII digits, i.e. 0-9, they will be honored when a date string is formatted. For an example, 2020 is ২০২০ with Locale mni-Beng-IN (Manipuri India Bangla).
4. When prepare the Kerberos response, the Apache Kerby library uses system locale for date string, but then asks for bytes in US_ASCII encoding. The result bytes will be incorrect when the date representation is an unicode string like ২০২০. The Kerberos response will hence be mal-formatted.
5. JDK's sun.security.krb5.internal.KerberosTime only supports ASCII encoded date strings, e.g. 20200730020307Z, it breaks when it sees the above mal-formatted Kerberos response. This in turn leads to the symptom of this test failure, i.e. java.io.IOException: Parse Generalized time, invalid format.

So in summary, item 2 from above list is the critical piece that is missing from v6.8. This can be fixed by adding java.locale.providers=SPI,COMPAT to the specific gradle file of this test. However, I wonder whether it's better to add it for all tests so they are consistent with production code and v7.6+. I am a bit surprised we haven't seen more failures similar to this for the v6.8 branch. Maybe because we still use Joda time for this branch and it works differently? I'll talk to @elastic/es-core-infra to formulate a solution.

I am also thinking raising an issue for Apache Kerby to always encode the date string in an ASCII compatible way regardless of the system locale. Since JDK's Kerberos implementation does not support anything other than ASCII, it makes more sense to ensure it. Also the Kerberos spec itself does recognise it has issues in supporting full i18n.

[ywangd]

Fixed by #60447
Also raised an issue for Apach Kerby.