[CI] Secure Hdfs Fixture in JDK16

[albertzaharovits]

Build scan: https://gradle-enterprise.elastic.co/s/plztmz7djijsu

Repro line: ./gradlew ':x-pack:plugin:searchable-snapshots:qa:hdfs:integTestSecure' --tests "org.elasticsearch.xpack.searchablesnapshots.hdfs.HdfsSearchableSnapshotsIT" -Dtests.security.manager=true -Druntime.java=16

Reproduces locally?: Yes

Applicable branches: 7.x, master

Failure history:
https://build-stats.elastic.co/app/kibana#/discover?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-90d,mode:quick,to:now))&_a=(columns:!(build.job-name,build.branch),index:b646ed00-7efc-11e8-bf69-63c8ef516157,interval:auto,query:(language:lucene,query:HdfsSearchableSnapshotsIT),sort:!(process.time-start,desc))
Failure excerpt:

  [run]	
    "/var/lib/jenkins/.java/openjdk16/bin/java" "$@" > run.log 2>&1 ; if [ $? != 0 ]; then touch run.failed; fi	
secureHdfsFixture output:	
-----------------------------------------	
  failure marker exists: true	
  pid file exists: false	
  ports file exists: false	
	
  [log]	
    Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8	
    Exception in thread "main" java.lang.IllegalArgumentException: Can't get Kerberos realm	
    	at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:65)	
    	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:296)	
    	at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:342)	
    	at hdfs.MiniHDFS.main(MiniHDFS.java:99)	
    Caused by: java.lang.IllegalAccessException: class org.apache.hadoop.security.authentication.util.KerberosUtil cannot access class sun.security.krb5.Config (in module java.security.jgss) because module java.security.jgss does not export sun.security.krb5 to unnamed module @123a3fe3	
    	at java.base/jdk.internal.reflect.Reflection.newIllegalAccessException(Reflection.java:385)	
    	at java.base/java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:687)	
    	at java.base/java.lang.reflect.Method.invoke(Method.java:559)	
    	at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:107)	
    	at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:63)

[elasticmachine]

Pinging @elastic/es-security (Team:Security)

[albertzaharovits]

@jbaiera Do you have any hunches about this?

[jbaiera]

Seems related to #66972. I put in a fix for this for JDK 16 builds a little while back (#67391) but it seems that the issue has resurfaced. Looking at the command line for the secured hdfs fixture, it lacks the open exports argument that the fix PR was supposed to add. I suspect some small detail is keeping that from being added in the build. I'll investigate.

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[cbuescher]

Some more today: https://gradle-enterprise.elastic.co/s/mvhntb4vvwnbc

[mark-vieira]

Looking at this again, I don't see a compelling reason why the fixture should be running with RUNTIME_JAVA_HOME. That's not what we're testing here. We are testing that Elasticsearch behaves properly on Java 16, not these fixtures.

That said, they will eventually need to support Java 16 as we'll switch the build to that as soon as 16 goes GA and Gradle has support so we should get this sorted regardless.

[mark-vieira]

@jbaiera FYI, I've disabled these tests (and fixtures) on Java 16 for now via #68182 on both master and 7.x. Make sure to revert that change in both branches once a fix is in for this.

[ywangd]

@mark-vieira It seems the mute didn't work. It still failed on master https://gradle-enterprise.elastic.co/s/nydha2cre3ntu

[mark-vieira]

Good catch, looks like the searchable-snapshots project also uses that fixture. I'll open another PR to mute that one as well.

[mark-vieira]

Searchable snapshots secure tests also disabled on Java 16 with #68189. You'll want to revert that as well when this gets fixed.

[masseyke]

This one seems to be fixed, I'm pretty sure due to #78407.

[jbaiera]

Indeed. This should be remediated now. Closing!