The JCK is the Oracle Java Compatibility Kit used to certify that a Java platform implementation conforms to the Java platform specifications.
The JCK is only available to Oracle Licensees.
The test.jck project is test automation capable of executing the non-interactive (non-graphical) JCK tests. Note that this means that if all these automated tests run successfully that does not necessarily mean that an implementation can be certified as Java compatible since the interactive tests which are not executed by this automation must also be executed.
The automation is useful as a means of ensuring consistent JCK test execution across a range of operating systems and hardware so that regressions which would cause an implementation to become not Java Compatible can be detected quickly.
The automation depends on the System Test Framework (STF) project. This also provides the additional capabilities of monitoring the test processes for hangs and unexpected core dumps.
The automation here essentially executes command lines of the form:
<path-to-java-under-test> -jar <path-to-jck>/JCK-runtime-8b/lib/jtlite.jar -config <path-to-aqa-systemtest>/aqa-systemtest/openjdk.test.jck/config/jck8b/runtime.jti @<path-to-jtbfile>
The runtime.jti file in the aqa-systemtest repository is a JCK 'interview' configuration file (name=value pairs) which was originally created manually on an individual test machine.
The JCK harness provides a method for the configuration values to be overridden at test execution time by redefining values via a jtb file (xxx.jtb) and passing the name of that file to the test harness. A number of the values used in the testing are specific to the machine executing the tests. The test automation (Jck.java) creates a unique jtb file for each test run, overriding these values with those appropriate for the current test machine. The jtb file is placed into a JTB directory within the test run's unique results directory. All the test output from the JCK harness itself is also directed to the test results directory.
This is an example command line generated by the automation:
C:/jdk-8/bin/java -jar C:/jck/jck8b/JCK-runtime-8b/lib/jtlite.jar -config C:/Users/userid/git/aqa-systemtest/openjdk.test.jck/config/jck8b/runtime.jti @C:/stf_temp/20171003-120635-Jck/results/JTB/stf_generated.jtb
Some of the JCK tests require additional processes such as rmiregistry or tnameserver processes. The automation also manages those processes as required by the tests.
To run the non-interative JCK tests using this tooling:
- Clone and build the stf (test framework) and aqa-systemtest repositories as described here.
- Set JAVA_HOME to the jdk that needs to be tested.
- Configure the JCK as described in Setup.
- cd aqa-systemtest/build
- make test.jck8b On Ubuntu the default shell is dash (/bin/sh -> dash). The JCK devtools tests run shell scripts which are incompatible with dash and they contain no #!/<:shell> directive. Setting the environment variable SHELL does not change the behaviour, nor does setting the shell for the user (in /etc/passwd). Resetting the /bin/sh symbolic link to bash works, but is obviously not ideal.
The test.jck automation requires that the JCK has first been configured for execution (unzipped and the native ('C') tests built for the platform under test), and placed beneath a root directory with jck/<jck-version> subdirectories.
For example, for JCK release jck8b, this would be the structure required:
<JCK_ROOT>/jck/jck8b/excludes
<JCK_ROOT>/jck/jck8b/JCK-compiler-8b
<JCK_ROOT>/jck/jck8b/JCK-devtools-8b
<JCK_ROOT>/jck/jck8b/JCK-runtime-8b
<JCK_ROOT>/jck/jck8b/natives
The default location for <JCK_ROOT> is the -systemtest-prereqs STF argument value (which may also be specified via the PREREQS_ROOT) makefile value. If a user has opted for the locations recommended in the instructions for the test repositories, this would be the directory structure for the git repositories and the test and prereqs (including the JCK):
/home/user/git/stf
/home/user/git/stf/........ <- stf repository files
/home/user/git/aqa-systemtests
/home/user/git/aqa-systemtests/........ <- aqa-systemtest repository files
/home/user/systemtest_prereqs
/home/user/systemtest_prereqs/junit/........ <- junit files
/home/user/systemtest_prereqs/jck
/home/user/systemtest_prereqs/jck/jck8b <- Java 8 JCK (jck8b) files
/home/user/systemtest_prereqs/......... <- other prereq directories
The JCK tests also require certain configuration files. Some of these are included in the aqa-systemtest repository. For example, for the jck8b JCK release:
test.jck/config/jck8b/compiler.jti
test.jck/config/jck8b/devtools.jti
test.jck/config/jck8b/runtime.jti
Two additional configuration files are required which must be set up according to the test machine environment. There are templates for these files included in the aqa-systemtest repository
test.jck/config/default/jcktest.properties
test.jck/config/default/krb5.conf
The files must be copied to a config/<config-name> subdirectory beneath a systemtest-prereqs directory and the values updated to reflect the test machine setup. If a value other than 'default' is used for config-name then config-name will need to be supplied to the test at execution time.
The simplest way to set up these files is to place them alongside the 'ready to run' JCK itself - for example:
<JCK_ROOT>/jck/jck8b/config/default/jcktest.properties
<JCK_ROOT>/jck/jck8b/config/default/krb5.conf
The correct values are set according to the Kerberos and Http server setup which the JCK tests are to access. The JCK documentation describes how to set up those services.
The natives can be built by executing the GNU makefile test.jck/src/native/makefile as follows:
make PLATFORM=xxxx SRCDIR=<path-to-jck-root>/JCK-runtime-8b OUTPUT=<path-to-jck-root>/natives
make help shows the vaild values for PLATFORM.
The natives will be placed in the OUTPUT/PLATFORM directory.
- Clone and build the stf (test framework) and aqa-systemtest repositories as described here.
- Set JAVA_HOME to the jdk that needs to be tested.
- Configure the JCK as described in Setup.
- cd aqa-systemtest/build
- make <test-target>
The JCK execution model is to provide the test harness with a directory in the test cases tree structure. The harness then executes all tests in that directory and all subdirectories. This provides considerable flexibility in defining which tests are to be executed in a test run.
- Setting a higher level directory means that more tests are run than setting a lower level directory. So the higher the 'start directory' the fewer the number of commands required to run all the tests.
- The higher the 'start directory', the longer the tests take to execute.
A large number (c. 250) of test targets have been created to run the JCK tests.
These targets set the 'start directory' to the root directory of eack of the three major Java 8 JCK components
- i.e. they execute all the tests in that component. Some of these targets take hours to execute.
test.jck8b.runtime.api test.jck8b.runtime.api.noconfig test.jck8b.runtime.lang test.jck8b.runtime.vm test.jck8b.runtime.xml_schema test.jck8b.compiler.api test.jck8b.compiler.lang test.jck8b.devtools.jaxws test.jck8b.devtools.java2schema test.jck8b.devtools.schema_bind test.jck8b.devtools.schema2java test.jck8b.devtools.xml_schema
The test.jck8b.runtime.api.noconfig target executes all the runtime api tests except those which required the Kerberos and Http server setup. So if a user has only unzipped the JCK and built the natives for the platform under test, running target test.jck8b.runtime.api.noconfig rather than test.jck8b.runtime.api will avoid running the tests which would fail because of that missing setup.
The other make test.jcknn.xxxx targets run the next directory level down as the 'start directory'. This provides the ability to run the tests in smaller pieces for instance in a build pipeline, making it easier to determine where in the JCK any test failures lie.
In particular, this target may be useful: test.jck8b.noconfig - Excludes the test suites which require Kerberos and Http server setup
The 'make test' targets execute stf.pl command lines with a predetermined set of command line arguments.
The stf.pl commands can be run directly giving full control of the various options:
cd stf.core/scripts
perl stf.pl -test-root=<path-to-aqa-systemtest> -systemtest-prereqs=<path-to-systemtest-prereqs(;<path-to-JCK)> -test=Jck -test-args="tests=<>,jckversion=<>,testsuite=<>,site=<>,executiontype=<>,withagent=<>"
- tests : Is the name of test package to run.
eg: 'api/java_awt', 'vm/jni', 'lang/ANNOT', 'xml_schema/nistData/list/language'
There are 239 such packages for the JCK version jck8b.
- jckversion : Is the jck testsuite version to use: -- specify jck8b for Java8 -- specify jck9 for Java9 -- specify jck10 for Java10
These are the subdirectories referred to in the setup section above.
-
testsuite: Is the type of the testsuite which we are going to run. There are - RUNTIME|COMPILER|DEVTOOLS.
-
config: This option is to identify the location of the installation specific settings in config/<config-name>/jcktest.properties and /config/<config-name>/krb5.conf. The default value is 'default'. The test packages which use these settings are api/java_net, api/java_nio, api/org_ietf and api/javax_security, so if those tests are excluded there is no need to setup these files. The files contain values for Kerberos and Http services - URLs of machines hosting the services and the authentication and IP details to be passed to the JCK test harness in order to run the tests.
-
executiontype: The stf automation runs tests in multijvm way.
Example command lines:
Eg:
cd /home/user/git/stf/stf.core/scripts
perl stf.pl -test-root="/home/user/git/aqa-systemtest" -systemtest-prereqs="/home/user/systemtest_prereqs" -test=Jck -test-args="tests=api/java_math,jckversion=JCK8B,testsuite=RUNTIME"
- By default, the config is set to "default", executiontype set to "multijvm" and withagent is set to "off"
- The JCK will have been placed under /home/user/systemtest_prereqs/jck.
Or
cd /home/user/git/stf/stf.core/scripts
perl stf.pl -test-root="/home/user/git/aqa-systemtest" -systemtest-prereqs="/home/user/systemtest_prereqs;/home/user/jck_installation" -test=Jck -test-args="tests=api/java_math,jckversion=JCK8B,testsuite=RUNTIME"
- The JCK will have been placed under /home/user/jck_installation/jck.
Help on stf perl command:
perl stf.pl -test=Jck -help
This command will list the detailed explanation for these test arguments and example inputs.
JVM options are passed to the JCK tests by the JCK test harness, which requires that they be placed in a configuration (.jtb) file (along with many other options).
This file is created dynamically during the setup stage of the STF test execution using the .jti files present in the aqa-systemtest repository as a template, and then substituting certain values as determined from interrogating the machine on which the tests are executing, and the jcktest.properties and krb5.conf configuration files.
So in order to pass extra JVM options to the JCK tests, the values must be passed via the -java-args-setup argument rather than the normal -java-args argument. Eg:
perl stf.pl -vv -test=Jck -test-args="tests=vm/cldc,jckversion=jck8b,testsuite=runtime,site=bangalore,executiontype=multijvm,withagent=off" -java-args-setup="-Xverify:nofallback"
To pass the extra JVM options to the JCK harness running the tests, use the normal '-java-args' option. Eg:
perl stf.pl -vv -test=Jck -test-args="tests=vm/cldc,jckversion=jck8b,testsuite=runtime,site=bangalore,executiontype=multijvm,withagent=off" -java-args="-Xmx512M"
The JCK has seven categories of tests:
- Compiler.API Api tests
- Compiler.Lang Language specification tests
- Devtools Java to schema : schema to Java - mapping tests, JAX-WS tests, Schema binding tets , Tests on W3C XML Schema test suite.
- Runtime.API Api tests
- Runtime.VM Interpreter tests
- Runtime_Lang Language specification tests
- Runtime_Xml_Schema Tests based on W3C XML Schema test suite
The JCK once unzipped contains three high level directories which contain the test suites that form the JCK:
- JCK-complier
- JCK-devtools
- JCK-runtime
Following the JCK setup procedure described above, the following directories will have been added over and above those received via the JCK distribution:
- excludes : This folder contains the exclude and known failure list for each JCK test suite.
- natives : This folder contains the platform specific native libraries for tests in JCK-runtime written in native code.
- config : This folder contains settings for tests which require access to http and Kerberos services
The JCK contains extensive documentation regarding the test suites and the harness used to execute the tests. This should be read. In particular:
- Inside the doc/jck directory of all the test suites there is a JCK_Users_Guide.pdf.
- Inside the doc/javatest folder of all the test suites there are the files javatestGUI.pdf, javatestCLI.pdf, javatestAgent.pdf.
The Java Compatibility Kit is only available to JCK licensees. The code in this test.jck project does not require the JCK to be present in order to execute - it just tries to run the JCK from the <JCK_ROOT>/jck location. If the JCK cannot be found (for instance if you execute the tests but you do not have a copy of the JCK because you are not a licensee), the tests will fail with an appropriate 'JCK not found' error message.
Running the JCK tests consists of the following basic operations:
- Starting the JavaTest Harness
- Generating a Configuration file -- Sample configuration (.jti) files are available inside Runtime, Compiler and Devtools test suites under linux, win32 and solaris folders for reference.
- Executing the tests
- Debugging test failures
The JCK has excellent tools for configuring the tests and executing them either interactively or from a command line on a given test machine. For users with access to the JCK and the machine on which the tests are to execute it is recommended that you familiarise yourself with these tools.
The JCK JavaTest harness also provides tools to help identify the cause of test execution failures.
Remember that when the stf test.jck automation runs (stf.pl -test=Jck) the configuration files for the test are generated along with setup.pl and execute.pl scripts which contain the command lines to run the tests. They are placed in the unique results directory for the test (by default /tmp/stf/date-time-test or C:\stf_temp\date-time-test). These can be very useful should a user want to rerun the test in a different way such as with different JVM options or a smaller subset of tests - simply locate the command in the generated setup.pl or execute.pl file, modify it and re-execute the script.