My attempt at an outside-in testing example in Java using a Tomcat servlet container.
Source: https://github.com/mbland/tomcat-servlet-testing-example
This project incorporates the String Calculator kata to demonstrate Test-Driven Development in the context of a Test Pyramid-based testing strategy.
Though I've been a programmer for years across many other programming languages, I'm learning a lot about the Java ecosystem for the first time. It will take some time before this project looks like a complete working example that makes sense.
The plan is to develop an exercise comprised of the following steps:
- Set up a continuous integration pipeline based on GitHub Actions.
- Creating a walking skeleton implementation and adding a large end-to-end test to validate it, likely using Selenium WebDriver as well as headless Chrome.
- Developing the String Calculator using TDD and small unit tests.
- Adding a medium integration test to ensure the Servlet passes parameters to the internal String Calculator logic and passes back the results.
- Adding tests for frontend JavaScript components.
- Using test doubles in unit tests. This may involve extending the String Calculator example or adding a completely different one, possibly based on Apache Solr.
I've got an initial Hello, World!
servlet running under Tomcat, validated by
straightforward medium tests and an initial Selenium WebDriver test, all running
under JUnit. Everything runs from the command line, IntelliJ IDEA, and VSCode
(all on macOS), and in the GitHub Actions continuous integration pipeline.
The next step is to add a proper HTML <form> and WebDriver test to complete the walking skeleton (i.e., a complete, minimally functional application deployment).
I run Arm64/aarch64 builds of Ubuntu Linux and Windows 11 Professional
under Parallels Desktop for Mac on Apple Silicon. vite.config.js
contains
a workaround to allow WebDriver to use Chromium or Firefox on Linux, as
no aarch64 build of Google Chrome is available. (I hope to contribute
this workaround upstream, at which point I'll remove it from vite.config.js
.)
Also, it doesn't appear as though nested virtualzation will ever be supported by the aarch 64 Windows 11 on an Apple M1. This means the Docker-based tests won't work in this situation.
FWIW, Windows Subsystem for Linux 2 also requires nested virtualization. WSL 1 will work fine, but it appears prospects to run Docker in WSL 1 are rather dim.
This project uses the Java® Platform, Standard Edition & Java Development Kit Version 21.
On macOS and Linux, I installed the latest JDK 21.0.1 from Eclipse Temurin™ Latest Releases via SDKMAN!. On Windows, I downloaded it from the Download the Microsoft Build of OpenJDK page.
See strcalc/src/main/frontend/README.md for guidance on setting up the frontend development and build environment.
The frontend environment isn't required for ./gradlew test
, which runs the
small Java unit tests from the exercise. Conversely, the Java environment isn't
strictly required to develop, build, and test the frontend in isolation.
However, you will need to have both environments installed to build most of the
Gradle targets. This includes the medium integration and large system tests in
strcalc/src/test/java
, which also depend on the frontend build.
Optional: Install the Tomcat servlet container
This step is optional, as the bin/tomcat-docker.sh script will run Tomcat locally in a Docker container defined by dockerfiles/Dockerfile.tomcat-test.
I followed the Tomcat 10.1 Setup instructions to install Tomcat locally
at /opt/tomcat/apache-tomcat-10.1.16
. I created bin/tomcat.sh
as a thin
wrapper around Tomcat's bin/catalina.sh
that detects and sets JAVA_HOME
and sets CATALINA_HOME
. Verified that it was installed correctly via
bin/tomcat.sh start
and visiting http://localhost:8080/.
This step is optional, as the Docker container launched by bin/tomcat-docker.sh is already configured to do this by running bin/update-tomcat-config-logging.sh.
By default, Tomcat emits its HTTP access logs to
$CATALINA_HOME/logs/localhost_access_log.YYYY-MM-DD.txt
. This is configured by
the following block in $CATALINA_HOME/conf/server.xml
:
<!-- Access log processes all example.
Documentation at: /docs/config/valve.html
Note: The pattern used is equivalent to using pattern="common" -->
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="localhost_access_log" suffix=".txt"
pattern="%h %l %u %t "%r" %s %b" />
When running and experimenting locally, it can be helpful and convenient to emit
these logs directly to the terminal instead. To emit the HTTP access logs to
standard output instead, edit or replace this <Valve>
element with the
following:
<Valve className="org.apache.catalina.valves.AccessLogValve"
directory="/dev" prefix="stdout"
suffix="" rotatable="false" buffered="false"
pattern="%h %l %u %t "%r" %s %b" />
(The above is based on the answer to "Stack Overflow: Docker, Tomee, Logging, STDOUT, AWS".)
Alternatively, if you're into the YOLO thing, apply this update by running:
bin/update-tomcat-config-logging.sh $CATALINA_HOME/conf/server.xml
For more information on Tomcat access logging, see:
Optional: Install Gradle
This step is optional, as the gradlew and gradlew.bat wrapper scripts in the root directory of the repository will install Gradle.
On macOS and Linux, I installed Gradle 8.4 via SDKMAN!. On Windows, I followed the Gradle manual installation steps.
(When I started the project, this also required installing the 20.0.2 JDK, since Gradle 8.4 could build Java 21, but needed Java 20 to run. At some point this no longer appeared necessary, though the Gradle compatibility matrix for Java has yet to reflect this. The project now uses Gradle 8.5, which is explicitly supported on Java 21, but it's still important to know about this potential mismatch.)
After running gradle init
to create a library as part of the Gradle
Tutorial:
- Renamed the
lib
directory tostrcalc
. - Replaced the
java-library
plugin in strcalc/build.gradle.kts withwar
and made other changes per the Gradle WAR plugin. - Adjusted
gradle/libs.versions.toml
accordingly, including enabling aprovided
dependency on the desired Jakarta Servlet Specification version.
Install IntelliJ IDEA and/or Visual Studio Code
The project supports both IntelliJ IDEA and VSCode.
I recently used VSCode extensively to develop my EListMan project in Go, but this being Java, I primarly use IntelliJ IDEA. (I still prefer to edit this file and other Markdown files in VSCode, however.)
I installed all the extensions from the VSCode: Extension Pack for Java except for Visual Studio IntelliCode.
A lot of VSCode references refer to Java: Configure Java Runtime in the Command Palette, but I can't find it across macOS, Linux, or Windows. I can't seem to find any online resources from anyone else for whom it also doesn't exist.
I did eventually find Managing Java Projects in VS Code: Configure Runtime
for Projects, which provides guidance on how to configure Java via
settings.json
. On macOS and Linux, my config looks like the following
(replacing $HOME
with my actual home directory):
"java.configuration.runtimes": [
{
"name": "JavaSE-21",
"path": "$HOME/.sdkman/candidates/java/21.0.1-tem",
"sources" : "$HOME/.sdkman/candidates/java/21.0.1-tem/lib/src.zip",
"javadoc" : "https://docs.oracle.com/en/java/javase/21/docs/api",
"default": true
}
],
and on Windows:
"java.configuration.runtimes": [
{
"name": "JavaSE-21",
"path": "C:\\Program Files (Arm)\\Microsoft\\jdk-21.0.1.12-hotspot",
"sources" : "C:\\Program Files (Arm)\\Microsoft\\jdk-21.0.1.12-hotspot\\lib\\src.zip",
"javadoc" : "https://docs.oracle.com/en/java/javase/21/docs/api",
"default": true
}
]
- Getting Started with Java in VS Code
- Java build tools in VS Code
- VSCode: Gradle for Java
- VSCode: Navigate and edit Java source code
- Testing Java with Visual Studio Code
Optional: Create Tomcat > Local run configuration
This step is optional, as the bin/tomcat-docker.sh script will run Tomcat locally in a Docker container defined by dockerfiles/Dockerfile.tomcat-test.
In IntelliJ, you can run your locally installed Tomcat server via Run > Edit Configurations... > Add New Configuration (+) > Tomcat Server > Local.
- Under Server > Before launch, remove the "Build" item and add a new "Build Artifacts" task.
- From the popup window for that operation, select: Build 'Gradle : tomcat-servlet-testing-example : strcalc.war' artifact.
This installs the WAR file (as described by the Tomcat deployment document)
under the /strcalc
context (i.e., Tomcat serves the servlet at
http://localhost:8080/strcalc).
Write "Hello, World!" servlet and JUnit test
The initial servlet responds with the classic hardcoded value Hello, World!
.
The initial test expects Tomcat to be running and for
http://localhost:8080/strcalc to return this value.
Note that the value of the @WebServlet
annotation is /add
. Tomcat appends
this value to the context path (/strcalc
), creating the servlet path
/strcalc/add
.
- Stack Overflow: Difference between / and /* in servlet mapping url pattern
- Introduction to Servlets
- Exploring the New HTTP Client in Java
- The Java EE 6 Tutorial: Java Servlet Technology
Add a bin/tomcat-docker.sh
script to launch the Tomcat Docker image
This script can be run manually.
The full path is com.mike_bland.training.testing.utils.LocalServer
.
This class runs git
and docker
commands to emulate the
bin/tomcat-docker.sh
script on demand for
StringCalculatorTomcatContractTest
. A key difference is that LocalServer
will allocate a unique port for every test run, so that it won't conflict with
an existing local instance.
These are JUnit composed annotations based on the guidance from:
These tasks and updates use JUnit composed annotations and the includeTags
config option based on guidance from:
Note that with no @SmallTest
methods defined, running the test
task produces
the following warning:
$ ./gradlew test --warning-mode all --rerun-tasks
[...snip...]
> Task :strcalc:test
No test executed. This behavior has been deprecated. This will fail with an
error in Gradle 9.0. There are test sources present but no test was executed.
Please check your test configuration. Consult the upgrading guide for further
information:
https://docs.gradle.org/8.4/userguide/upgrading_version_8.html#test_task_fail_on_no_test_executed
[...snip...]
The same is true for test-medium
and test-large
when no @MediumTest
or
@LargeTest
methods are present. The SmallPlaceholderTest
and
LargePlaceholderTest
classes exist to silence this warning until actual
@SmallTest
and @LargeTest
methods appear.
After syncing the Gradle project, you can use IntelliJ IDEA's Gradle integration to convert the Gradle test tasks from strcalc/build.gradle.kts directly into IntelliJ run configurations:
- Navigate to View > Tool Windows > Gradle or click the Gradle elephant logo in the right side toolbar to open the Gradle tool window.
- Navigate through the Gradle project hierarchy and select the target you'd like
to turn into a run configuration, e.g.,
test
. - Double click the target or hit enter/return to create and run the run configuration for that target.
- Open the run configuration and modify it as desired.
Doing this instead of creating a typical JUnit run configuration ensures
that IntelliJ runs the same tests the same way as the gradlew
command. This,
in turn, helps ensure that IntelliJ runs the same tests the same way as they're
run in continuous integration. This includes running with the same code coverage
configuration and output. See Setting up continuous integration
below.
You can still create JUnit run configurations, but these gradlew
targets
already ensure that all dependencies of the task are built first. This is
particularly helpful when it comes to medium and large tests.
-
Specifically for this project, the medium and large tests depend on the
strcalc.war
artifact, used to build a temporary Tomcat Docker image for testing. The Gradle test tasks are already configured to depend on thewar
task. -
Setting this up in a JUnit run configuration would require setting up Modify options (⌥M) > Before Launch: Add before launch task > Run Gradle task.
Code coverage is always generated for these targets (though not automatically displayed) when using the regular Run command. To display this output, see Showing JaCoCo coverage generated by Gradle tasks.
Viewing the coverage generated by the Gradle test
task in IntelliJ may or may
not work. See the Viewing code coverage in IntelliJ IDEA
section below.
Per the Gradle WAR plugin, files in src/main/webapp
are copied to the root
of the servlet WAR file. Adding the src/main/webapp/index.html
landing page
necessitated two things:
- Updating the
@WebServlet
annotation onStringCalculatorServlet
from""
to"/add"
. This enables Tomcat to serveindex.html
as/strcalc/
, and the servlet now serves/strcalc/add/
. - Updating the
helloWorldPlaceholder
test method tolandingPageHelloWorld
. This test now checks that the response is of Content-Typetext/html
instead oftext/plain
, and that it contains"Hello, World!"
, not that it matches exactly.
Changing the test assertion to check that the response body contains "Hello, World!"
necessitated using the Hamcrest matcher library. We could've used:
assertTrue(resp.body().contains("Hello, World!"));
But when it fails, we get very little information about the problem:
Expected :true
Actual :false
Using the Hamcrest matcher:
assertThat(resp.body(), containsString("Hello, World!"));
we're able to see more information:
java.lang.AssertionError:
Expected: a string containing "Hello, World!"
but: was "<DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>String Calculator - Mike Bland Training</title>
</head>
<body>
<p>Hello, world!</p>
</body>
</html>
"
Setting up continuous integration using GitHub Actions
Added the .github/CODEOWNERS file and .github/workflows/run-tests.yaml file for GitHub Actions. Configured using the setup-java GitHub Actions plugin and the gradle/gradle-build-action GitHub Actions plugin.
run-tests.yaml runs gradlew merge-test-reports
to aggregate individual TEST-*.xml
JUnit results files
into TESTS-TestSuites.xml
files. This task uses org.apache.ant:ant-junit
based on advice from Merging Per-Suite JUnit Reports into Single File with
Gradle + Kotlin. (See also: Using Ant from Gradle.)
The aggregated files are uploaded via the actions/upload-artifact GitHub Actions plugin. .github/workflows/publish-test-results.yaml then uses the dorny/test-reporter GitHub Actions plugin. That plugin downloads the test results and makes them available via the status icon next to the commit hash in the GitHub UI.
As explained by the dorny/test-reporter page, the separate
publish-test-results.yaml
file is necessary to ensure the plugin has
permission to launch a check run:
When someone pushes code to the repository, GitHub automatically sends the
check_suite
event with an action ofrequested
to all GitHub Apps installed on the repository that have thechecks: write
permission.
Not having checks:write
permission for the plugin can lead to the cryptic error:
Error: HttpError: Resource not accessible by integration
For more on GITHUB_TOKEN
permissions, see Assigning permissions to jobs.
Code coverage shows which lines of code are executed by a test or test suite and which aren't. It is one common and important component of a suite of Vital Signs to ensure high code and software quality.
JaCoCo (i.e., Java Code Coverage) is a popular framework for collecting code coverage in Java. IntelliJ IDEA can display code coverage statistics and display which lines are covered, partially covered, and uncovered. It can generate and consume JaCoCo coverage in addition to its own builtin coverage implementation.
-
Code coverage does not indicate that you have enough tests, or that your tests are any good, or they're validating what you expect they are. It can definitively tell whether specific code is exercised by a test (or any test at all), which could reveal gaps in your test suite. Used skillfully, it can be of great use in the process of refactoring, making small changes that improve code quality while preserving behavior. (Using code coverage in this way requires a set of techniques that I hope to cover in their own separate training module(s).)
-
The ideal is to have high code coverage from your entire small (or small-ish) test suite. Each test should cover relatively little, very specific areas of the code. Larger tests should not generate code coverage, as such tests by definition will execute broader areas of the code, diluting the value of the measurements.
The way to think about this is to think of the information available when a smaller test fails versus a larger test:
-
When a smaller test fails, the potential area of the code responsible for the failure is fairly narrow.
-
When a larger test fails, the potential area of the code responsible for the failure is potentially the entire application.
-
-
Also consider the time required to run them: Smaller tests run multiple orders of magnitude faster than larger tests. Having high code coverage from a suite of highly targeted smaller tests that you can run frequently while developing enables tighter feedback and recovery loops. This translates into higher confidence and faster velocity.
On the contrary, larger tests have a broader scope, consume more resources, and run more slowly. While still critically important, these qualities make running larger tests frequently while coding impractical. They should be used sparingly to validate higher level systemic properties, as using them to validate lower level details renders them brittle.
Therefore, code coverage from larger tests does practically nothing to tighten feedback loops and to help resolve coding errors quickly, rendering it far less useful.
-
That said, if you do choose to collect code coverage from all tests, take care to partition the reports by test size. Coverage from the larger test suites should be less than or equal to that from the smaller test suite—not the other way around!
Relying on code coverage percentages inflated by coverage from large tests risks instilling a false sense of security. It could potentially mask gaps in smaller test coverage that could in turn mask serious, yet preventable problems.
Enabling the Gradle JaCoCo Plugin and generating HTML and XML reports
Adding the Gradle JaCoCo Plugin to the strcalc/build.gradle.kts file was
straightforward. With the plugin enabled, the test
task automatically
generates JaCoCo coverage data at strcalc/build/jacoco/test.exec
. Then the
jacocoTestReport
task creates an HTML report viewable via:
open -a safari strcalc/build/reports/jacoco/test/html/index.html
The jacocoXmlTestReport
task generates an XML report which our GitHub Actions
continuous integration system will publish. For details on publishing
options, see:
The options for viewing code coverage directly in IntelliJ IDEA include:
- Using a run configuration to generate and display code coverage
- Showing JaCoCo coverage generated by Gradle tasks
For information on IntelliJ's code coverage options and operations beyond what's described below, see IntelliJ Idea: Code coverage.
IntelliJ allows you to configure specific run configurations to generate and display code coverage via Run > Run CONFIGURATION_NAME with Coverage. It allows you to use wither JaCoCo or IntelliJ's own builtin code coverage implementation. You can configure the Gradle run configurations or any JUnit run configuration you define yourself.
For instructions, see IntelliJ: Run with coverage.
- NOTE as of 2023-11-05: For now, I recommend configuring your run configurations to use IntelliJ's builtin coverage implementation. See the JaCoCo and Java version compatibility between the project and IntelliJ section below for details.
The JaCoCo coverage results automatically generated by the Gradle test tasks won't automatically display in IntelliJ, but you can import them. The IntelliJ IDEA: Manage coverage suites documentation covers most of the process, but specifically for this project:
- Open Run > Show Coverage Data... (⌥⌘F6) to open the Choose Coverage Suite to Display window.
- Click the + symbol to open the file picker.
- Select
strcalc/build/jacoco/test.exec
and click the Open button. - Make sure the JaCoCo Coverage > test.exec item is checked and click the Show Selected button.
Code coverage markers should now appear in the left gutter of the code editor of
each src/main/java
file (green for covered, red for uncovered).
-
If no coverage markers appear, see the JaCoCo and Java version compatibility between the project and IntelliJ section below.
-
NOTE as of 2023-11-05: The coverage markers from this process won't appear in IntelliJ for this project at present.
Here are a few fine details to be aware of:
-
The coverage percentage reported from IntelliJ's builtin coverage reporter will differ from JaCoCo's slightly, as documented in:
As mentioned in that answer, it's not that one version is wrong and the other is right. Differences in implementation account for differences in reported percentages, particularly when it comes to differences in counting implicitly generated methods. Both will still show which code is or isn't covered in every source file.
-
However, the Principle of Least Surprise could apply here. Relying on JaCoCo's results is likely to avoid misunderstandings between coverage percentages reported in IntelliJ and from command line and continuous integration runs.
-
The Run > Show Coverage Data... (⌥⌘F6) option actually allows you to show both IntelliJ and JaCoCo coverage results together, or to switch between them. It also allows you to select
.exec
files from different Gradle subprojects and test targets, as well as coverage from different run configurations in general.
The JaCoCo version bundled with IntelliJ may not be the latest available. This can prevent JaCoCo coverage from appearing in IntelliJ if your project uses a newer version of Java than the builtin JaCoCo version can support. There appears to be no way for users to update the JaCoCo version bundled with IntelliJ.
-
As of 2023-11-05, this project uses Java 21.0.1-temurin and JaCoCo 0.8.11 (defined in strcalc/build.gradle.kts). Per the JaCoCo Change History and the following mailing list message, Java 21 requires at least JaCoCo 0.8.9:
However, my current IntelliJ IDEA version 2023.2.4 ships with JaCoCo 0.8.8.202204050719/5dcf34a. There's an open issue requesting an upgrade to v0.8.11:
If you encounter this problem, I recommend using IntelliJ's builtin in coverage in your run configurations until a new IntelliJ release resolves the issue. Just be aware, as noted above, that while the reported percentages may differ from JaCoCo, the code editor coverage markings should be largely the same.
This problem will manifest by producing output similar to the following (edited for readability) when using:
-
Run > Run CONFIGURATION_NAME with Coverage: this output will appear in the output pane
-
Run > Show Coverage Data... (⌥⌘F6): this output will appear in the IntelliJ debug log. You can find this log via: Help > Show Log in Finder.
Note that this problem will not produce a modal error window! If all the percentages in the Coverage pane show
100% (0/0)
, this is likely the culprit.
2023-11-05 14:47:39,944 [6679894] INFO - #c.i.c.JaCoCoCoverageRunner -
Error while analyzing .../strcalc/build/classes/java/.../*.class
with JaCoCo 0.8.8.202204050719/5dcf34a.
java.io.IOException:
Error while analyzing .../strcalc/build/classes/java/.../*.class
with JaCoCo 0.8.8.202204050719/5dcf34a.
at org.jacoco.core.analysis.Analyzer.analyzerError(Analyzer.java:163)
at org.jacoco.core.analysis.Analyzer.analyzeClass(Analyzer.java:135)
[ ...snip... ]
Caused by: java.lang.IllegalArgumentException:
Unsupported class file major version 65
at org.objectweb.asm.ClassReader.<init>(ClassReader.java:199)
at org.objectweb.asm.ClassReader.<init>(ClassReader.java:180)
[ ...snip... ]
Publishing continuous integration coverage results to Coveralls
Coveralls is a commercial product for visualizing current code coverage and historical trends. It's free for all open source projects, and offers pricing tiers for commercial customers. See the tomcat-servlet-testing-example Coveralls report (also linked from the coverage badge at the top of this file) to see Coveralls in action.
The continuous integration system publishes JaCoCo code coverage results to Coveralls using the coverallsapp/github-action GitHub Actions plugin. Coveralls will make the coverage results for the tested commit available via the status icon next to the commit hash in the GitHub UI. It will also add comments to pull requests summarizing how the changes affect code coverage.
If you prefer not to use Coveralls, you can search the GitHub Actions marketplace for JaCoCo related GitHub Actions plugins. A couple of promising ones are:
Note that, though the instructions of neither of the above plugins show it, you may need to split the configuration similarly to the dorny/test-reporter plugin:
- Use the actions/upload-artifact GitHub Actions plugin at the end of the
.github/workflows/run-tests.yaml file to upload
strcalc/build/reports/jacoco/jacocoXmlTestReport/jacocoXmlTestReport.xml
. - Set the
checks: write
permission in .github/workflows/publish-test-results.yaml so the coverage results can be posted as part of a check run. - Use the actions/download-artifact GitHub Actions plugin in the .github/workflows/publish-test-results.yaml file to download the report.
- Configure the selected plugin to process the downloaded
jacocoXmlTestReport.xml
file.
It's possible to develop the frontend code served by Vite and the backend code served by Tomcat simultaneously.
For now, this requires running a fresh build of the app in Docker or setting up your own IntelliJ IDEA Run Configuration. I hope to have another solution set up shortly that would require neither.
# Compile the current code into strcalc.war.
./gradlew build
# Serve the strcalc.war file via Docker
./bin/tomcat-docker.sh
This is enabled by the CORSFilter
settings in the application's
web.xml file. See the comment
in that file for further references.
cd strcalc/src/main/frontend
STRCALC_BACKEND='http://localhost:8080/strcalc/' pnpm dev
In preview mode (pnpm build && pnpm preview
), the STRCALC_BACKEND value
will not propagate to the compiled bundle. However, entering the
following in the browser console will enable the compiled version to
communciate with the backend:
globalThis.STRCALC_BACKEND='http://localhost:8080/strcalc/'
Coming soon...
TODO(mbland): Document how the following are configured:
- Gradle WAR Plugin - now writes to/includes files from
strcalc/build/webapp
- Selenium WebDriver - include references to:
- TestTomcat (for medium tests)
Coming soon...
© 2023 Mike Bland <[email protected]> (https://mike-bland.com/)
This software is made available as Open Source software under the Mozilla Public License 2.0. For the text of the license, see the LICENSE.txt file. See the MPL 2.0 FAQ for a higher level explanation.