Skip to content

Commit

Permalink
Add support for Java via JMH
Browse files Browse the repository at this point in the history
  • Loading branch information
michaelmior committed Sep 22, 2022
1 parent 5f23f8d commit e5b1a19
Show file tree
Hide file tree
Showing 12 changed files with 472 additions and 2 deletions.
37 changes: 37 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -293,6 +293,43 @@ jobs:
fail-on-alert: true
- run: node ./dist/scripts/ci_validate_modification.js before_data.js 'Benchmark.Net Benchmark'

jmh:
name: Run JMH Java Benchmark Framework example
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-node@v1
- uses: actions/setup-java@v2
with:
distribution: 'adopt'
java-version: '11'
- uses: actions/cache@v1
with:
path: ~/.npm
key: ${{ runner.os }}-node-${{ hashFiles('package-lock.json') }}
- run: npm ci
- run: npm run build
- name: Save previous data.js
run: |
git fetch origin gh-pages
git checkout gh-pages
cp ./dev/bench/data.js before_data.js
git checkout -
- name: Run benchmark
run: |
cd examples/java
mvn clean verify
java -jar target/benchmarks.jar -wi 1 -i 3 -f 1 -rf json
- name: Store benchmark result
uses: ./
with:
name: JMH Benchmark
tool: "jmh"
output-file-path: examples/java/jmh-result.json
skip-fetch-gh-pages: true
fail-on-alert: true
- run: node ./scripts/ci_validate_modification.js before_data.js 'JMH Benchmark'

only-alert-with-cache:
name: Run alert check with actions/cache
runs-on: ubuntu-latest
Expand Down
35 changes: 35 additions & 0 deletions .github/workflows/java.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: JMH Example
on:
push:
branches:
- master

jobs:
benchmark:
name: Run JMH benchmark example
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-java@v2
with:
distribution: 'adopt'
java-version: '11'
- name: Run benchmark
run: |
cd examples/java
mvn clean verify
java -jar target/benchmarks.jar -wi 1 -i 3 -f 1 -rf json
- name: Store benchmark result
uses: benchmark-action/github-action-benchmark@v1
with:
name: JMH Benchmark
tool: 'jmh'
output-file-path: examples/java/jmh-result.json
# Use personal access token instead of GITHUB_TOKEN due to https://github.community/t/github-action-not-triggering-gh-pages-upon-push/16096
github-token: ${{ secrets.GITHUB_TOKEN }}
auto-push: true
# Show alert with commit comment on detecting possible performance regression
alert-threshold: '200%'
comment-on-alert: true
fail-on-alert: true
alert-comment-cc-users: '@michaelmior'
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,5 @@
/coverage
/dist
/.idea
target/
jmh-result.json
7 changes: 6 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ This action currently supports the following tools:
- [BenchmarkTools.jl][] for Julia packages
- [Benchmark.Net][benchmarkdotnet] for .Net projects
- [benchmarkluau](https://github.com/Roblox/luau/tree/master/bench) for Luau projects
- [JMH][jmh] for Java projects
- Custom benchmarks where either 'biggerIsBetter' or 'smallerIsBetter'

Multiple languages in the same repository are supported for polyglot projects.
Expand All @@ -48,6 +49,7 @@ definitions are in [.github/workflows/](./.github/workflows) directory. Live wor
| C++ (Catch2) | [![C++ Catch2 Example Workflow][catch2-badge]][catch2-workflow-example] | [examples/catch2](./examples/catch2) |
| Julia | [![Julia Example][julia-badge]][julia-workflow-example] | [examples/julia](./examples/julia) |
| .Net | [![C# Benchmark.Net Example Workflow][benchmarkdotnet-badge]][benchmarkdotnet-workflow-example] | [examples/benchmarkdotnet](./examples/benchmarkdotnet) |
| Java | [![Java Example Workflow][java-badge]][java-workflow-example] | [examples/java](./examples/java) |
| Luau | Coming soon | Coming soon |

All benchmark charts from above workflows are gathered in GitHub pages:
Expand Down Expand Up @@ -354,7 +356,7 @@ Name of the benchmark. This value must be identical across all benchmarks in you
- Default: N/A

Tool for running benchmark. The value must be one of `"cargo"`, `"go"`, `"benchmarkjs"`, `"pytest"`,
`"googlecpp"`, `"catch2"`, `"julia"`, `"benchmarkdotnet"`,`"benchmarkluau"`, `"customBiggerIsBetter"`, `"customSmallerIsBetter"`.
`"googlecpp"`, `"catch2"`, `"julia"`, "jmh", `"benchmarkdotnet"`,`"benchmarkluau"`, `"customBiggerIsBetter"`, `"customSmallerIsBetter"`.

#### `output-file-path` (Required)

Expand Down Expand Up @@ -600,6 +602,7 @@ Every release will appear on your GitHub notifications page.
[cpp-badge]: https://github.com/benchmark-action/github-action-benchmark/workflows/C%2B%2B%20Example/badge.svg
[catch2-badge]: https://github.com/benchmark-action/github-action-benchmark/workflows/Catch2%20C%2B%2B%20Example/badge.svg
[julia-badge]: https://github.com/benchmark-action/github-action-benchmark/workflows/Julia%20Example%20with%20BenchmarkTools.jl/badge.svg
[java-badge]: https://github.com/benchmark-action/github-action-benchmark/workflows/JMH%20Example/badge.svg
[github-action]: https://github.com/features/actions
[cargo-bench]: https://doc.rust-lang.org/cargo/commands/cargo-bench.html
[benchmarkjs]: https://benchmarkjs.com/
Expand All @@ -615,12 +618,14 @@ Every release will appear on your GitHub notifications page.
[cpp-workflow-example]: https://github.com/benchmark-action/github-action-benchmark/actions?query=workflow%3A%22C%2B%2B+Example%22
[catch2-workflow-example]: https://github.com/benchmark-action/github-action-benchmark/actions?query=workflow%3A%22Catch2+C%2B%2B+Example%22
[julia-workflow-example]: https://github.com/benchmark-action/github-action-benchmark/actions?query=workflow%3A%22Julia+Example+with+BenchmarkTools.jl%22
[java-workflow-example]: https://github.com/benchmark-action/github-action-benchmark/actions?query=workflow%3A%22JMH+Example%22
[help-watch-release]: https://docs.github.com/en/github/receiving-notifications-about-activity-on-github/watching-and-unwatching-releases-for-a-repository
[help-github-token]: https://docs.github.com/en/actions/security-guides/automatic-token-authentication
[minimal-workflow-example]: https://github.com/benchmark-action/github-action-benchmark/actions?query=workflow%3A%22Example+for+minimal+setup
[commit-comment-workflow-example]: https://github.com/benchmark-action/github-action-benchmark/actions?query=workflow%3A%22Example+for+alert+with+commit+comment
[google-benchmark]: https://github.com/google/benchmark
[catch2]: https://github.com/catchorg/Catch2
[jmh]: https://openjdk.java.net/projects/code-tools/jmh/
[lighthouse-ci-action]: https://github.com/treosh/lighthouse-ci-action
[lighthouse-ci]: https://github.com/GoogleChrome/lighthouse-ci
[BenchmarkTools.jl]: https://github.com/JuliaCI/BaseBenchmarks.jl
Expand Down
172 changes: 172 additions & 0 deletions examples/java/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,172 @@
<!--
Copyright (c) 2014, Oracle America, Inc.
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
* Neither the name of Oracle nor the names of its contributors may be used
to endorse or promote products derived from this software without
specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
THE POSSIBILITY OF SUCH DAMAGE.
-->

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>org.openjdk.jmh.samples</groupId>
<artifactId>jmh-sample</artifactId>
<version>1.0</version>
<packaging>jar</packaging>

<name>JMH benchmark sample: Java</name>

<!--
This is the demo/sample template build script for building Java benchmarks with JMH.
Edit as needed.
-->

<dependencies>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-core</artifactId>
<version>${jmh.version}</version>
</dependency>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-generator-annprocess</artifactId>
<version>${jmh.version}</version>
<scope>provided</scope>
</dependency>
</dependencies>

<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

<!--
JMH version to use with this project.
-->
<jmh.version>1.29</jmh.version>

<!--
Java source/target to use for compilation.
-->
<javac.target>1.8</javac.target>

<!--
Name of the benchmark Uber-JAR to generate.
-->
<uberjar.name>benchmarks</uberjar.name>
</properties>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.0</version>
<configuration>
<compilerVersion>${javac.target}</compilerVersion>
<source>${javac.target}</source>
<target>${javac.target}</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<finalName>${uberjar.name}</finalName>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>org.openjdk.jmh.Main</mainClass>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
</transformers>
<filters>
<filter>
<!--
Shading signed JARs will fail without this.
http://stackoverflow.com/questions/999489/invalid-signature-file-when-attempting-to-run-a-jar
-->
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
<pluginManagement>
<plugins>
<plugin>
<artifactId>maven-clean-plugin</artifactId>
<version>2.5</version>
</plugin>
<plugin>
<artifactId>maven-deploy-plugin</artifactId>
<version>2.8.1</version>
</plugin>
<plugin>
<artifactId>maven-install-plugin</artifactId>
<version>2.5.1</version>
</plugin>
<plugin>
<artifactId>maven-jar-plugin</artifactId>
<version>2.4</version>
</plugin>
<plugin>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.9.1</version>
</plugin>
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>2.6</version>
</plugin>
<plugin>
<artifactId>maven-site-plugin</artifactId>
<version>3.3</version>
</plugin>
<plugin>
<artifactId>maven-source-plugin</artifactId>
<version>2.2.1</version>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.17</version>
</plugin>
</plugins>
</pluginManagement>
</build>

</project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
/*
* Copyright (c) 2005, 2013, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License version 2 only, as
* published by the Free Software Foundation. Oracle designates this
* particular file as subject to the "Classpath" exception as provided
* by Oracle in the LICENSE file that accompanied this code.
*
* This code is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
* version 2 for more details (a copy is included in the LICENSE file that
* accompanied this code).
*
* You should have received a copy of the GNU General Public License version
* 2 along with this work; if not, write to the Free Software Foundation,
* Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
*
* Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
* or visit www.oracle.com if you need additional information or have any
* questions.
*/
package org.openjdk.jmh.samples;

import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

public class JMHSample_01_HelloWorld {

/*
* This is our first benchmark method.
*
* The contract for the benchmark methods is very simple:
* annotate it with @Benchmark, and you are set to go.
* JMH will run the test by continuously calling this method, and measuring
* the performance metrics for its execution.
*
* The method names are non-essential, it matters they are marked with
* @Benchmark. You can have multiple benchmark methods
* within the same class.
*
* Note: if the benchmark method never finishes, then JMH run never
* finishes as well. If you throw the exception from the method body,
* the JMH run ends abruptly for this benchmark, and JMH will run
* the next benchmark down the list.
*
* Although this benchmark measures "nothing", it is the good showcase
* for the overheads the infrastructure bear on the code you measure
* in the method. There are no magical infrastructures which incur no
* overhead, and it's important to know what are the infra overheads
* you are dealing with. You might find this thought unfolded in future
* examples by having the "baseline" measurements to compare against.
*/

@Benchmark
public void wellHelloThere() {
// this method was intentionally left blank.
}

/*
* ============================== HOW TO RUN THIS TEST: ====================================
*
* You are expected to see the run with large number of iterations, and
* very large throughput numbers. You can see that as the estimate of the
* harness overheads per method call. In most of our measurements, it is
* down to several cycles per call.
*
* a) Via command-line:
* $ mvn clean install
* $ java -jar target/benchmarks.jar ".*JMHSample_01.*"
*
* JMH generates self-contained JARs, bundling JMH together with it.
* The runtime options for the JMH are available with "-h":
* $ java -jar target/benchmarks.jar -h
*
* b) Via Java API:
*/

public static void main(String[] args) throws RunnerException {
Options opt = new OptionsBuilder()
.include(".*" + JMHSample_01_HelloWorld.class.getSimpleName() + ".*")
.forks(1)
.build();

new Runner(opt).run();
}
}
Loading

0 comments on commit e5b1a19

Please sign in to comment.