Rework perf tests

build.sbt

@@ -507,13 +507,11 @@ lazy val perfTests: ProjectMatrix = (projectMatrix in file("perf-tests"))
       "io.gatling.highcharts" % "gatling-charts-highcharts" % "3.11.12" % "test" exclude (
         "com.fasterxml.jackson.core", "jackson-databind"
       ),
-      "io.gatling" % "gatling-test-framework" % "3.11.2" % "test" exclude ("com.fasterxml.jackson.core", "jackson-databind"),
+      "io.gatling" % "gatling-test-framework" % "3.11.12" % "test" exclude ("com.fasterxml.jackson.core", "jackson-databind"),
       "com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.17.1",
       "nl.grons" %% "metrics4-scala" % Versions.metrics4Scala % Test,
       "com.lihaoyi" %% "scalatags" % Versions.scalaTags % Test,
-      // Needs to match version used by Gatling
-      "com.github.scopt" %% "scopt" % "3.7.1",
-      "io.github.classgraph" % "classgraph" % "4.8.172" % Test,
+      "io.github.classgraph" % "classgraph" % "4.8.172",
       "org.http4s" %% "http4s-core" % Versions.http4s,
       "org.http4s" %% "http4s-dsl" % Versions.http4s,
       "org.http4s" %% "http4s-blaze-server" % Versions.http4sBlazeServer,

perf-tests/README.md

@@ -2,41 +2,38 @@
 
 To work with performance tests, make sure you are running JDK 21+, and that the `ALSO_LOOM` environment variable is set, because the `perf-tests` project includes `tapir-nima`, which require Loom JDK feature to be available.
 
-Performance tests are executed by running `PerfTestSuiteRunner`, which is a standard "Main" Scala application, configured by command line parameters. It executes a sequence of tests, where
-each test consist of:
-
-1. Starting a HTTP server if specified (Like Tapir-based Pekko, Vartx, http4s, or a "vanilla", tapirless one)
-2. Running a simulation in warm-up mode (5 seconds, 3 concurrent users)
-3. Running a simulation with user-defined duration and concurrent user count
-4. Closing the server
-5. Reading Gatling's simulation.log and building simulation results
-
-The sequence is repeated for a set of servers multiplied by simulations. Afterwards, all individual simulation results will be aggregated into a single report. 
-If no test servers are specified, only simulations are run, assuming a server started externally.
-Command parameters can be viewed by running:
+Performance tests are executed by running `perfTests/Gatling/testOnly sttp.tapir.perf.SimulationClassName`, assuming that a server under tests is available on `localhost:8080`.
 
+## Starting the server
+To run a test server, use a separate sbt session and start it using `ServerRunner`:
+```
+perfTests/runMain sttp.tapir.perf.apis.ServerRunner http4s.TapirMulti
+```
+Run it without server name to see a list of all available names. 
+Exception: If you're testing `NettySyncServer` (tapir-server-netty-sync), its server runner is located elsewhere:
 ```
-perfTests/Test/runMain sttp.tapir.perf.PerfTestSuiteRunner
+nettyServerSync3/Test/runMain sttp.tapir.netty.sync.perf.NettySyncServerRunner
 ```
+This is caused by `perf-tests` using Scala 2.13 forced by Gatling, while `NettySyncServer` is written excluisively for Scala 3.
+
+## Configuring and running simulations
 
-which displays help similar to:
+Simulations can be found in `sttp.tapir.perf.Simulations.scala`. To run one, use Gatling/testOnly:
 
 ```
-[error] Usage: perf [options]
-[error]   -s, --server <value>    Comma-separated list of short server names, or groups like 'netty.*', 'pekko.*', etc. Available servers: http4s.TapirInterceptorMulti, http4s.TapirMulti, http4s.Tapir, http4s.VanillaMulti, http4s.Vanilla, netty.cats.TapirInterceptorMulti, netty.cats.TapirMulti, netty.cats.Tapir, netty.future.TapirInterceptorMulti, netty.future.TapirMulti, netty.future.Tapir, pekko.TapirInterceptorMulti, pekko.TapirMulti, pekko.Tapir, pekko.VanillaMulti, pekko.Vanilla, play.TapirInterceptorMulti, play.TapirMulti, play.Tapir, play.VanillaMulti, play.Vanilla, vertx.TapirInterceptorMulti, vertx.TapirMulti, vertx.Tapir, vertx.VanillaMulti, vertx.Vanilla, vertx.cats.TapirInterceptorMulti, vertx.cats.TapirMulti, vertx.cats.Tapir
-[error]   -m, --sim <value>       Comma-separated list of short simulation names, or '*' for all. Available simulations: PostBytes, PostFile, PostLongBytes, PostLongString, PostString, SimpleGetMultiRoute, SimpleGet
-[error]   -u, --users <value>     Number of concurrent users, default is 1
-[error]   -d, --duration <value>  Single simulation duration in seconds, default is 10
-[error]   -g, --gatling-reports   Generate Gatling reports for individuals sims, may significantly affect total time (disabled by default)
+perfTests/Gatling/testOnly sttp.tapir.perf.SimpleGetSimulation
 ```
-Generating Gatling reports is useful if you want to verify additional data like latency or throughput distribution over time.
-If you want to run a test server separately from simulations, use a separate sbt session and start it using `ServerRunner`:
 
+The simulation will first run in warmup mode, then it will run with specified user count and duration. To set these values, use system properties:
+`tapir.perf.user-count` and `tapir.perf.duration-seconds`. These values can be passed to sbt console on startup:
 ```
-perfTests/runMain sttp.tapir.perf.apis.ServerRunner http4s.TapirMulti
+sbt -Dtapir.perf.user-count=100 -Dtapir.perf.duration-seconds=60
 ```
-
-This is useful when profiling, as `perfTests/runMain` will be a forked JVM isolated from the JVM that runs Gatling.
+or within an already running interactive sbt session:
+```
+set ThisBuild/javaOptions += ""-Dtapir.perf.user-count=100"
+```
+If not set, default values will be used (see `sttp.tapir.perf.CommonSimulations`).
 
 ## Profiling 
 
@@ -56,82 +53,21 @@ After opening the flamegraph in your browser, use the spyglass icon to search fo
 
 Note that profiling noticeably affects performance, so it's recommended to measure throughput/latency without the profiler attached.
 
-## Examples
-
-1. Run all sims on all pekko-http servers with other options set to default:
-```
-perfTests/Test/runMain sttp.tapir.perf.PerfTestSuiteRunner -s pekko.* -m *
-```
-
-2. Run all sims on http4s servers, with each simulation running for 5 seconds:
-```
-perfTests/Test/runMain sttp.tapir.perf.PerfTestSuiteRunner -s http4s.Tapir,http4s.TapirMulti,http4s.Vanilla,http4s.VanillaMulti -m * -d 5
-```
+## Latency reports
 
-3. Run some simulations on some servers, with 3 concurrent users instead of default 1, each simulation running for 15 seconds, 
-and enabled Gatling report generation:
-```
-perfTests/Test/runMain sttp.tapir.perf.PerfTestSuiteRunner -s http4s.Tapir,netty.future.Tapir,play.Tapir -m PostLongBytes,PostFile -d 15 -u 3 -g
-```
-
-4. Run a netty-cats server with profiling, and then PostBytes and PostLongBytes simulation in a separate sbt session, for 25 seconds:
-```
-perfTests/runMain sttp.tapir.perf.apis.ServerRunner netty.cats.TapirMulti
-// in a separate sbt session:
-perfTest/Test/runMain sttp.tapir.perf.PerfTestSuiteRunner -m PostBytes,PostLongBytes -d 25
-```
-
-## Reports
-
-Each single simulation results in a latency HDR Histogram report printed to stdout as well as a file:
+Additionally to standard Gatling reports, each single simulation results in a latency HDR Histogram report printed to stdout as well as a file:
 
 ```
 [info] ******* Histogram saved to /home/kc/code/oss/tapir/.sbt/matrix/perfTests/SimpleGetSimulation-2024-02-26_10_30_22
 ```
 
 You can use [HDR Histogram Plotter](https://hdrhistogram.github.io/HdrHistogram/plotFiles.html) to plot a set of such files.
 
-The main report is generated after all tests, and contains results for standard Gatling latencies and mean throughput in a table combining
-all servers and tests. They will be printed to a HTML and a CSV file after the suite finishes:
-```
-[info] ******* Test Suite report saved to /home/alice/projects/tapir/.sbt/matrix/perfTests/tapir-perf-tests-2024-01-22_16_33_14.csv
-[info] ******* Test Suite report saved to /home/alice/projects/tapir/.sbt/matrix/perfTests/tapir-perf-tests-2024-01-22_16_33_14.html
-```
-
-These reports include information about throughput and latency of each server for each simulation.
-
-How the aggregation works: After each non-warmup test the results are read from `simulation.log` produced by Gatling and aggregated by `GatlingLogProcessor`. 
-The processor then uses 'com.codehale.metrics.Histogram' to calculate 
-p99, p95, p75, and p50 percentiles for latencies of all requests sent during the simulation.
-
 ## Adding new servers and simulations
 
 To add a new server, go to `src/main/scala` and put an object extending `sttp.tapir.perf.apis.ServerRunner` in a subpackage of `sttp.tapir.perf`. 
-It should be automatically resoled by the `TypeScanner` utility used by the `PerfTestSuiteRunner`.
+It should be automatically resolved by the `TypeScanner` utility used by the `ServerRunner`.
 
 Similarly with simulations. Go to `src/test/scala` and a class extending `sttp.tapir.perf.PerfTestSuiteRunnerSimulation` under `sttp.tapir.perf`. See the `Simulations.scala` 
 file for examples.
 
-## Testing WebSockets
-
-`WebSocketsSimulation` cannot be executed using `PerfTestSuiteRunner`, as it requires special warmup and injection setup, it also won't store gatling log in a format expected by our report builder.
-For WebSockets we want to measure latency distribution, not throughput, so use given instructions to run it and read the report:
-
-1. Adjust simulation parameters in the `sttp.tapir.perf.WebSocketsSimulation` class
-2. Start a server using `ServerRunner`, for example:
-```
-perfTests/runMain sttp.tapir.perf.apis.ServerRunner http4s.Tapir
-```
-If you're testing `NettySyncServer` (tapir-server-netty-sync), its server runner is located elsewhere:
-```
-nettyServerSync3/Test/runMain sttp.tapir.netty.sync.perf.NettySyncServerRunner
-```
-This is caused by `perf-tests` using Scala 2.13 forced by Gatling, while `NettySyncServer` is written excluisively for Scala 3.
-
-3. Run the simulation using Gatling's task:
-```
-perfTests/Gatling/testOnly sttp.tapir.perf.WebSocketsSimulation 
-```
-4. A HdrHistogram report will be printed to stdout and to a file. Check output for the full path.
-5. Stop the server manually.
-6. Use [HDR Histogram Plotter](https://hdrhistogram.github.io/HdrHistogram/plotFiles.html) to plot histogram file(s)

perf-tests/src/main/scala/sttp/tapir/perf/apis/ServerRunner.scala

@@ -19,7 +19,9 @@ object ServerRunner extends IOApp {
   private val runtimeMirror = universe.runtimeMirror(getClass.getClassLoader)
 
   def run(args: List[String]): IO[ExitCode] = {
-    val shortServerName = args.head
+    val shortServerName = args.headOption.getOrElse {
+      throw new IllegalArgumentException(s"Unspecified server name. Use one of: ${TypeScanner.allServers}")
+    }
     for {
       killSwitch <- startServerByTypeName(ServerName.fromShort(shortServerName))
       _ <- IO.never.guarantee(killSwitch)

perf-tests/src/main/scala/sttp/tapir/perf/apis/TypeScanner.scala

@@ -0,0 +1,38 @@
+package sttp.tapir.perf.apis
+
+import io.github.classgraph.ClassGraph
+
+import scala.jdk.CollectionConverters._
+import scala.reflect.ClassTag
+import scala.util.{Failure, Success, Try}
+
+import sttp.tapir.perf.Common._
+
+/** Uses the classgraph library to quickly find all possible server runners (objects extending ServerRunner)
+  */
+object TypeScanner {
+  def findAllImplementations[T: ClassTag](rootPackage: String): List[Class[_]] = {
+    val superClass = scala.reflect.classTag[T].runtimeClass
+
+    val scanResult = new ClassGraph()
+      .enableClassInfo()
+      .acceptPackages(rootPackage)
+      .scan()
+
+    try {
+      val classes =
+        if (superClass.isInterface)
+          scanResult.getClassesImplementing(superClass.getName)
+        else
+          scanResult.getSubclasses(superClass.getName)
+      classes.loadClasses().asScala.toList
+    } finally {
+      scanResult.close()
+    }
+  }
+
+  lazy val allServers: List[String] =
+    findAllImplementations[ServerRunner](rootPackage)
+      .map(_.getName)
+      .map(c => c.stripPrefix(s"${rootPackage}.").stripSuffix("Server$"))
+}