soerenhenning / radargun

RadarGun is a Java framework for automated performance tests - just like jUnit for unit tests.

RadarGun - Performance Testing Framework

RadarGun is a Java framework for automated performance tests - just like jUnit for unit tests.

RadarGun performance tests consists of two parts:

1. The Benchmark Benchmarks have to be written with the Java Microbenchmarking Harness (JMH). They define the code segment (i.e., a sequence of instructions) that should be tested. Since conducting statically significant performance measurements in Java is not as easy as it seems, we recommend the paper DOs and DON’Ts of Conducting Performance Measurements in Java. Moreover, there exists several tutorials on JHM in the web.
2. The Assertions Assertions define an interval in which a test execution is seen as sucessful or not. Since the execution time highly dependends on the executing machine, assertions are defined individually per machine. RadarGun picks then the right assertion. See Declaration of Assertions for more information.

Usage

RadarGun can be executed in two different ways: First, via the command line and, second, using its Java API.

Execution via Command Line

The easiest way to execute RadarGun is via the command line: Just download RadarGun, make sure to have a recent Java installed, and type the following. Do not forget to replace <your_benchmarks.jar> by your benchmarks jar file.

On Unix-like operating systems (such as Linux and macOS):

java -cp "RadarGun.jar:<your_benchmarks.jar>" radargun.Main

On Windows (classpaths must be separeted by ;):

java -cp "RadarGun.jar;<your_benchmarks.jar>" radargun.Main

This will simply execute all JMH benchmarks. To make RadarGun comparing the benchmarks results with your predefined assertions, you have to tell RadarGun where they are.

From file system: If you have placed your assertions on the filesystem, append --assertions with a comma-separated list of files or directories to the command. A passed file has to be a YAML file containing assertions. When passing a directory, RadarGun considers all containing YAML files. Of course, you can mix both files and directories.

From classpaths: If your assertions are located in the classpath, for instance, directly in the benchmarks project, append --cp-assertions with a comma-separated list of files. Here, no directories are allowed for technical reasons.

RadarGun provides some more options, for example, to configure its output:

Usage: RadarGun [options]
  Options:
    --assertions
      Comma-sperated list of assertions in the filesystem (files or directories)
    --console-output
      Console output
      Default: true
    --cp-assertions
      Comma-sperated list of assertions in the classpath (must be files)
    --csv
      Enables CSV output to specified directory
    --exit-on-fail
      Enable exit on fail after all tests run
      Default: false
    --exit-on-fail-immediately
      Enable exit on fail of first test
      Default: false
    --help
      Print usage
    --jmh-output
      Enable JMH output
      Default: false
    --output-stream
      Output stream ('out' for System.out or 'err' for System.err)
      Default: System.out

Using the Java API

To use RadarGun within your Java (or other JVM-based language) project, you have to add it to your project's build path. You can either download the sources directly or use a build or dependency management tool such as Apache Maven.

RadarGun is currently only available as a so called Snapshot. However, this will change in a very near future. This version contains the latest changes and updates, but may be unstable. Therefore, it is not meant for productive usage. If you use Maven, please add the following depenency to the dependencies's section of your pox.xml:

<dependency>
  <groupId>de.soeren-henning</groupId>
  <artifactId>radargun</artifactId>
  <version>1.0.0-SNAPSHOT</version>
</dependency>

If you did not add the Sonatype snapshot repository yet to your pom.xml, add also following lines to it, as otherwise Maven will not be able to find the needed artifacts:

<repositories>
  <repository>
    <id>sonatype.oss.snapshots</id>
    <url>https://oss.sonatype.org/content/repositories/snapshots/</url>
  </repository>
</repositories>

For other build tools, adding the dependencies is similar.

RadarGun uses the Pipe-And-Filter framework TeeTime. Thus, a good starting point to work with it, is to use its radargun.Configuration.

Declaration of Assertions

Assertions are defined in YAML files in the following format:

---
identifier: MacAddressIdentifier
parameters: [01:23:45:67:89:AB]
tests:
  myproject.benchmark.MyFirstBenchmark.run: [70, 90]
  myproject.benchmark.MySecondBenchmark.run: [6.4, 6.7]
  myproject.benchmark.MyThirdBenchmark.run: [1300, 1400]

First, it defines the machine by an identifier class and the parameters by which it will be created (see Machine Identification). Afterwards, all assertions are declared by the fully qualified name of the benchmark and the lower and upper bound for the permitted benchmarks result.

A YAML file can contain multiple of such blocks (separated by a --- line) and, moreover, RadarGun can handle multiple of YAML files. RadarGun picks the right assertions.

Machine Identification

RadarGun provides the following machine identifers:

• NetworkAddressIdentifier Tests for a one or more network addresses, which can either be IP addresses or host names.
• MacAddressIdentifier Tests for one or more MAC addresses.
• WindowsComputernameIdentifier Tests for one or more computer names. Limited to Windows systems only.
• WildcardIdentifier Matches for all machines. It can be used when no machine distinction is necessary. (E.g., if there is only one machine.)
• Dismiss Identifier Matches for no machine. Normally, just used internally by RadarGun.

Of course, you can implement your own identifier. To do so, you just have to implement the MachineIdentifier interface. If your identifier is not located in the package radargun.comparsion.machine.identification, you have to define its fully qualified name in the assertions files.

Build RadarGun by Yourself

RadarGun uses Apache Maven as build tool. To build RadarGun an existing Maven and JDK installation is required. Simply execute:

mvn install

radargun's Issues

Add description to command line options

The Options class should provide adequate descriptions for each command line parameter and, moreover, should provide a short version for most parameters.

Handle NaNs in CSV export

For values that are not finite numbers, the table field just simply left empty

Getting exception when csv output directory does not exist

Instead of getting such an exception, the directory such be created instead.

Exception in thread "main" radargun.lib.teetime.framework.ExecutionException: 3003 - {Thread[Thread for BenchmarkRunnerStage-0,5,]=[java.lang.IllegalStateException: java.nio.file.NoSuchFileException: results\teetime.benchmark.Port2PortBenchmark.queue.csv]} error(s) occurred while execution. Check thrown exception(s).
        at radargun.lib.teetime.framework.Execution.waitForTermination(Execution.java:97)
        at radargun.lib.teetime.framework.Execution.executeBlocking(Execution.java:119)
        at radargun.Main.execute(Main.java:17)
        at radargun.Main.main(Main.java:22)

Add an MachineIdentifier for Host/Computername

Most OS have the concept of an computer name. This name should be used for this identifier.

Name clash

Hi,

I don't know if you've done any lookup on your project's name before starting that, but I'd like to let you know that there's already another project using the same name: https://github.com/radargun/radargun/ is a framework for benchmarking distributed systems, key-value DBs getting the most focus.

Since your project is newer (I can see commits starting in May 2017, while the above mentioned one conceived the name in 2010), it's probably your call to decide if keeping the name is worth possible confusion.

Cheers,

Radim