hyperfoil / hyperfoil Goto Github PK

View Code? Open in Web Editor NEW

98.0 98.0 34.0 5.29 MB

Microservice-oriented load driver solving the coordinated-omission fallacy

Home Page: https://hyperfoil.io

License: Apache License 2.0

Java 98.42% Shell 0.23% HTML 0.15% Dockerfile 0.03% JavaScript 0.99% CSS 0.18%

coordinated-omission java load-driver performance

hyperfoil's Introduction

Hyperfoil is microservice-oriented distributed benchmark framework that solves the coordinated-omission fallacy.

Project website: https://hyperfoil.io.

Prerequisites

Java 17
Apache Maven 3.8

Getting Started

mvn package

To run without test cases do

mvn -DskipTests=true package

Then the distribution is either in distribution/target/hyperfoil-<version>-SNAPSHOT.zip or in

cd distribution/target/distribution/

Image

We publish the image at quay.io/hyperfoil/hyperfoil.

Contributing

Contributions to Hyperfoil are managed on GitHub.com

Contributions are most welcome !

You can reach the community on Zulip.

Please, consult our Code of Conduct policies for interacting in our community.

Consider giving the project a star on GitHub if you find it useful.

License

Apache-2.0 license

hyperfoil's People

Contributors

Stargazers

Watchers

hyperfoil's Issues

Missing Maven version information for metainf-services.

Seeing this error when running the Maven build at the root project.

[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[ERROR] 'dependencies.dependency.version' for org.kohsuke.metainf-services:metainf-services:jar is missing. @ io.hyperfoil:hyperfoil-core:[unknown-version], /home/whitingjr/tmp/Hyperfoil/core/pom.xml, line 83, column 21
[ERROR] 'dependencies.dependency.version' for org.kohsuke.metainf-services:metainf-services:jar is missing. @ io.hyperfoil:hyperfoil-clustering:[unknown-version], /home/whitingjr/tmp/Hyperfoil/clustering/pom.xml, line 57, column 21
[ERROR] 'dependencies.dependency.version' for org.kohsuke.metainf-services:metainf-services:jar is missing. @ io.hyperfoil:hyperfoil-distribution:[unknown-version], /home/whitingjr/tmp/Hyperfoil/distribution/pom.xml, line 40, column 21
 @ 
[ERROR] The build could not read 3 projects -> [Help 1]
[ERROR]

Am using OpenJDK 11.0.3.

Does event bus guarantee ordering of events on event bus?

Do we need to attach a sequenceID to 1s histograms that are sent across event-bus from agents to controller, so that we aggregate histogram data in the correct order? Does the event bus guarantee order? if not, can we configure vert.x event bus to guarantee ordering?

Client certificates, verifying server certificates

All the SSL security stuff. Right now we're using InsecureTrustManagerFactory.INSTANCE.

Handle membership changes

When agents abruptly leave, controller does not register this. We should also detect when this happens during run and fail it.

Randomize ports in testsuite

Testsuite uses fixed ports, and when I have something else running on 8080/8088 it blows up. Make the testsuite more resilient to occupied ports.

Deploy artifacts to Sonatype

https://issues.sonatype.org/browse/OSSRH-46004

cli does not upload benchmark with ~ in path

Although autocomplete with a ~ in the file path works in the cli, the upload fails with (No such file or directory)

[hyperfoil@localhost]$ upload ~/projects/quarkus/rest-http-crud/hyperfoil-runner/src/test/resources/singleRequest.hf.yaml
ERROR: ~/projects/rest-http-crud/hyperfoil-runner/src/test/resources/singleRequest.hf.yaml (No such file or directory)
Failed to load the benchmark.

With an absoute file path, the upload works;

[hyperfoil@localhost]$ upload /home/johara/projects/rest-http-crud/hyperfoil-runner/src/test/resources/singleRequest.hf.yaml
Loaded benchmark single-request, uploading...
... done.

Add auto-generated help command to CLI

That should be a builtin Aesh feature which should be just enabled.

Linked resources downloader

HTML parser that automatically downloads all images, css files, html pages in frames...

Queues

In some scenarios having a fixed-size queues would be more comfortable than handling arrays and counters.

Promote Hyperfoil in Service Mesh Landscape

layer5io/layer5#78

A lot of "java.lang.IllegalStateException: Too long header line" during run

During a benchmark run, I am seeing a lot of the exception below.

The dataset grows to 299 KB during the run, I suspect that we are quickly running out of buffer space for large result sets

13:11:28,084 WARN  (nioEventLoopGroup-2-1) [i.h.c.c.n.Http1xConnection] Exception in Http1xConnection{/127.0.0.1:38204 -> localhost/127.0.0.1:8080, size=1} java.lang.IllegalStateException: Too long header line.
	at io.hyperfoil.core.client.netty.Http1xRawBytesHandler.copyLastLine(Http1xRawBytesHandler.java:219)
	at io.hyperfoil.core.client.netty.Http1xRawBytesHandler.readChunks(Http1xRawBytesHandler.java:210)
	at io.hyperfoil.core.client.netty.Http1xRawBytesHandler.channelRead(Http1xRawBytesHandler.java:75)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:844)

Parsing exception message is not returned to client through rest api

If there is a parsing exception for a benchmark uploaded via rest api, the exception message is not returned to the client.

Client side message

HTTP/1.1 500 Internal Server Error
content-length: 21

Internal Server Error

server side message

10:44:41,088 ERROR (vert.x-eventloop-thread-0) [i.v.e.w.i.RoutingContextImplBase] Unexpected exception in route java.lang.IllegalArgumentException: No initial sequences.
	at io.hyperfoil.api.config.ScenarioBuilder.build(ScenarioBuilder.java:100)
	at io.hyperfoil.api.config.PhaseBuilder$ConstantPerSec.buildPhase(PhaseBuilder.java:416)
	at io.hyperfoil.api.config.PhaseBuilder$ConstantPerSec.buildPhase(PhaseBuilder.java:392)
	at io.hyperfoil.api.config.PhaseBuilder.lambda$null$3(PhaseBuilder.java:157)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1492)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
...

The exception message should be returned to the client

Can not define randomItem: file: in initialSequence in an aliased scenario

If you define a randomItem step loading from a file in an intialSequence defined in an aliased scenario, when that alised scenario is referenced in another phase, the file is reloaded, but data already exists in String[] list.

the following exception is thrown:
BenchmarkDefinitionException("randomItem cannot combine list and file");

ClassNotFoundException: io.netty.util.concurrent.DefaultPromise$1

Occasionally, at the end of a run, the following exception is logged;

Exception in thread "nioEventLoopGroup-2-1" java.lang.NoClassDefFoundError: io/netty/util/concurrent/DefaultPromise$1
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:432)
	at io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:94)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:939)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: io.netty.util.concurrent.DefaultPromise$1
	at org.codehaus.plexus.classworlds.strategy.SelfFirstStrateg[

Add serverlog and agentlog options to CLI

Avoid the need to exit CLI to checkout the logs.

Configurable IP for HTTP targets

In situations where we're benchmarking DNS round robin from single agent we'd have to setup multiple targets (as IPs) explicitly but make sure that the Host header and SNI host are set up correctly.

Stats: responses > requests

Something fishy about $SUBJ... I have seen differences only in runs that did 100,000s of requests so far.

Benchmark completes with java.lang.IndexOutOfBoundsException

With the simple benchmark below;

name: single-request
http:
  baseUrl: http://localhost:8080
phases:
- example:
    constantPerSec: 
      usersPerSec: 10
      duration: 60s
      scenario:
      - test:
        - httpRequest:
            GET: /fruits
            sync: true

The benchmark completes, but outputs the following exceptions. Just before the first 1s stat interval;

11:47:39,749 ERROR (nioEventLoopGroup-9-1) [i.h.a.s.SequenceInstance] #6 test(0) failure invoking step io.hyperfoil.core.steps.HttpRequestStep@31f7f610 java.lang.IndexOutOfBoundsException: index 16
	at java.util.concurrent.atomic.AtomicReferenceArray.checkedByteOffset(AtomicReferenceArray.java:78)
	at java.util.concurrent.atomic.AtomicReferenceArray.get(AtomicReferenceArray.java:125)
	at io.hyperfoil.api.statistics.Statistics.active(Statistics.java:247)
	at io.hyperfoil.api.statistics.Statistics.incrementRequests(Statistics.java:74)
	at io.hyperfoil.core.steps.HttpRequestStep.invoke(HttpRequestStep.java:156)
	at io.hyperfoil.api.session.SequenceInstance.progress(SequenceInstance.java:27)
	at io.hyperfoil.core.session.SessionImpl.runSession(SessionImpl.java:244)
	at io.hyperfoil.core.session.SessionImpl.call(SessionImpl.java:195)
	at io.hyperfoil.core.session.SessionImpl.call(SessionImpl.java:26)
	at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:73)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

And at the end of the run;

11:47:39,821 WARN  (nioEventLoopGroup-9-1) [i.h.c.s.SessionImpl] #6 Session completed with requests in-flight!
java.lang.RuntimeException: java.lang.IndexOutOfBoundsException: index 16
11:47:39,823 ERROR (nioEventLoopGroup-9-1) [i.h.c.s.SessionImpl] #6 Uncaught error java.lang.IndexOutOfBoundsException: index 16
	at java.util.concurrent.atomic.AtomicReferenceArray.checkedByteOffset(AtomicReferenceArray.java:78)
	at java.util.concurrent.atomic.AtomicReferenceArray.get(AtomicReferenceArray.java:125)
	at io.hyperfoil.api.statistics.Statistics.active(Statistics.java:247)
	at io.hyperfoil.api.statistics.Statistics.incrementResets(Statistics.java:94)
	at io.hyperfoil.core.steps.HttpResponseHandlersImpl.handleThrowable(HttpResponseHandlersImpl.java:121)
	at io.hyperfoil.core.steps.HttpResponseHandlersImpl.handleThrowable(HttpResponseHandlersImpl.java:28)
	at io.hyperfoil.core.client.netty.Http1xConnection.cancelRequests(Http1xConnection.java:137)
	at io.hyperfoil.core.client.netty.Http1xConnection.close(Http1xConnection.java:209)
	at io.hyperfoil.core.session.SessionImpl.cancelRequests(SessionImpl.java:289)
	at io.hyperfoil.core.session.SessionImpl.runSession(SessionImpl.java:273)
	at io.hyperfoil.core.session.SessionImpl.call(SessionImpl.java:195)
	at io.hyperfoil.core.session.SessionImpl.call(SessionImpl.java:26)
	at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:73)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

Max unfinished sessions as SLAs

Currently we detect number of open sessions only when the phase finishes. It would be more fair to observe that through whole phase, possibly failing the phase prematurely.

In addition, right now crossing the threshold is observed as an error. We should do better since in staircase scenarios this is a perfectly valid termination criterion.

Cookie repeater defects

We don't handle "Secure" cookies correctly (those should be returned only on https connection).
Cookies shouldn't take port into account. https://stackoverflow.com/questions/1612177/are-http-cookies-port-specific

NPE on ill defined benchmark with iterations

Benchmark definition ;

name: simple benchmark
simulation:
  http:
    baseUrl: http://localhost:8080
  phases:
  - rampUp:
      rampPerSec:
        initialUsersPerSec: 1
        targetUsersPerSec: 10000
        duration: 30s
        scenario: &scenario
          initialSequences:
          - testSequence:
            - sla:
                meanResponseTime: 1s
            - httpRequest:
                GET: /fruits
                sync: true
                handler:
                  status:
                    range: 200
  - steadyState:
      constantPerSec:
        usersPerSec: 10000
        startAfter:
          phase: rampUp
        duration: 60s
        scenario: *scenario

Throws this exception;

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal io.hyperfoil:hyperfoil-maven-plugin:0.1-SNAPSHOT:run (default-cli) on project rest-http-crud-sailrocket-runner: Execution default-cli of goal io.hyperfoil:hyperfoil-maven-plugin:0.1-SNAPSHOT:run failed.
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution default-cli of goal io.hyperfoil:hyperfoil-maven-plugin:0.1-SNAPSHOT:run failed.
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:148)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
Caused by: java.lang.NullPointerException
    at io.hyperfoil.api.config.PhaseBuilder.iterationReferences (PhaseBuilder.java:235)
    at io.hyperfoil.api.config.PhaseBuilder$ConstantPerSec.buildPhase (PhaseBuilder.java:416)
    at io.hyperfoil.api.config.PhaseBuilder$ConstantPerSec.buildPhase (PhaseBuilder.java:391)
    at io.hyperfoil.api.config.PhaseBuilder.lambda$null$3 (PhaseBuilder.java:156)
    at java.util.stream.ReferencePipeline$3$1.accept (ReferencePipeline.java:193)
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining (ArrayList.java:1374)
    at java.util.stream.AbstractPipeline.copyInto (AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto (AbstractPipeline.java:471)
    at java.util.stream.ForEachOps$ForEachOp.evaluateSequential (ForEachOps.java:151)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential (ForEachOps.java:174)
    at java.util.stream.AbstractPipeline.evaluate (AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.forEach (ReferencePipeline.java:418)
    at java.util.stream.ReferencePipeline$7$1.accept (ReferencePipeline.java:270)
    at java.util.stream.IntPipeline$4$1.accept (IntPipeline.java:250)
    at java.util.stream.Streams$RangeIntSpliterator.forEachRemaining (Streams.java:110)
    at java.util.Spliterator$OfInt.forEachRemaining (Spliterator.java:693)
    at java.util.stream.AbstractPipeline.copyInto (AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto (AbstractPipeline.java:471)
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential (ReduceOps.java:708)
    at java.util.stream.AbstractPipeline.evaluate (AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.collect (ReferencePipeline.java:499)
    at io.hyperfoil.api.config.PhaseBuilder.build (PhaseBuilder.java:160)
    at io.hyperfoil.api.config.SimulationBuilder.lambda$build$3 (SimulationBuilder.java:108)
    at java.util.stream.ReferencePipeline$7$1.accept (ReferencePipeline.java:267)
    at java.util.HashMap$ValueSpliterator.forEachRemaining (HashMap.java:1620)
    at java.util.stream.AbstractPipeline.copyInto (AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto (AbstractPipeline.java:471)
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential (ReduceOps.java:708)
    at java.util.stream.AbstractPipeline.evaluate (AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.collect (ReferencePipeline.java:499)
    at io.hyperfoil.api.config.SimulationBuilder.build (SimulationBuilder.java:108)
    at io.hyperfoil.api.config.BenchmarkBuilder.build (BenchmarkBuilder.java:60)
    at io.hyperfoil.core.parser.BenchmarkParser.buildBenchmark (BenchmarkParser.java:85)
    at io.hyperfoil.maven.RunMojo.buildBenchmark (RunMojo.java:94)
    at io.hyperfoil.maven.RunMojo.execute (RunMojo.java:62)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)

Update to JDK LTS version 11 (and 9).

As of today we have JDK12 at GA state and JDK13 only round the corner.

This project needs to keep up with the frequent release cycles to remain fresh.

Log runs separately

It would be convenient if each run would be logged into separate file (maybe using RoutingAppender?)

ClassCastException: io.hyperfoil.core.session.SessionImpl cannot be cast to java.lang.Runnable

When testing a -scheduleDelay step, the following exception is thrown;

19:13:34,133 ERROR (nioEventLoopGroup-2-1) [i.h.a.s.SequenceInstance] #200 testSequence(0) failure invoking step io.hyperfoil.core.steps.ScheduleDelayStep@3845dbc6 java.lang.ClassCastException: io.hyperfoil.core.session.SessionImpl cannot be cast to java.lang.Runnable
	at io.hyperfoil.core.steps.ScheduleDelayStep.invoke(ScheduleDelayStep.java:67)
	at io.hyperfoil.api.session.SequenceInstance.progress(SequenceInstance.java:27)
	at io.hyperfoil.core.session.SessionImpl.runSession(SessionImpl.java:283)
	at io.hyperfoil.core.session.SessionImpl.call(SessionImpl.java:234)
	at io.hyperfoil.core.session.SessionImpl.call(SessionImpl.java:26)
	at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:73)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

HTTP chunk size calculation incorrect

The HTTP chunk size calculation in Http1xRawBytesHandler assumed a decimal number chunk size, whereas the chunk size is returned as a hexadecimal number

Enable a debug mode for standalone instance

allowing a debug mode from the shell script would let users call bin/standalone.sh --debug which would open a debug port for remote debugging

Merge ValidatorResults into StatisticsSnapshot

No keep-alive connections

While in practice we'd usually simulate keep-alive connections to prevent running out of ports, scenario where the connection lifespan would be limited to a session makes sense, too. With proper guidance (reducing TIMED_WAIT state duration) we should include this in the design.

Fail faster when connections cannot be estabilished

Currently benchmarks fail when connection pools cannot be pre-filled, but sometimes this process takes longer than agent initialization timeout. We should respond with failure earlier.

Document library dependencies

Hyperfoil may need some native dependencies, e.g. for ALPN support which requires OpenSSL (at least on older Java versions). Document that somewhere.

Normally tcnative-boringssl requires openssl and apr; on Fedora 30, though libxcrypt-compat is required as well.

Stats: responses < requests

Contrary to #7 I am seeing fewer responses than requests. I suspect a bug in accumulating stats on controller rather than really not receiving the responses/not recording the values.

IPv6 address parsing

In some places we do primitive address parsing, assuming that everything after a colon is port number. This does not work with IPv6 addresses.

Factory-less service loaders

Factories usually don't contain any business logic, it's just boilerplate. We could just use an annotation to Builder's construtor, stating name and acceptsParam.

NoClassDefFoundError: io/netty/util/concurrent/GlobalEventExecutor$2

Occasionally, benchmark completes are logs this incomplete exception stack trace;

[ERROR] Failed to submit a listener notification task. Event loop shut down?
java.lang.NoClassDefFoundError: io/netty/util/concurrent/GlobalEventExecutor$2
    at io.netty.util.concurrent.GlobalEventExecutor.startThread (GlobalEventExecutor.java:225)
    at io.netty.util.concurrent.GlobalEventExecutor.execute (GlobalEventExecutor.java:213)
    at io.netty.util.concurrent.DefaultPromise.safeExecute (DefaultPromise.java:768)
    at io.netty.util.concurrent.DefaultPromise.notifyListeners (DefaultPromise.java:432)
    at io.netty.util.concurrent.DefaultPromise.setSuccess (DefaultPromise.java:94)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run (SingleThreadEventExecutor.java:939)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run (FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run (Thread.java:748)
Caused by: java.lang.ClassNotFoundException: io.netty.util.concurrent.GlobalEventExecutor$2
    at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass (SelfFirstStrategy.java:50)
    at org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass (ClassRealm.java:271)
    at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass (ClassRealm.java:247)
    at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass (ClassRealm.java:239)
    at io.netty.util.concurrent.GlobalEventExecutor.startThread (GlobalEventExecutor.java:225)
    at io.netty.util.concurrent.GlobalEventExecutor.execute (GlobalEventExecutor.java:213)
    at io.netty.util.concurrent.DefaultPromise.safeExecute (DefaultPromise.java:768)
    at io.netty.util.concurrent.DefaultPromise.notifyListeners (DefaultPromise.java:432)
    at io.netty.util.concurrent.DefaultPromise.setSuccess (DefaultPromise.java:94)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run (SingleThreadEven

JSON stats persister errors

I've seen some errors from the JSON stats persister when the deployment fails:

07:46:08,818 ERROR (vert.x-worker-thread-9) [i.h.c.ControllerVerticle] Failed to persist statistics com.fasterxml.jackson.core.JsonGenerationException: Current context not Object but root
        at com.fasterxml.jackson.core.JsonGenerator._reportError(JsonGenerator.java:1961)
        at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeEndObject(UTF8JsonGenerator.java:351)
        at io.hyperfoil.core.impl.statistics.StatisticsStore.walkPhaseIterFork(StatisticsStore.java:272)
        at io.hyperfoil.core.impl.statistics.StatisticsStore.writeJson(StatisticsStore.java:307)
        at io.hyperfoil.core.impl.statistics.StatisticsStore.persist(StatisticsStore.java:480)
        at io.hyperfoil.clustering.ControllerVerticle.lambda$persistRun$31(ControllerVerticle.java:503)
        at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:272)
        at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)

Content-Type: application/json support

Current POST benchmark operation does not take Content-Type: application/json. Please add support to it as I am able to push & execute json instead of yaml.

Output session pool statistics at end of run to help size initial pool size

Session pool statistics would help sizing the maxSessionsEstimate for subsequent runs

Verify functionality of openshift routing

We need to verify that if we create new connections to a load balanced application in OpenShift that if we send multiple requests, and ramp up connections/requests, that the requests are routed through the router and not directly to the pod. We need to ensure that as the load increases, requests made over existing connections are sent to the router and not directly to the pods

Add example driver for 3scale testing.

A sample project to contribute will be a 3scale driver.

The characteristics of this driver are:

demonstrates driver initialization using programmatic API
other things ???

Statistics attribute everything to 0th iteration

Recent changes in statistics intended to make step id unique for each phase. I forgot that different iterations must share the scenario (and therefore the steps) - otherwise iterations couldn't share sessions.

Therefore we must adjust APIs again to send the phase as well...

can not upload a data file to use in benchmark

If you want to load data from a file, e.g.

randomItem: 
  -  file:

There is currently no way to upload and reference in a distributed manor. A full path name can be defined in the benchmark definition, but for a distributed benchmark, that file would need to be in the same location on all machines

Cacheing

Let's honor HTTP Cache-Control and Expires headers.

Configuration option allowing Hyperfoil to generate results regardless of session count audit.

At the moment the results file is not generated when the SLA check fails.
This makes the production of performance graphs that include the hockey stick curve not possible. At the moment only a straight line can be plotted. No further results at higher load are possible.
People are used to looking at a graph showing the degradation of results at higher loads.

An option called 'ignoreSLAaudit' for example can you used to configure Hyperfoil.

Support Audit post run audit

Method for aggregating data at end of run to test if benchmark passes/fails benchmark audits

Session statistics calculation amiss

Session statistics calculation is outputting some strange numbers;

Timestamp,Address,MinSessions,MaxSessions
1554923476852,a940dc91-4b62-4a1f-8875-b3e0fb16de24,36,40
1554923477850,a940dc91-4b62-4a1f-8875-b3e0fb16de24,37,40
1554923478844,a940dc91-4b62-4a1f-8875-b3e0fb16de24,37,40
1554923479843,a940dc91-4b62-4a1f-8875-b3e0fb16de24,37,40
1554923482942,a940dc91-4b62-4a1f-8875-b3e0fb16de24,39,40
1554923507842,a940dc91-4b62-4a1f-8875-b3e0fb16de24,-2,0
1554923508843,a940dc91-4b62-4a1f-8875-b3e0fb16de24,-37,0
1554923509842,a940dc91-4b62-4a1f-8875-b3e0fb16de24,-76,0
1554923510842,a940dc91-4b62-4a1f-8875-b3e0fb16de24,-106,0
1554923511842,a940dc91-4b62-4a1f-8875-b3e0fb16de24,-138,0

Consistent naming var/fromVar/toVar

Verify that toVar, fromVar etc are consistently defined in steps, so that users writing benchmark do not need to be aware of step context. Consistent naming helps users read the benchmark yaml

Propagate termination error back to cli

If an error occurs running the benchmark from the cli, propagate the exception back to the cli so users know the reason for benchmark failure. atm, users need to check process output to determine the cause of error

ByteBuf leak under load

Running with 1000 concurrent users locally, I occasionally see ByteBuf leaks;

[ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:331)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176)
	io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:137)
	io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:147)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:748)
[ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.buffer.SimpleLeakAwareByteBuf.unwrappedDerived(SimpleLeakAwareByteBuf.java:143)
	io.netty.buffer.SimpleLeakAwareByteBuf.readRetainedSlice(SimpleLeakAwareByteBuf.java:67)
	io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:305)
	io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
	io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	io.hyperfoil.core.client.netty.BaseRawBytesHandler.handleBuffer(BaseRawBytesHandler.java:32)
	io.hyperfoil.core.client.netty.Http1xRawBytesHandler.channelRead(Http1xRawBytesHandler.java:158)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:748)

Export statistics

I would be really nice to have a command to export statistics to a file. It would really help when running hyperfoil in automated scripts, so we can store the results without parsing console output or searching in /tmp/hyperfoil dir.