Configuration option allowing Hyperfoil to generate results regardless of session count audit. about hyperfoil

[rvansa]

AFAIK it should be generated (no option needed) and it's a bug if this does not happen. Do you have a reproducible case?

[rvansa]

FWIW all the failures should be written down in all.json and in stats/failures.csv.

[rvansa]

@whitingjr Can you still reproduce the issue?

[whitingjr]

Yes I can reproduce this issue. The code currently will poison any subsequent phase when this condition occurs.

[rvansa]

That code makes sure that subsequent phases are not started. It is not relevant to writing out results of phases that were already executed.

[whitingjr]

This issue has nothing to with already executed phases.
I am suggesting for a flag to disable poisoning.

[rvansa]

Okay, now I understand - the issue states 'the results file is not generated' so I thought you had further issues with that part.

So you want the SLAs to behave as soft limits; only marking them. WDYT about adding a parameter type: soft|hard so that you can define them more fine-grained, not just a global policy?

Also note that while the reason to fail a phase is not only SLA failure (setting that in https://github.com/Hyperfoil/Hyperfoil/blob/master/clustering/src/main/java/io/hyperfoil/clustering/ControllerVerticle.java#L307 ) - if you don't set SLAs explicitly the only limit is not blocking on connections. Is this what you're experiencing? Because from my experience the staircase-style benchmark hits the roof by running of out sessions as the latency * throughput exceeds maximum concurrency (number of sessions).
Solving that means setting a hard limit on concurrency (I've planned to do that as well), + not failing the phase. So what kind of termination criteria would you use in that case?

[whitingjr]

SLAs have not been set explicitly. So as you say it's blocking on connections.
I've thought about termination. Using the duration of the phase is most intuitive. Using that to cease the phase measurement requires a reconciliation of sessions. A message on unfulfilled responses. Then discarding those unfulfilled sessions to permit the next phase to start cleanly.

[rvansa]

If it's blocking on connections just define the SLA explicitly with blockedRatio: 1.0, like

- httpRequest:
    sla:
    - blockedRatio: 1.0

EDIT: actually blockedRatio = 1.0 is not absolutely unlimited (1.0 means that the clients can spend the same time waiting for connection as waiting for response), but you can use some high value to satisfy your needs.

[rvansa]

I am not concerned about phase termination, that's certainly up to duration or maxDuration. I am asking about stopping further phases - or do you want to run just the maxIterations phases?

[whitingjr]

Don't stop further phases. That's what I am asking for.
I don't have a maxIterations phase.

[rvansa]

OK, so you have just the ramp-up - steady-state - ramp-down style scenario, and you don't want the ramp-up to cancel steady-state.

My question was about the behaviour if you change the policy in scenarios when phases have multiple iterations, but it's quite straightforward in the end.

I am working on that right now.

[whitingjr]

Great. Thank you.

[rvansa]

maxSession limit overrun policy handled in 6d0d76c
SLA for blocking on connections can be already customized.