hyperfoil / hyperfoil_test

Ansible role for starting & monitoring Hyperfoil benchmark

Hyperfoil is microservice-oriented distributed benchmark framework that solves the coordinated-omission fallacy.

Project website: https://hyperfoil.io.

Prerequisites

• Java 17
• Apache Maven 3.8

Getting Started

mvn package

To run without test cases do

mvn -DskipTests=true package

Then the distribution is either in distribution/target/hyperfoil-<version>-SNAPSHOT.zip or in

cd distribution/target/distribution/

Image

We publish the image at quay.io/hyperfoil/hyperfoil.

Contributing

Contributions to Hyperfoil are managed on GitHub.com

• Ask a question
• Raise an issue
• Feature request
• Code submission

Contributions are most welcome !

You can reach the community on Zulip.

Please, consult our Code of Conduct policies for interacting in our community.

Consider giving the project a star on GitHub if you find it useful.

License

Apache-2.0 license

hyperfoil_test's Issues

Support for benchmark with multiple files in configuration.

Attempting to upload a benchmark configuration fails on Hyperfoil 0.8-SNAPSHOT. The benchmark has this configured

        - testSequence:
          - randomCsvRow:
             file: /tmp/3scale.csv
             skipComments: 'True'

which causes an error

TASK [hyperfoil.hyperfoil_test : Upload benchmark template] ******************************************************************
fatal: [server]: FAILED! => {"changed": false, "connection": "close", "content": "Cannot read benchmark: line 1, column 1: Expected DocumentStartEvent, got StreamEndEvent: <org.yaml.snakeyaml.events.StreamEndEvent()>", "content_length": "134", "msg": "Status code was 400 and not [204]: HTTP Error 400: Bad Request", "redirected": false, "status": 400, "url": "http://server:8090/benchmark"}
	to retry, use: --limit @/home/benchuser/jwhiting/3scale/perftest-toolkit-hf/deployment/run.retry

PLAY RECAP *******************************************************************************************************************

The conditions for uploading a benchmark have changed over time. A benchmark should be "symmetrical" and self contained. This condition was added to the CsvRowStep in this commit.

It means 2 files need uploading at the same time for the benchmark I have. Currently Ansible uri module does not support multi-part file uploads with defined field names. Which is a shame.

Report not loaded because it does not contain any data with schema http://hyperfoil.io/run-schema/v2.0

I am running into this issue when running tests:

Running hyperfoil_version: 0.15

Used task from https://github.com/3scale-labs/perftest-toolkit/blob/master/deployment/run.yml:

- hosts: hyperfoil_controller                
  roles:                                     
    - hyperfoil.hyperfoil_test               
  vars:                                      
    test_name: 3scale-benchmark              
    test_files:                              
      - "{{ csv_dest_file_path }}"           
    shared_connections: 10                   
    users_per_sec: 10                        
    duration_sec: 30                         

I see from Release notes that release 0.15 changed Report schema to v3.0

File generated by template module writes to agent machine.

Hi,
The hyperfoil_test playbook uses the template module to generate a filtered file from a template.
The output file is stored not on the controller but on the agent.

In the use case where the controller and agent are the same this is not an issue.

In a distributed use case the hyperfoil_test play fails. When attempting to Upload the benchmark template. The uri module is expecting the generated file to be present on the controller. I am seeing this error

TASK [hyperfoil.hyperfoil_test : Upload benchmark template] *****************************************************************
task path: /home/benchuser/.ansible/roles/hyperfoil.hyperfoil_test/tasks/main.yml:8
ESTABLISH SSH CONNECTION FOR USER: None
SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/benchuser/.ansible/cp/eedfe41682 acmeclient2 '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
(0, '/home/benchuser\n', '')
ESTABLISH SSH CONNECTION FOR USER: None
SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/benchuser/.ansible/cp/eedfe41682 acmeclient2 '/bin/sh -c '"'"'( umask 77 && mkdir -p "echo /home/benchuser/.ansible/tmp/ansible-tmp-1566566172.1-112492023418465" && echo ansible-tmp-1566566172.1-112492023418465="echo /home/benchuser/.ansible/tmp/ansible-tmp-1566566172.1-112492023418465" ) && sleep 0'"'"''
(0, 'ansible-tmp-1566566172.1-112492023418465=/home/benchuser/.ansible/tmp/ansible-tmp-1566566172.1-112492023418465\n', '')
ESTABLISH SSH CONNECTION FOR USER: None
SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/benchuser/.ansible/cp/eedfe41682 acmeclient2 '/bin/sh -c '"'"'rm -f -r /home/benchuser/.ansible/tmp/ansible-tmp-1566566172.1-112492023418465/ > /dev/null 2>&1 && sleep 0'"'"''
(0, '', '')
fatal: [acmeclient2]: FAILED! => {
"changed": false,
"msg": "Could not find or access '/tmp/3scale-benchmark.yaml' on the Ansible Controller.\nIf you are using a module and expect the file to exist on the remote, see the remote_src option"
}
to retry, use: --limit @/home/benchuser/code/jboss_performance/Hyperfoil-test-runner-for-3Scale/ansible-hyperfoil-3scale-run.retry

PLAY RECAP ******************************************************************************************************************
acmeclient2 : ok=3 changed=0 unreachable=0 failed=1

Ansible issue with conflicting action statements.

Over on the perftest-toolkit project the adoption of Hyperfoil is seeing an issue. When running on an Ubuntu platform.
A user is seeing an issue when the Ansible hyperfoil.hyperfoil_test role is executed. Mentioned in this comment.
The Ansible error is copied here:

TASK [include_role : hyperfoil.hyperfoil_test] ***********************************************************************************************************************************************************************************************************************************
task path: /home/eguzki/git/perftest-toolkit2/deployment/run.yml:4
ERROR! conflicting action statements: shell, warn

The error appears to be in '/home/eguzki/.ansible/roles/hyperfoil.hyperfoil_test/tasks/main.yml': line 34, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

    test_files: []
- name: Upload benchmark template
  ^ here

undefined variable issue when running tests

I am running into this issue when running tests:

fatal: [injector-benchmarks.dev.3sca.net]: FAILED! => {
    "msg": "The task includes an option with an undefined variable. The error was: the inline if-expression on line 1 evaluated to false and no else section was defined.\n\nThe error appears to be in '/home/eguzki/.ansible/roles/hyperfoil.hyperfoil_test/tasks/main.yml': line 43, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n  failed_when: curl_cmd.rc > 1\n- name: Start benchmark\n  ^ here\n"
}

It might be (not sure of course as I am not familiar with the codebase) related to this commit from @rvansa . If I remove the condition added in the commit, the tests run successfully.

Used task from https://github.com/3scale-labs/perftest-toolkit/blob/master/deployment/run.yml:

- hosts: hyperfoil_controller                
  roles:                                     
    - hyperfoil.hyperfoil_test               
  vars:                                      
    test_name: 3scale-benchmark              
    test_files:                              
      - "{{ csv_dest_file_path }}"           
    shared_connections: 10                   
    users_per_sec: 10                        
    duration_sec: 30                         

Ansible version

$ ansible-playbook --version
ansible-playbook 2.9.6