blazemeter / taurus

Automation-friendly framework for Continuous Testing by

Taurus

Quick links: Taurus Documentation | Knowledge Base | Support Forum

Purpose

Hides the complexity of performance and functional tests with an automation-friendly convenience wrapper. Taurus relies on JMeter, Gatling, Locust.io, and Selenium WebDriver as its underlying tools. Free and open source under Apache 2.0 License.

Installation or Upgrade

Just install it using PyPi:

pip install bzt

More detailed instructions for Linux, Mac OS and Windows available here.

Getting Started

Create a file named test.yml with following contents:

---
execution:
- concurrency: 10
  ramp-up: 1m
  hold-for: 1m30s
  scenario: simple
  
scenarios:
  simple:
    think-time: 0.75
    requests:
    - http://blazedemo.com/
    - http://blazedemo.com/vacation.html

Then run bzt test.yml. After the tool finishes, observe resulting summary stats in console log (more reporting options here). All artifact files from the run will be placed in the directory mentioned in console log. Read more on command-line tool usage here.

taurus's Issues

Ability to run from within container?

I have looked for this answer a few places, and maybe I am just missing it, but I'd like to be able to run this in docker-compose and kubernetes.

The former would require a volume mount, and then I'm assuming it needs environment variables to point it to the directory with configs (or maybe the exact file(s) - I'm not sure).

The latter would be environment variables and probably just copying the files into the image in the Dockerfile.

Are there any examples or can someone please help me run both of these (or either)?

Cannot pass all the value got by an API into another

Hello,
We're using Taurus to make some Load Test with our API and a new changement in our API require some new tests.
Basically, we take values from an API call that returns us codes, then we make a PUT call with another endpoint.
It actually looks like this :

POST TO CREATE AND RETURNS ARRAY OF IDs

PUT EACH ID individually :

But in order to improve our process, we wants now to put directly the array of IDs returned by the POST endpoint call directly into the serial_code attribute of our body (So without a foreach). I tried directly this :

But this doesn't work..
Can you please give us a solution for this issue, if you know how we can do it ?

Thanks a lot !

Sincerely,

wrong extractor is appended to BoundaryExtractor

https://github.com/Blazemeter/taurus/blob/ced9f031b1b3dba9d1dc4195558eb1522ac3be38/bzt/jmx/base.py#L1005C10-L1005C10

def _get_boundary_extractor(varname, subject, left, right, match_no, defvalue='NOT_FOUND', scope='', from_var=''):
....
subject = subjects.get(subject)
element = etree.Element("BoundaryExtractor", guiclass="BoundaryExtractorGui",
testclass="BoundaryExtractor", testname="Get %s" % varname, enabled="true")
element.append(JMX._string_prop("BoundaryExtractor.useHeaders", subject))
element.append(JMX._string_prop("BoundaryExtractor.refname", varname))
element.append(JMX._string_prop("BoundaryExtractor.lboundary", left))
element.append(JMX._string_prop("BoundaryExtractor.rboundary", right))
element.append(JMX._string_prop("RegexExtractor.default", defvalue))
element.append(JMX._string_prop("RegexExtractor.match_number", match_no))
element.extend(JMX.get_scope_props(scope, from_var))

jmeter integration - elapsed field missing from class

Hi,
I am trying to run a jmeter test with jmeter 5.6.2 and taurus 1.16.30 (docker image), and I receive the following error:

---- Debugging information ----
message             : Field not found in class.
field               : org.apache.jmeter.samplers.SampleSaveConfiguration.elapsed
-------------------------------

 Detail:com.thoughtworks.xstream.converters.ConversionException: 
---- Debugging information ----
cause-exception     : com.thoughtworks.xstream.converters.ConversionException
cause-message       : 
first-jmeter-class  : org.apache.jmeter.save.converters.TestElementConverter.unmarshal(TestElementConverter.java:109)
class               : org.apache.jmeter.save.ScriptWrapper
required-type       : org.apache.jmeter.save.ScriptWrapper
converter-type      : org.apache.jmeter.save.ScriptWrapperConverter
path                : /jmeterTestPlan/hashTree/hashTree/ResultCollector[4]/objProp/value/elapsed
line number         : 724
version             : 5.6.2

Similar error if I try to open modified_JMETER_SCRIPT.jmx with Jmeter 5.6.2

Looking at the modified_JMETER_SCRIPT.jmx I can find indeed the xml element elapsed set to true. If I remove that element, the test can be loaded by JMeter.

Is there something I need to do/set in order to have this working?

Thank you,
John

Homepage SSL certificate issues

Hi! Homebrew maintainer here. While upgrading the bzt package for Homebrew in Homebrew/homebrew-core#125639, our Linux CI complained that its homepage was unreachable (error here). A local test on Linux with curl revealed the following error:

$ curl -v https://gettaurus.org/
*   Trying 34.67.81.175:443...
* Connected to gettaurus.org (34.67.81.175) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

It seems that this issue has existed for some time. The links below are some related discussions, the earliest among which was posted in May 2022:

• Homebrew/homebrew-core#111035
• https://groups.google.com/g/codename-taurus/c/FS_cC9yMoqw
• https://groups.google.com/g/codename-taurus/c/tbZ8YR_iV0s

In the second link, a user named Bart Quintens said:

It seems the certificate chain for gettaurus.org is not complete. Missing intermediate certificate (see https://www.ssllabs.com/ssltest/analyze.html?d=gettaurus.org)

Influxdb reporter is unavailable.

Hi!

The recently added InfluxDB Reporter is very attractive!

However, I got an error when I consulted the documentation.
https://gettaurus.org/docs/Reporting/#Influxdb-Reporter

Like this.

(...)
    modules:
      influxdb:
        class: influxdb_reporter.InfluxdbStatusReporter
        host: localhost # influxdb address (default: localhost)
        port: 8086 # port (default: 8086)
        database: jmeter # the datasource name
        measurement: taurus # the measurement name
        username: ... # username if influxdb authentication enabled
        password: ...  # password if influxdb authentication enabled
        summary: true  # overall samples count and percent of failures
        percentiles: true  # send average times and percentiles
        send-data: true # enable/disable the sending
        send-interval: 30s   # send data each n-th second

received error on log;

ERROR: Config Error: 'application' property must be set. You should fix your configuration.

With reference to the source, data was sent to Influxdb correctly when specified as follows.

modules:
  influxdb-reporter:
    host: localhost # influxdb address (default: localhost)
    port: 8086 # port (default: 8086)
    database: jmeter # the datasource name
    measurement: taurus # the measurement name
    username: taurus # username if influxdb authentication enabled
    password: password  # password if influxdb authentication enabled
    summary: true  # overall samples count and percent of failures
    percentiles: true  # send average times and percentiles
    send-data: true # enable/disable the sending
    send-interval: 5s   # send data each n-th second
    application: hogehoge

Gatling "Empty results" issue due for we set group for request

Hi,
We are try to use Taurus do gatling testing, and the yaml look like:

execution:
  - executor: gatling
    scenario: sample
scenarios:
  sample:
    script: build/libs/abc.jar
    simulation: test.TestSimulation

in TestSimilation we use 'io.gatling.javaapi.core.group', code look like:

scenario("ScenarioName")
                .exec(session -> session.setAll(Map.of("timeStamp", nlocalDateTime, "baseUrl", BASE_URL)))
                .group("abc")
                .on(exec(login.doLogin)
                        .exec(access.doAccess).pause(1000, 5000)
                );    

Then we found gatling works fine with result like:

================================================================================
2023-06-02 03:51:27                                           5s elapsed
---- Requests ------------------------------------------------------------------
> Global                                                   (OK=2      KO=0     )
> abc / Login... (OK=1      KO=0     )
> abc / Access... (OK=1      KO=0     )

---- Requisition Simulation ----------------------------------------------------
[--                                                                        ]  0%
          waiting: 60     / active: 1      / done: 0     
================================================================================

But Taurus get error:
ERROR: Child Process Error: Empty results, most likely sample (GatlingExecutor) failed. Actual reason for this can be found in logs under /tmp/artifacts
By looking the source code, we can see if we set ".group("abc")" in scenario, the 'req_hierarchy' in below will be always not none and '__parse_request' always return None, could we know if there any reason doesn't support 'req_hierarchy'?

      req_hierarchy = fields[0].split(',')[0]
       if req_hierarchy:
           if error:
               self._group_errors[req_hierarchy].add(error)
           return None

taurus/bzt/modules/gatling.py

Line 603 in 2ce9dbe

return None

whehter can support private repository link nexus) to download jmeter plugin

I have a jmeter plugin from our own nexus private repository, but I didn't find any configuration supporting private repository
so want to know whether there is any configuration that can support private repository link nexus) to download Jmeter plugin

K6 Executor: Scenario defined within K6 .js only executing 1 iteration

Hi,

When running a K6 test through Taurus and having the scenario defined within the K6 script itself instead of the Taurus yml file, only 1 iteration is completed before shutting down the test.

Example K6 Test.

import http from "k6/http";
import { check, sleep } from "k6";

export const options = {
  scenarios: {
    POC: {
      executor: 'constant-vus',
      vus: 20,
      duration: '3m',
    },
  },
  tags: {
    name: 'GET-Request',
  }
};

export default function () {
  let res = http.get('https://www.google.com');
  
  check(res, {
    "status was 200": (r) => r.status == 200
  })
  sleep(1);
}

Example Taurus Yml

execution:
- executor: k6
  scenario: POC

scenarios:
  POC:
   script: K6-Demo.js  # has to be a valid K6 script

settings:
  artifacts-dir: Artifacts/POC_LoadTest-K6/%Y-%m-%d_%H-%M-%S-%f/

modules:
  console:
    disable: true

On inspecting the bzt log I noticed the following '--iterations 1' is added to the cmdline output

[2023-11-27 11:58:44,143 DEBUG root] Executing shell: ['k6', 'run', '--out', 'csv=C:\\Source\\K6 Test\\K6-Defined-Scenario\\Artifacts\\POC_LoadTest-K6\\2023-11-27_11-58-43-074044\\kpi.csv', '--iterations', '1', 'C:\\Source\\K6 Test\\K6-Defined-Scenario\\K6-Demo.js'] at C:\Source\K6 Test\K6-Defined-Scenario

I would expect Taurus to just execute the K6 Run command with the specified file, which would then use the scenario from within the K6 test, rather than defaulting iterations to 1.

How to set custom jmeter plugins source

Hi.

How to set in yaml the URL (e.g. artifactory) where download plugins from?
Jmeter has this settings https://jmeter-plugins.org/wiki/PluginsManager/#Plugins-Sources-Configuration

Thanks.

Docker Image Fails Inspection

Hello Team,
We have noticed that version 1.6.26 contains several outdated libraries with critical or high vulnerabilities. Could you please let us know if there are any plans in place to resolve these issues? Many of these vulnerabilities have open CVEs, and they do not appear to be false positives.

Here are the links to some of these security vulnerabilities with Critical or High Severity:

https://ubuntu.com/security/CVE-2023-38432
https://nvd.nist.gov/vuln/detail/CVE-2021-37137
https://nvd.nist.gov/vuln/detail/CVE-2023-22796
https://nvd.nist.gov/vuln/detail/CVE-2023-22796
https://ubuntu.com/security/CVE-2023-2156
https://nvd.nist.gov/vuln/detail/CVE-2021-37136
https://nvd.nist.gov/vuln/detail/CVE-2022-34169
https://nvd.nist.gov/vuln/detail/CVE-2023-22796
https://nvd.nist.gov/vuln/detail/CVE-2019-10086
https://nvd.nist.gov/vuln/detail/CVE-2022-45688

[Urgent] Docker Image file - Trivy High Vulnerabilities

Hi Team,

This is on the same line with the existing github issue #1770
We have noticed that version 1.6.26 contains several outdated libraries with critical or high vulnerabilities. Could you please let us know if there are any plans in place to resolve these issues?
We are depending on these docker image for doing loadtesting.

Here are the list of security vulnerabilities with High Severity:

• curl - CVE-2023-38545
• firefox - CVE-2022-25235 ,CVE-2022-25236,CVE-2022-26485,CVE-2022-26486
• libc-bin - CVE-2023-4911
• libcurl3-gnutls - CVE-2023-38545
• linux-libc-dev - CVE-2023-20569
• locales - CVE-2023-4911
• commons-beanutils:commons-beanutils - CVE-2019-10086
• io.netty:netty-codec - CVE-2021-37136, CVE-2021-37137
• net.minidev:json-smart - CVE-2023-1370
• org.apache.xmlgraphics:batik-bridge - CVE-2022-44729
• org.json:json - CVE-2022-45688,CVE-2022-45689,CVE-2022-45690 ,CVE-2023-5072
• org.testng:testng - CVE-2022-4065

Need Help with Configuring Taurus-InfluxDB Integration

Hey everyone,

I'm in a bit of a pickle with setting up Taurus to work with my InfluxDB, and I could really use some guidance from the community.

So, here's the deal: both my InfluxDB instance and Taurus are running fine on their own. I've got a local InfluxDB up with everything set (organization, bucket, token, you name it), and it's working perfectly. The same goes for Taurus – I've been running some example and custom test scenarios without a hitch.

However, when it comes to getting Taurus to send its test results over to InfluxDB, I'm hitting a wall.
The documentation points me to specify a "database", "measurement", "username", and "password".
This is where I'm scratching my head.
Is the "database" supposed to be my bucket in InfluxDB? And for "measurement", can I just put in anything, or is there a specific format I need to follow?

Also, I'm wondering why I'm using a username and password for this. Isn't a token a safer bet? I tried going by the book with the username and password, but all I got was this error:

13:48:40 ERROR: InfluxDBClientError: 401: {"code":"unauthorized","message":"Unauthorized"}
[... error stack trace ...]

I've been digging for answers and trying different things (e.g. commenting out the username and password), but I'm still stuck.
Has anyone else faced something similar or knows what I might be doing wrong?

Any tips on getting this setup right would be a lifesaver!

Thanks a bunch in advance!

P.S.: I'm reporting this as an issue because I think the documentation may use some improvement. However, I'm happy to close the issue and move it to discussion, should that be the correct place for this kind of thing.

Selenium library is not downloaded before test execution due to "Failed to check for updates, server returned 5xx."

Selenium library is not downloaded before test execution:

08:42:20  08:42:20 WARNING: Selenium check failed.
08:42:20  08:42:20 INFO: Installing Selenium None...
08:42:23  08:42:23 WARNING: Failed to check for updates, server returned 5xx.

locustio-taurus-wrapper.py csv writer can't handle error messages which have comments in them

Hello,

I've discovered an issue with the locustio-taurus-wrapper.py in Taurus, when handling failed requests. It doesn't seem to account for error messages which might contain commas or serialized json data. Please see below is an extract from the locust.log for the error message I've seen:

[2023-05-03 16:40:17,446] COMP/ERROR/root: Uncaught exception in event handler: 
Traceback (most recent call last):
  File "C:\Program Files\Python311\Lib\site-packages\locust\event.py", line 40, in fire
    handler(**kwargs)
  File "C:\Program Files\Python311\Lib\site-packages\bzt\resources\locustio-taurus-wrapper.py", line 75, in __on_request_write
    self.writer.writerow(self.__getrec(request_type, name, response_time, response_length, exception))
  File "C:\Program Files\Python311\Lib\csv.py", line 154, in writerow
    return self.writer.writerow(self._dict_to_list(rowdict))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_csv.Error: need to escape, but no escapechar set

Here is the error message that is being returned from locust.io :

Error report
# occurrences Error
------------------|---------------------------------------------------------------------------------------------------------------------------------------------
117 POST LogoutApi: CatchResponseError('Failed to Logout, Status: 429, error: {"HttpStatus":429,"ErrorMessage":"Request throttled","ErrorCode":5002}')
------------------|---------------------------------------------------------------------------------------------------------------------------------------------

I pinpointed it is relating to line 106 where the dialog is being set. I fixed this locally by setting the dialog to csv.QUOTE_NONNUMERIC but this needs to be change for it to work on blazemeter too

Chromedriver Installation Path Discrepancy on Windows 11

Hello Team,

I'm experiencing an issue with Chromedriver installation during the bzt run.

After downloading a zip file, the system seems to be looking for it in the following path:

15:23:03 ERROR: FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\Users\opopov\.bzt\selenium-taurus\tools\drivers\chromedriver\117.0.5938.88\chromedriver.exe-win64'

However, the actual location of the file is here:

"C:\Users\opopov\.bzt\selenium-taurus\tools\drivers\chromedriver\117.0.5938.88\chromedriver-win64\chromedriver.exe"

It seems like there's a discrepancy between the expected and actual paths.

Enhancing WebDriverIO Integration: Access or use of `settings.env`

I am currently working on setting up WebDriverIO-based tests using Taurus. In the usual setup, the wdio.conf.js file contains the baseUrl property. However, I noticed that the examples provided in this repository do not include this property. I initially assumed that Taurus would inject this information (from a settings.env in the wdio.yml file), but upon further inspection, I found that no base URL is defined in the Taurus configuration files and that it is hard-coded in the test itself.

I referred to the documentation for guidance, but it does not seem to cover this use case. This has led me to question whether the integration with WebDriverIO was designed with extensibility in mind.

In my project, I rely on settings.env, which is locally defined in a separate settings.yml file and included in the main YAML file. In the CI environment, this file is dynamically generated. This settings.yml file serves as the single source of truth and enables me to run different sets of tests (e.g. Locust and K6) against the same site. My current task is to add WebDriverIO-based tests to the same project.

The current requirement to hard-code the base Url (or use other means, such as environment variables in the wide tests) adds complexity to my project. It necessitates a refactor to ensure that all project parts use the same data source (settings).

I propose that Taurus should support the use of ${SOME_ENV_VARIABLE} within the execution section of the Taurus configuration file. This would allow for the definition of properties like baseUrl directly in the Taurus configuration, enabling multiple WebDriverIO executors with different configurations, including different base URLs. Furthermore, Taurus would call wdio with the --baseUrl argument, eliminating the need for any injection in the wdio.conf.js file.

This enhancement would increase the flexibility and extensibility of Taurus. I look forward to your thoughts on this proposal.

NO_PROXY nonProxy settings not working

Use case
I want to run tests towards internal websites and external websites. External websites are only available through a proxy server. Configuring a proxy server is also needed for bzt to function correctly (download packages). Without it it fails on fresh machines complaining that it can't download the plugins etc. So I have to configure a proxy even if I only want to run tests on internal sites.

Problem
When I configure settings-proxy-address I can access and test external websites. But Internal websites cannot be tested because these requests end up at the proxy server and I get a "Gatewat Timeout" message.

I've tried nonProxy and NO_PROXY settings in the yml file, but it doesn't help. I found nonProxy in the code, but the documentation doesn't mention it as a setting.

Also setting $env:NO_PROXY previously to running this script doesn't work.

Also tried verbose logging, but the nonProxy code doesn't output extra logging.

Log output

PS C:\Users\test\Source\Repos\test-jmeter> bzt .\taurus-simple-proxy-example.yml
15:21:48 INFO: Taurus CLI Tool v1.15.3
15:21:49 INFO: Starting with configs: ['.\\taurus-simple-proxy-example.yml']
15:21:49 INFO: Configuring...
15:21:49 INFO: Using proxy 'http://proxy.internal.company.com:3128'
15:21:49 INFO: Artifacts dir: C:\Users\test\Source\Repos\test-jmeter\2023-05-11_15-21-49.123340
15:21:49 INFO: Preparing...
15:21:52 WARNING: Thread group detection: plugin for ConcurrentThreadGroup not found, regular ThreadGroup will be used
15:21:54 WARNING: Failed to check for updates
15:21:56 INFO: Starting...
15:21:56 INFO: Waiting for results...
15:21:57 INFO: Did not mute console logging
15:21:57 INFO: Waiting for finish...
15:23:18 WARNING: Please wait for graceful shutdown...
15:23:18 INFO: Shutting down...
15:23:18 INFO: Post-processing...
15:23:18 INFO: Test duration: 0:01:22
15:23:18 INFO: Samples count: 100, 100.00% failures
15:23:18 INFO: Average times: total 75.021, latency 75.021, connect 0.016
15:23:18 INFO: Percentiles:
+---------------+---------------+
| Percentile, % | Resp. Time, s |
+---------------+---------------+
|           0.0 |        74.944 |
|          50.0 |        75.008 |
|          90.0 |        75.008 |
|          95.0 |        75.072 |
|          99.0 |        75.072 |
|          99.9 |        75.072 |
|         100.0 |        75.072 |
+---------------+---------------+
15:23:18 INFO: Request label stats:
+---------------------------------------+--------+-------+--------+-----------------+
| label                                 | status |  succ | avg_rt | error           |
+---------------------------------------+--------+-------+--------+-----------------+
| https://website.internal.company.com |  FAIL  | 0.00% | 75.021 | Gateway Timeout |
+---------------------------------------+--------+-------+--------+-----------------+

taurus-simple-proxy-example.yml

settings:
  proxy:
    address: http://proxy.internal.company.com:3128
    nonProxy: website.internal.company.com

execution:
- concurrency: 100
  ramp-up: 1s
  hold-for: 1s
  scenario: quick-test

scenarios:
  quick-test:
    requests:
    - https://website.internal.company.com

reporting:
- module: junit-xml
  filename: taurus-simple-proxy-example-junit.xml
- module: passfail
  criterias:
  - failures>0%, stop as failed

BZT incorrectly reports error when running a gRPC based scenario with a K6Executor

I have a scenario that uses the K6 executor to apply a gRPC workload against a server. When I use bzt 1.16.26 to run the scenario, the test appears to execute successfully and I do see results in kpi.csv file. But during post processing, I see the following error in the console

...
22:31:17 ERROR: Child Process Error: Empty results, most likely context-create (K6Executor) failed. Actual reason for this can be found in logs under /Users/lambere/git/ftl/build/blazemeter/context-create/artifacts/2023-11-09_22-31-11.718091

...

snippet from bzt.log

[2023-11-09 22:31:17,629 INFO Engine] Post-processing...
[2023-11-09 22:31:17,629 DEBUG Engine.local] Post-process k6/context-create
[2023-11-09 22:31:17,629 DEBUG Engine] Add existing artifact (move=False): /Users/lambere/git/ftl/build/blazemeter/context-create/artifacts/2023-11-09_22-31-11.718091/kpi.csv
[2023-11-09 22:31:17,629 DEBUG Engine] No need to copy /Users/lambere/git/ftl/build/blazemeter/context-create/artifacts/2023-11-09_22-31-11.718091/kpi.csv
[2023-11-09 22:31:17,631 DEBUG Engine.local] Exception in post_process of K6Executor: Empty results, most likely context-create (K6Executor) failed. Actual reason for this can be found in logs under /Users/lambere/git/ftl/build/blazemeter/cont
ext-create/artifacts/2023-11-09_22-31-11.718091 Traceback (most recent call last):
  File "/opt/homebrew/Cellar/bzt/1.16.26/libexec/lib/python3.12/site-packages/bzt/modules/provisioning.py", line 165, in post_process
    raise ToolError(message, diagnostics)
bzt.ToolError: Empty results, most likely context-create (K6Executor) failed. Actual reason for this can be found in logs under /Users/lambere/git/ftl/build/blazemeter/context-create/artifacts/2023-11-09_22-31-11.718091

[2023-11-09 22:31:17,634 DEBUG Engine] post_process: Empty results, most likely context-create (K6Executor) failed. Actual reason for this can be found in logs under /Users/lambere/git/ftl/build/blazemeter/context-create/artifacts/2023-11-09_2
2-31-11.718091
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/bzt/1.16.26/libexec/lib/python3.12/site-packages/bzt/engine/engine.py", line 356, in post_process
    module.post_process()
  File "/opt/homebrew/Cellar/bzt/1.16.26/libexec/lib/python3.12/site-packages/bzt/modules/provisioning.py", line 174, in post_process
    reraise(exc_info, exc_value)

As stated above, there are results in kpi.csv file. The file contains values for the following metrics:

metric_name,timestamp,metric_value,check,error,error_code,expected_response,group,method,name,proto,scenario,service,status,subproto,tls_version,url,extra_tags,metadata
data_sent,1699597873,0.000000,,,,,::setup,,,,,,,,,,,
data_received,1699597873,0.000000,,,,,::setup,,,,,,,,,,,
iteration_duration,1699597873,0.041542,,,,,::setup,,,,,,,,,,,
grpc_req_duration,1699597873,122.119750,,,,,,Create,<REDACTED>,,default,<REDACTED>,0,,,<REDACTED>,,
checks,1699597873,1.000000,status is OK,,,,,,,,default,,,,,,,
...

Note, I've redacted some values from the contents of the file.

I DO NOT see this problem when I use the K6 Executor to run a "REST" based workload.

Environment Details

[2023-11-09 22:31:11,672 INFO root] Taurus CLI Tool v1.16.26
[2023-11-09 22:31:11,672 DEBUG root] Build: 14557 Fri Sep 22 06:30:08 UTC 2023
[2023-11-09 22:31:11,672 DEBUG root] Extended git info:  93b5d487 Changelog for 1.16.26
[2023-11-09 22:31:11,672 DEBUG root] Command-line options: {'log': '/var/folders/5g/vcbxp4bn64v09lrd7cnrfy8h0000gp/T/bzt_plyp7lh6.log', 'option': None, 'quiet': None, 'verbose': None, 'no_system_configs': None, 'aliases': []}
[2023-11-09 22:31:11,673 DEBUG root] Python: CPython 3.12.0
[2023-11-09 22:31:11,673 DEBUG root] OS: uname_result(system='Darwin', node='beezer', release='22.6.0', version='Darwin Kernel Version 22.6.0: Wed Jul  5 22:21:53 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T6020', machine='arm64')
[2023-11-09 22:31:11,673 DEBUG root] Path to interpreter: /opt/homebrew/Cellar/bzt/1.16.26/libexec/bin/python
[2023-11-09 22:31:11,673 DEBUG root] Path to packages: ['/opt/homebrew/Cellar/bzt/1.16.26/libexec/bin', '/opt/homebrew/Cellar/[email protected]/3.12.0/Frameworks/Python.framework/Versions/3.12/lib/python312.zip', '/opt/homebrew/Cellar/[email protected]/3.12.0/Frameworks/Python.framework/Versions/3.12/lib/python3.12', '/opt/homebrew/Cellar/[email protected]/3.12.0/Frameworks/Python.framework/Versions/3.12/lib/python3.12/lib-dynload', '/opt/homebrew/Cellar/bzt/1.16.26/libexec/lib/python3.12/site-packages', '/opt/homebrew/opt/libcython/libexec/lib/python3.12/site-packages', '/opt/homebrew/opt/six/lib/python3.12/site-packages', '/opt/homebrew/opt/numpy/lib/python3.12/site-packages', '/opt/homebrew/opt/python-certifi/lib/python3.12/site-packages', '/opt/homebrew/opt/python-lxml/lib/python3.12/site-packages', '/opt/homebrew/opt/python-pytz/lib/python3.12/site-packages', '/opt/homebrew/opt/pyyaml/lib/python3.12/site-packages', '/opt/homebrew/lib/python3.12/site-packages']
[2023-11-09 22:31:11,673 DEBUG root] Default python: None
[2023-11-09 22:31:11,673 DEBUG root] Default python3: /usr/bin/python3
[2023-11-09 22:31:11,673 DEBUG root] Using personal config: /Users/lambere/.bzt-rc
[2023-11-09 22:31:11,673 INFO root] Starting with configs: ['/Users/lambere/.bzt-rc', './taurus.yaml']

docker build is not successful.

docker build is not successful.

When I do a docker build using the dockerfile as of 2023.5.16, the following error occurs and I cannot build.

 => [17/24] RUN apt-get -y install --no-install-recommends gpg-agent   && gpg -k   && gpg --no-default-keyring -  11.6s
 => ERROR [18/24] RUN mkdir -p /etc/bzt.d   && echo '{"install-id": "Docker"}' > /etc/bzt.d/99-zinstallID.json     0.6s
------
 > [18/24] RUN mkdir -p /etc/bzt.d   && echo '{"install-id": "Docker"}' > /etc/bzt.d/99-zinstallID.json   && echo '{"settings": {"artifacts-dir": "/tmp/artifacts"}}' > /etc/bzt.d/90-artifacts-dir.json   && cp `python3 -c "import bzt; print('{}/resources/chrome_launcher.sh'.format(bzt.__path__[0]))"`     /opt/google/chrome/google-chrome   && bzt -install-tools -v   && google-chrome-stable --version && firefox --version && dotnet --version | head -1:
#0 0.538 Traceback (most recent call last):
#0 0.538   File "<string>", line 1, in <module>
#0 0.538 ModuleNotFoundError: No module named 'bzt'
#0 0.542 cp: missing destination file operand after '/opt/google/chrome/google-chrome'
#0 0.542 Try 'cp --help' for more information.
------
Dockerfile:59
--------------------
  58 |     # auto installable tools
  59 | >>> RUN mkdir -p /etc/bzt.d \
  60 | >>>   && echo '{"install-id": "Docker"}' > /etc/bzt.d/99-zinstallID.json \
  61 | >>>   && echo '{"settings": {"artifacts-dir": "/tmp/artifacts"}}' > /etc/bzt.d/90-artifacts-dir.json \
  62 | >>>   && cp `python3 -c "import bzt; print('{}/resources/chrome_launcher.sh'.format(bzt.__path__[0]))"` \
  63 | >>>     /opt/google/chrome/google-chrome \
  64 | >>>   && bzt -install-tools -v \
  65 | >>>   && google-chrome-stable --version && firefox --version && dotnet --version | head -1
  66 |
--------------------
ERROR: failed to solve: process "/bin/sh -c mkdir -p /etc/bzt.d   && echo '{\"install-id\": \"Docker\"}' > /etc/bzt.d/99-zinstallID.json   && echo '{\"settings\": {\"artifacts-dir\": \"/tmp/artifacts\"}}' > /etc/bzt.d/90-artifacts-dir.json   && cp `python3 -c \"import bzt; print('{}/resources/chrome_launcher.sh'.format(bzt.__path__[0]))\"`     /opt/google/chrome/google-chrome   && bzt -install-tools -v   && google-chrome-stable --version && firefox --version && dotnet --version | head -1" did not complete successfully: exit code: 1

environment

Windows10Pro + WSL2 + Ubuntu 20.04 LTS

WSL version: 1.2.5.0
Kernel version: 5.15.90.1
WSLg version: 1.0.51
MSRDC version: 1.2.3770
Direct3D version: 1.608.2-61064218
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.19045.2965

Commands executed

1. git clone https://github.com/Blazemeter/taurus.git
2. cd taurus
3. docker build -t mytaurus ./

Other information

Comment out "bzt -install-tools -v " to build the Docker image
Running "bzt -install-tools -v" manually in that image will produce the following error

ver. 1.16.19

[2023-05-22 07:10:15,517 DEBUG root] Exception: Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/bzt/cli.py", line 679, in main
    code = executor.perform(parsed_configs)
  File "/usr/local/lib/python3.9/dist-packages/bzt/cli.py", line 288, in perform
    for module_name in sys.modules:
RuntimeError: dictionary changed size during iteration

Cause

The problem is caused by overriding the module_name variable in a function that logs details of missing versions of imported modules in the following PR.

PR:write interpreter details into bzt.log (#1416)

To build Docker.

RUN $PIP_INSTALL chardet
RUN $PIP_INSTALL bzt==1.16.18

You need to install a version of bzt that does not contain the relevant PR.

Certificate expired for JMeter 5.5 (v1.16.27)

The output of curl https://packages.blazemeter.com/jmeter/apache-jmeter-5.5.zip -o jmeter.zip shows certificate expired:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Even with TAURUS_DISABLE_DOWNLOADS=1, taurus bzt (v1.16.27) exits with error:

...
[2024-02-09 18:41:22,592 INFO Engine.jmeter.JMeter] Will install JMeter into /home/dev/.bzt/jmeter-taurus/5.5
[2024-02-09 18:41:22,592 DEBUG Engine.jmeter.JMeter.JMeterMirrorsManager] Retrieving mirrors from page: https://jmeter.apache.org/download_jmeter.cgi
[2024-02-09 18:41:22,593 DEBUG Engine.jmeter.JMeter.JMeterMirrorsManager] Exception: Traceback (most recent call last):
  File "/home/dev/.pyenv/versions/3.10.12/lib/python3.10/site-packages/bzt/utils.py", line 1692, in mirrors
    tmp_file = downloader.get(self.base_link)[0]
  File "/home/dev/.pyenv/versions/3.10.12/lib/python3.10/site-packages/bzt/utils.py", line 1425, in get
    raise TaurusInternalException("Downloads are disabled by TAURUS_DISABLE_DOWNLOADS env var")
bzt.TaurusInternalException: Downloads are disabled by TAURUS_DISABLE_DOWNLOADS env var

[2024-02-09 18:41:22,593 ERROR Engine.jmeter.JMeter.JMeterMirrorsManager] Can't fetch https://jmeter.apache.org/download_jmeter.cgi
[2024-02-09 18:41:22,593 DEBUG Engine.jmeter.JMeter.JMeterMirrorsManager] Total mirrors: 2
[2024-02-09 18:41:22,593 INFO Engine.jmeter.JMeter] Downloading: https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-5.5.zip
[2024-02-09 18:41:22,593 DEBUG root] No progressbar for non-tty output: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
[2024-02-09 18:41:22,593 ERROR Engine.jmeter.JMeter] Error while downloading https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-5.5.zip: Downloads are disabled by TAURUS_DISABLE_DOWNLOADS env var
[2024-02-09 18:41:22,595 INFO Engine.jmeter.JMeter] Downloading: https://packages.blazemeter.com/jmeter/apache-jmeter-5.5.zip
[2024-02-09 18:41:22,595 DEBUG root] No progressbar for non-tty output: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
[2024-02-09 18:41:22,596 ERROR Engine.jmeter.JMeter] Error while downloading https://packages.blazemeter.com/jmeter/apache-jmeter-5.5.zip: Downloads are disabled by TAURUS_DISABLE_DOWNLOADS env var
[2024-02-09 18:41:22,596 ERROR root] Internal Error: JMeter download failed: No more links to try
[2024-02-09 18:41:22,597 DEBUG root] TaurusInternalException: JMeter download failed: No more links to try
  File "/home/dev/.pyenv/versions/3.10.12/lib/python3.10/site-packages/bzt/cli.py", line 275, in perform
    self.engine.prepare()
  File "/home/dev/.pyenv/versions/3.10.12/lib/python3.10/site-packages/bzt/engine/engine.py", line 220, in prepare
    self.__prepare_provisioning()
  File "/home/dev/.pyenv/versions/3.10.12/lib/python3.10/site-packages/bzt/engine/engine.py", line 635, in __prepare_provisioning
    self.provisioning.prepare()
  File "/home/dev/.pyenv/versions/3.10.12/lib/python3.10/site-packages/bzt/modules/provisioning.py", line 72, in prepare
    executor.prepare()
  File "/home/dev/.pyenv/versions/3.10.12/lib/python3.10/site-packages/bzt/modules/jmeter.py", line 257, in prepare
    self.install_required_tools()
  File "/home/dev/.pyenv/versions/3.10.12/lib/python3.10/site-packages/bzt/modules/jmeter.py", line 814, in install_required_tools
    tool.install()
  File "/home/dev/.pyenv/versions/3.10.12/lib/python3.10/site-packages/bzt/modules/jmeter.py", line 1641, in install
    self.__install_jmeter(jmeter_dir)
  File "/home/dev/.pyenv/versions/3.10.12/lib/python3.10/site-packages/bzt/modules/jmeter.py", line 1513, in __install_jmeter
    jmeter_dist = self._download()
  File "/home/dev/.pyenv/versions/3.10.12/lib/python3.10/site-packages/bzt/utils.py", line 1520, in _download
    raise TaurusInternalException(error_message)
[2024-02-09 18:41:22,597 INFO Engine] Post-processing...
...

Gatling enhancements to support include-scenario and extract-regexp for regex,json,css, xpath

Changes to incorporate include-scenario + response extractions are already implemented/tested with full pass of code coverage, need help to approve and merge

Robot Framework Concurrency Support

Hi.

Is Taurus able to run robot executor with concurrency method ?
Or is it only run robot test sequential but looping until XX iterations ?

Support for MS Test with Selenium C#

Hi I see there is support for Selenium C# with NUnit. But i was wondering if it supports MS Test as well ? Could you please let me know ?

chromedriver download URLs were changed starting from chromedriver 115. Need to update bzt accordingly.

As announced on https://chromedriver.chromium.org/ :
Starting with M115 the latest Chrome + ChromeDriver releases per release channel (Stable, Beta, Dev, Canary) are available at the Chrome for Testing availability dashboard. For automated version downloading one can use the convenient JSON endpoints.

So URL https://chromedriver.storage.googleapis.com/LATEST_RELEASE is not valid anymore and will always return 114 version, while in reality, the latest stable chromedriver version is already 116.

Here is a link to the new endpoints: https://github.com/GoogleChromeLabs/chrome-for-testing#json-api-endpoints

So bzt is not able to install chromedriver 115 or higher.
Need to adjust chromedriver download logic according to new changes.

Selenium Executor: Message: session not created: Chrome failed to start: exited normally.

Hi,
I pulled the image "blazemeter/taurus:latest" and tried running it. The result is an error: Message: session not created: Chrome failed to start: exited normally.

My command:
docker run -it --rm -v `pwd`/scenarios:/bzt-configs blazemeter/taurus tc.yml

My tc.yml (My tc.yml file is in the scenarios folder):

execution:
- executor: selenium
  runner: apiritif
  scenario: example

scenarios:
  example:
    browser: Chrome
    requests:
      - label: Login
        actions:
          - go('https://www.google.com/')

Logs:

14:13:28 INFO: Taurus CLI Tool v1.16.30
14:13:28 INFO: Starting with configs: ['/root/.bzt-rc', 'tc.yml']
14:13:28 INFO: Configuring...
14:13:28 INFO: Proxy settings not set
14:13:28 INFO: Artifacts dir: /tmp/artifacts
14:13:28 INFO: Preparing...
14:13:29 INFO: Using Cgroups1LocalMonitor for engine health monitoring
14:13:29 INFO: Latest stable version of chromedriver is 124.0.6367.78
14:13:29 INFO: Used version of chromedriver is 123.0.6312.122
14:13:29 INFO: Latest stable version of geckodriver is 0.34.0
14:13:29 INFO: Used version of geckodriver is 0.34.0
14:13:33 INFO: Starting...
14:13:33 INFO: Waiting for results...
14:13:33 INFO: Iterations set to 1
14:13:33 INFO: Waiting for finish...
14:13:34 WARNING: Failed to check for updates, server returned 5xx.
14:13:35 WARNING: Please wait for graceful shutdown...
14:13:35 INFO: Shutting down...
14:13:36 INFO: Post-processing...
14:13:36 INFO: Test duration: 0:00:03
14:13:36 INFO: Samples count: 1, 100.00% failures
14:13:36 INFO: Average times: total 0.594, latency 0.000, connect 0.000
14:13:36 INFO: Percentiles:
+---------------+---------------+
| Percentile, % | Resp. Time, s |
+---------------+---------------+
|           0.0 |         0.594 |
|          50.0 |         0.594 |
|          90.0 |         0.594 |
|          95.0 |         0.594 |
|          99.0 |         0.594 |
|          99.9 |         0.594 |
|         100.0 |         0.594 |
+---------------+---------------+
14:13:36 INFO: Request label stats:
+--------------+--------+-------+--------+------------------------------------------------------------------------+
| label        | status |  succ | avg_rt | error                                                                  |
+--------------+--------+-------+--------+------------------------------------------------------------------------+
| test_example |  FAIL  | 0.00% |  0.594 | Message: session not created: Chrome failed to start: exited normally. |
+--------------+--------+-------+--------+------------------------------------------------------------------------+
14:13:36 INFO: Artifacts dir: /tmp/artifacts
14:13:36 INFO: Done performing with code: 0

Pls help me fix that. Thank you.