microsoft / azure-load-testing Goto Github PK

License: MIT License

azure-load-testing's Introduction

Azure Load Testing

Azure Load Testing is a fully managed load testing service that enables you to generate high-scale load. The service will simulate traffic for your applications, regardless of where they're hosted. Developers, testers, and quality assurance (QA) engineers can use it to optimize application performance, scalability, or capacity.

You can create a load test by using existing test scripts, based on Apache JMeter, a popular open-source load and performance tool. For Azure-based applications, detailed resource metrics help you identify performance bottlenecks. Continuous integration and continuous deployment (CI/CD) workflows allow you to automate regression testing.

Check out the documentation and create your first load test.

Frequently Asked Questions

Please visit the FAQs wiki page.

Known Issues for Public Preview

Please review the known issues wiki page before reporting new issues.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

azure-load-testing's People

Contributors

Stargazers

Watchers

azure-load-testing's Issues

Unable to upload Jmeter script

*Describe the bug
Unable to upload the Jmeter script(JMX script file), I noticed "Validation failed - Unable to validate JMX script" errors.
To Reproduce
Steps to reproduce the behavior:

Go to 'Create tests under the Azure Load Testing' In the Test plan section, I am trying to upload JMX file, but I am noticing the Validation failed errors.
Please fine the attached screenshot for more details

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

Chrome Browser

Additional context
Add any other context about the problem here.

AB#1460561

[Feature Request]-Server Side Metrics for non Azure Applicaions

Is your feature request related to a problem? Please describe.
Service Side Metrics are only available for Azure Application Components, what about application running on-premise or other clouds. To get the complete performance metrics, service side metrics is also one of the important measuring criteria e.g resource utilization, deadlocks etc.

Describe the solution you'd like
An agent installed on servers to collect metrics and sent it to Azure Load Testing Dashboard.

[BUG] Test 1

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

[BUG] test issue

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

[Feature Request] Information Request - Range of IP's used by Azure Load Testing

Please describe.
Currently looking at using Azure Load Testing with internal API's but would require a range of IP's to whitelist

Describe the solution you'd like
Would it be possible to get a range of IP's that Azure Load Testing uses?

AB#1457325

[Feature Request] Fragment files

When creating a new test, we need to provide a JMX file in Test plan tab.
However, it seems that it's missing a crucial feature of JMeter, which is Fragments files, which helps, for example, developers and QA managers to modularize huge test files.

For example:

I want to test a flow to obtain a user list in a authenticated dashboard by using "1-user-list.jmx" file.
I have a fragment file to make the authentication flow that is being used in multiple test files "0-login.jmx".

This is not possible since that the preview version doesn't allow the user to upload those fragment list, which could lead to difficulties on migrating to Azure Load Testing service.

Possible solution:

I don't know if allowing multiple files to be uploaded is a good idea, instead, I think having a different tab to upload fragment files can solve the problem.

AB#1451982

[BUG] Load Test: Downloaded results file too large to open in Excel (650+ MB for a one-minute test)

Describe the bug
Load Test: run test, download results file - file too large to open in Excel (600mb+)

To Reproduce

Create Load Test
Run Load Test
Select Download Results from the top menu.
Attempt to open file in Excel.

Expected behavior
Downloaded file should open in Excel.

Screenshots

Desktop (please complete the following information):

OS: Windows 10 (latest, Microsoft internal)
Browser: Edge
Version: Latest

Example: A sample downloaded results file for a simple App Service one-minute test was 682 megabytes.

AB#1467963

[Feature Request] CLI or ARM template provisioning

Is your feature request related to a problem? Please describe.
My customer wants to be able to automate the process of test creation and running so that a "standard set" of tests can be provisioned and run later.

Describe the solution you'd like
Test creation by an ARM template or through CLI calls.

Describe alternatives you've considered
I can see that there is a VS Code extension and also there are actions for pipelines. But the customer really wants to be able to deploy a library of tests and for others to later execute these. The execute element could be done in the portal, but I can't see how you can lodge a set of test scripts to be used later.

Additional context
This is in no sense a blocker for the customer, but it would be good to understand more on what automation is planned for load testing.

AB#1458968

[BUG] Azure Load Testing Portal displaying incorrect Response time for 90, 95 and 99 percentile.

Describe the bug
Azure Load Testing Portal displaying incorrect Response time for 90, 95 and 99 percentile as compare to JMeter dashboard.

To Reproduce

Expected behavior
Client side metric in Azure Load testing portal should display P99 for post call as 11 instead of 10.44
Client side metric in Azure Load testing portal should display P99 for get call as 10 instead of 12.48

Screenshots

AB#1465388

[BUG] Test issue

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

Test

AB#1452409

[Feature Request]

Is your feature request related to a problem? Please describe.
The customer would like to be able to run tests annually and so be able to access all of the results from previous load test runs as a basis of comparison. This needs to include the load test results and the metrics from the system under test. The concern is that the metrics from the system under test will "age out".

Describe the solution you'd like
Some more documentation on the retention period of test results and the logs from the system under test and/or how to manage these across several years, so the testers can do performance comparisons across several years.

Describe alternatives you've considered
You can export the JMeter results, which is good for looking at the test side, but it is not clear whether the system under test metrics can be exported before they age out.

Additional context

AB#1458950

[BUG]

Describe the bug
The Azure DevOps pipeline step that was installed from the marketplace fails to run

To Reproduce
Steps to reproduce the behavior:

Create a new YAML pipeline in Azure DevOps
Add the Azure Load Test activity
Attempt to run the pipeline

Expected behavior
I expect the the load test to run in some manner.

Screenshots

The following is the debug trace from the pipeline

##[debug]Evaluating condition for step: 'AzureLoadTest'
##[debug]Evaluating: SucceededNode()
##[debug]Evaluating SucceededNode:
##[debug]=> True
##[debug]Result: True
Starting: AzureLoadTest

Task : Azure Load Testing
Description : Automate performance regression testing with Azure Load Testing
Version : 1.1.87
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/load-testing/tutorial-cicd-azure-pipelines#azure-load-testing-task

##[debug]Using node path: C:\agents\2.195.2\externals\node10\bin\node.exe
##[debug]agent.TempDirectory=D:\a_temp
##[debug]loading inputs and endpoints
##[debug]loading ENDPOINT_AUTH_0db1e250-96d4-4cd1-9c2e-658befae7a8d
##[debug]loading ENDPOINT_AUTH_PARAMETER_0db1e250-96d4-4cd1-9c2e-658befae7a8d_AUTHENTICATIONTYPE
##[debug]loading ENDPOINT_AUTH_PARAMETER_0db1e250-96d4-4cd1-9c2e-658befae7a8d_SERVICEPRINCIPALID
##[debug]loading ENDPOINT_AUTH_PARAMETER_0db1e250-96d4-4cd1-9c2e-658befae7a8d_SERVICEPRINCIPALKEY
##[debug]loading ENDPOINT_AUTH_PARAMETER_0db1e250-96d4-4cd1-9c2e-658befae7a8d_TENANTID
##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
##[debug]loading ENDPOINT_AUTH_SCHEME_0db1e250-96d4-4cd1-9c2e-658befae7a8d
##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
##[debug]loading INPUT_CONNECTEDSERVICENAMEARM
##[debug]loading INPUT_LOADTESTCONFIGFILE
##[debug]loading INPUT_LOADTESTRESOURCE
##[debug]loading INPUT_RESOURCEGROUP
##[debug]loading SECRET_SYSTEM_ACCESSTOKEN
##[debug]loaded 14
##[debug]Agent.ProxyUrl=undefined
##[debug]Agent.CAInfo=undefined
##[debug]Agent.ClientCert=undefined
##[debug]Agent.SkipCertValidation=undefined
##[debug]connectedServiceNameARM=0db1e250-96d4-4cd1-9c2e-658befae7a8d
##[debug]0db1e250-96d4-4cd1-9c2e-658befae7a8d auth param serviceprincipalid = ***
##[debug]0db1e250-96d4-4cd1-9c2e-658befae7a8d auth param serviceprincipalkey = ***
##[debug]0db1e250-96d4-4cd1-9c2e-658befae7a8d data SubscriptionID = 2e500704-1ab3-49c6-bbbb-8b2bec98ffdc
##[debug]0db1e250-96d4-4cd1-9c2e-658befae7a8d auth param tenantid = ***
##[debug]loadTestConfigFile=D:\a\1\s\SampleApp.yaml
##[debug]secrets=undefined
##[debug]env=undefined
##[debug]resourceGroup=loadtest-rg
##[debug]loadTestResource=loadtestingpreview
##[debug]task result: Failed
##[debug]Processed: ##vso[task.complete result=Failed;]
Finishing: AzureLoadTest

Desktop (please complete the following information):

Windows 10
Edge
Azure DevOps

Smartphone (please complete the following information):

Additional context
Very little to go on, it seems the step or marketplace action just does not run. It's equivalent in GitHub actions works (if that's any help). Used debug settings, but little to go on in terms of being able to troubleshoot this.

AB#1458947

[BUG]

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

[BUG] Pipeline error - Cannot find module 'azure-pipelines-task-lib/task'

Describe the bug
Trying to call the AzureLoadTest task, it fails with the following information. Looks like the task was just updated to 1.1.90. I hadn't tried it before to know if it worked with 1.1.89.

==============================================================================
Task : Azure Load Testing
Description : Automate performance regression testing with Azure Load Testing
Version : 1.1.90
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/load-testing/tutorial-cicd-azure-pipelines#azure-load-testing-task

##[debug]Using node path: /home/vsts/agents/2.198.3/externals/node10/bin/node
internal/modules/cjs/loader.js:638
throw err;
^

Error: Cannot find module 'azure-pipelines-task-lib/task'
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
at Function.Module._load (internal/modules/cjs/loader.js:562:25)
at Module.require (internal/modules/cjs/loader.js:692:17)
at require (internal/modules/cjs/helpers.js:25:18)
at Object. (/home/vsts/work/_tasks/AzureLoadTest_b61b360c-31ac-4389-922b-303ea8befde7/1.1.90/index.js:32:12)
at Module._compile (internal/modules/cjs/loader.js:778:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
at Module.load (internal/modules/cjs/loader.js:653:32)
at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at Function.Module._load (internal/modules/cjs/loader.js:585:3)
##[error]Exit code 1 returned from process: file name '/home/vsts/agents/2.198.3/externals/node10/bin/node', arguments '"/home/vsts/work/_tasks/AzureLoadTest_b61b360c-31ac-4389-922b-303ea8befde7/1.1.90/index.js"'.
##[debug]Microsoft.VisualStudio.Services.Agent.Util.ProcessExitCodeException: Exit code 1 returned from process: file name '/home/vsts/agents/2.198.3/externals/node10/bin/node', arguments '"/home/vsts/work/_tasks/AzureLoadTest_b61b360c-31ac-4389-922b-303ea8befde7/1.1.90/index.js"'.
at Microsoft.VisualStudio.Services.Agent.Util.ProcessInvoker.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, InputQueue1 redirectStandardIn, Boolean inheritConsoleHandler, Boolean keepStandardInOpen, Boolean highPriorityProcess, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.Agent.ProcessInvokerWrapper.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, InputQueue1 redirectStandardIn, Boolean inheritConsoleHandler, Boolean keepStandardInOpen, Boolean highPriorityProcess, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.Agent.Worker.Handlers.DefaultStepHost.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, Boolean inheritConsoleHandler, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.Agent.Worker.Handlers.NodeHandler.RunAsync()
at Microsoft.VisualStudio.Services.Agent.Worker.TaskRunner.RunAsync()
at Microsoft.VisualStudio.Services.Agent.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)

To Reproduce
Steps to reproduce the behavior:
Added task

task: AzureLoadTest@1
inputs:
azureSubscription: 'Visual Studio'
loadTestConfigFile: 'load/weatherforecast.yaml'
resourceGroup: 'rg-loadtesting'
loadTestResource: 'lt-azug-demo'

Expected behavior
I would expect it to work or complain about my settings I'm passing in.

AB#1483955

[BUG] Incorrect Virtual Users Count displayed as part of Client metrics

Describe the bug
Client metrics should display max virtual user count while it seems to be average count.
Please check below snapshot.

Expected behavior
Max Virtual user count should be 30.

AB#1465667

[BUG] Test issue

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

AB#1463627

[BUG] Unable to create the Load Testing resource through the Portal.

Describe the bug
Cannot create the resource from the Portal. It seems to be a resource name validation issue.

To Reproduce
Steps to reproduce the behavior:

Go to https://portal.azure.com/#create/Microsoft.CloudNativeTesting
Fill in any resource name. I've tried all lower case, all upper, no numbers, only numbers etc.
See error:
Azure resource names cannot contain special characters \/""[]:|<>+=;,?*@&, whitespace, or begin with '_' or end with '.' or '-'.

Expected behavior
Be able to create the resource.

Screenshots

Desktop (please complete the following information):

OS: Windows
Browser Chrome
Version 94

[BUG] Load Test: Create, run twice, delete second run - entire test deleted (all test runs)

Describe the bug
Load Test: Create, run twice, delete second run - entire test deleted (all test runs)

To Reproduce

Create Load Test
Run Load Test
Run Load Test again
Open second run of the test. Delete test run.

Expected behavior
Test should not be deleted.

Screenshots

Desktop (please complete the following information):

OS: Windows 10 (latest, Microsoft internal)
Browser: Edge
Version: Latest

Submitted by Conrad to Nick Trough in December 2021 who sent to product team.

AB#1467962

JSR223 and custom jar support

Is there support for JSR223 scripts and custom jars? If not now, is there a target date for such support?

[BUG] test 2

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

[BUG] Real pricing vs in docs

I have created this resource for one day and ececuted one test (that would not last longer than few minutes. What I see in billing for it shows as:

While in pricing it shows $10 per month:
https://azure.microsoft.com/en-us/pricing/details/load-testing/

Is that bug or I have been billed upfront for whole month? (I got billed for this one day 7.68euro (I have deleted resource to avoid additional costs just in case)

[BUG] test devdiv

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

[Feature Request] - Load Test with CSV input

Is your feature request related to a problem? Please describe.
I need to be able to run a perf test with a CSV input. Although this is in the Test Configuration YAML, it's not clear how to do this. I receive an error 'File Path not allowed in Test Plan CSVDataSet Config'

Describe the solution you'd like
Could you provide an example with includes a CSV input to a test.

[Feature Request]-Ability to update existing jmeter script

Is your feature request related to a problem? Please describe.
It is frustrating that I have to create a new test every time I have an update to an existing jmeter script. I don't want to have to enter the pass/fail criteria and select the azure resource(s) be monitored over and over again.

Describe the solution you'd like
Ability to upload a replacement for the current jmeter script but retain all of the current options.

Describe alternatives you've considered
Ability to clone/copy an existing test and allow me to upload a new jmeter script but retain all other settings

Additional context
Add any other context or screenshots about the feature request here.

[BUG] Create and Run Load Test: Results Disappear

Describe the bug
Load Test: Create test, run, exit, return: test results Disappear

To Reproduce

Create a Load Test (JMeter has URL to Azure Function; added resources)
Run Test
Exit and return to Test Results page

Expected behavior
Test results should not disappear.

Screenshots

Desktop (please complete the following information):

OS: Windows 10 (latest, Microsoft internal)
Browser: Edge
Version: Latest

Submitted by Conrad to Nick Trough in December 2021 who sent to product team.

Error displayed when returning to test results (see attached screen shot):

{ "shellProps": { "sessionId": "98b854b976244f5ea45cdea6ca046693", "extName": "Microsoft_Azure_CloudNativeTesting", "contentName": "NewReport" }, "error": { "message": "t.throughput.toFixed is not a function", "error": { "message": "t.throughput.toFixed is not a function", "name": "TypeError", "stack": "TypeError: t.throughput.toFixed is not a function\n at t._initStatistics (https://afd.hosting-ms.portal.azure.net/cloudnativetesting/Content/Dynamic/iFwL3xKDLUX3.js:7:8307)\n at t. (https://afd.hosting-ms.portal.azure.net/cloudnativetesting/Content/Dynamic/iFwL3xKDLUX3.js:7:12460)\n at s (https://ms.portal.azure.com/Content/Dynamic/IMnrqdwlqpqW.js:17:2614)\n at Object.next (https://ms.portal.azure.com/Content/Dynamic/IMnrqdwlqpqW.js:17:1907)\n at o (https://ms.portal.azure.com/Content/Dynamic/IMnrqdwlqpqW.js:17:1680)", "extension": "Microsoft_Azure_CloudNativeTesting" }, "code": null }}

AB#1467961

[Feature Request] Is there a way to load test Postman Newman Scripts / API tests ?

[Feature Request] Is there a way to load test Postman Newman Scripts / API tests ? In our organisation we use Postman to test API functionality, it would be great if we can reuse those scripts to load test our api's rather than having to convert them to JMeter to make it work in Azure Load Testing Feature ..

[BUG] Load Test Metrics: Run Test and Configure Metrics: Re-run. Results of previous runs deleted.

Describe the bug
Run Test and Configure Metrics: Re-run. Results of previous runs deleted.

To Reproduce

Create Load Test
Run Load Test
Configure Metrics on Test Run

Results of all previous runs deleted

Expected behavior
Previous runs should not be deleted when reconfiguring test metrics.

Screenshots

Desktop (please complete the following information):

OS: Windows 10 (latest, Microsoft internal)
Browser: Edge
Version: Latest

Submitted by Conrad to Nick Trough in December 2021, who sent to product team.

AB#1467981

[BUG] test issue wit

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

[Feature Request] Load Testing: End-to-End Testing from ingress to services

Is your feature request related to a problem? Please describe.
The request is for the potential for end-to-end testing including database and network components.

Describe the solution you'd like
The ability to perform end-to-end load testing across a variety of Azure services. For example: APIM -> ASE -> Azure Function -> Cosmos DB -> etc., including underlying entities such as AKS clusters, networking, etc.

Describe alternatives you've considered

Additional context
This feature greatly expands the capabilities of Azure Load Test moving it toward an enterprise testing solution.

AB#1467989

Support for vnet integration for load testing apps sitting on private ip range

Please provide support for load testing azure resources sitting behind private ips.

[BUG] All result artifacts are not available in Azure pipeline after load test execution

Describe the bug
All artifacts as part of results are not available after load test execution via Azure Pipeline.
There is only one csv file as part of results.

Expected behavior
There is no index.html file and other required files as part of result.

AB#1465664

[BUG] jdbc connection

Describe the bug
Trying to call sql queries in SQL DB on VM (whitelisted IPs) but cannot

Error
,java.sql.SQLException: Cannot load JDBC driver class 'com.microsoft.sqlserver.jdbc.SQLServerDriver',172.18.59.5

Expected behavior
on Desktop jmeter it works fine

Additional context
Do workers have jdbc driver? or there should be different syntax to connect to DB?

AB#1640622

[Feature Request] Azure load testing work with plugin

When uploading JMeter script to test plan but in that script, we have to use some plugin.

For example:

jmeter-parallel-0.11.jar
jmeter-plugins-json-2.7.jar
Could you please tell me if there are any ways to do this or not?

Thank you

AB#1460227

[BUG] Test

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

[BUG] test bug

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

AB#1451966

[BUG] tests blocked in status Provisioning before failing without reason

Describe the bug
Yesterday January 17th, Test was not working, blocked in status "Provisioning" for a long time before going to status "Failed".
Nothing appears in the Logs (can we activate the log in the resource by the way?).
Today, tests are working normally.

To Reproduce

Go to tests part in the Azure Load Testing Resource (create in region North Europe)
Re-run a test that already worked successfully
See status.
Reproduced in any tests that were successful.

Expected behavior
test should show status "Done".

Additional context
Add any other context about the problem here.

AB#1463624

__GetSecret seems not to work

I've tried to use KeyVault icw Azure Load Testing, following the steps described at https://docs.microsoft.com/en-us/azure/load-testing/how-to-parameterize-load-tests#secrets, but the __GetSecret seems not to be working.

What I did was:

Enabled the system assigned managed identity for my Load Testing resource
Create a KeyVault with a secret
Added the SMI to the KeyVault Access Policies with all Secret Permissions
At the test parameters configured a Secret with Name = ClientId and Secret = https://.vault.azure.net/secrets/ClientId/
Added a User Defined Variables Config Element to my JMeter Test Plan and a variable with Name = ClientId and Value =
${__GetSecret(ClientId)}
In my request use ${ClientId}

After running my Test I found the string "${__GetSecret(ClientId)}" iso the secret in the log files of the webserver.

So it looks like __GetSecret did not work.

Searching the internet for "__GetSecret " also seem to return only the Azure Load Testing page.
Is there something special needed to get this working?

AB#1455476

[BUG] JSR223PreProcessor hangs execution

Describe the bug
It seems like tests are failing when a JSR223PreProcessor with a groovy script is used.

To Reproduce
Run this test script, containing two steps:

SetServerNameViaJSR223, which puts a url in the vars.
GET Google, which performs a GET on the url defined in the vars.

<jmeterTestPlan version="1.2" properties="5.0" jmeter="5.4.3">
  <hashTree>
    <TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="Test Plan" enabled="true">
      <stringProp name="TestPlan.comments"></stringProp>
      <boolProp name="TestPlan.functional_mode">false</boolProp>
      <boolProp name="TestPlan.tearDown_on_shutdown">true</boolProp>
      <boolProp name="TestPlan.serialize_threadgroups">false</boolProp>
      <elementProp name="TestPlan.user_defined_variables" elementType="Arguments" guiclass="ArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
        <collectionProp name="Arguments.arguments"/>
      </elementProp>
      <stringProp name="TestPlan.user_define_classpath"></stringProp>
    </TestPlan>
    <hashTree>
      <ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="Thread Group" enabled="true">
        <stringProp name="ThreadGroup.on_sample_error">continue</stringProp>
        <elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller" enabled="true">
          <boolProp name="LoopController.continue_forever">false</boolProp>
          <stringProp name="LoopController.loops">1</stringProp>
        </elementProp>
        <stringProp name="ThreadGroup.num_threads">1</stringProp>
        <stringProp name="ThreadGroup.ramp_time">1</stringProp>
        <boolProp name="ThreadGroup.scheduler">false</boolProp>
        <stringProp name="ThreadGroup.duration"></stringProp>
        <stringProp name="ThreadGroup.delay"></stringProp>
        <boolProp name="ThreadGroup.same_user_on_next_iteration">true</boolProp>
      </ThreadGroup>
      <hashTree>
        <JSR223PreProcessor guiclass="TestBeanGUI" testclass="JSR223PreProcessor" testname="SetServerNameViaJSR223" enabled="true">
          <stringProp name="cacheKey">true</stringProp>
          <stringProp name="filename"></stringProp>
          <stringProp name="parameters"></stringProp>
          <stringProp name="script">vars.put(&apos;ServerName&apos;, &apos;www.google.com&apos;)</stringProp>
          <stringProp name="scriptLanguage">groovy</stringProp>
        </JSR223PreProcessor>
        <hashTree/>
        <HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="GET Google" enabled="true">
          <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
            <collectionProp name="Arguments.arguments"/>
          </elementProp>
          <stringProp name="HTTPSampler.domain">${ServerName}</stringProp>
          <stringProp name="HTTPSampler.port"></stringProp>
          <stringProp name="HTTPSampler.protocol">https</stringProp>
          <stringProp name="HTTPSampler.contentEncoding"></stringProp>
          <stringProp name="HTTPSampler.path"></stringProp>
          <stringProp name="HTTPSampler.method">GET</stringProp>
          <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
          <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
          <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
          <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
          <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
          <stringProp name="HTTPSampler.connect_timeout"></stringProp>
          <stringProp name="HTTPSampler.response_timeout"></stringProp>
        </HTTPSamplerProxy>
        <hashTree/>
      </hashTree>
    </hashTree>
  </hashTree>
</jmeterTestPlan>

When running the test it stays in status "Executing".
When stopping the test it stays in status "Cancelling".

Expected behavior
A single request to be executed and the test status to change to Done.
or
A failed test with proper logs / errors.

Additional context
In JMeter this works on desktop this works fine.
I'm pretty sure this worked ~2 weeks ago.

AB#1461497

[BUG] test issue

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

AB#1451969

[Feature Request] Load Test: Multi-Azure-Service Testing

Is your feature request related to a problem? Please describe.
Azure load testing currently supports two chained services (client; server). Would be good to have more than two services maximum for testing :

Describe the solution you'd like
For example, consider this architecture and flow:

Services:
• API Management (APIM)
• App Service
• AKS Cluster running App Service

Flow:

User calls URL in APIM
APIM calls App Service
App Service calls AKS cluster

Notes:
Architecture cannot be tested end-to-end with single test. Two tests are required. APIM is viewed as the client, App Service as Server, AKS is not profiled. If there are more components the same applies, a maximum of two can be chained (labeled client, server).

Describe alternatives you've considered

Additional context

AB#1467988

[Feature Request] Service Connection backed up by Certificate is not supported for running load test via Azure pipeline

Is your feature request related to a problem? Please describe.
We are running load test via Azure pipeline where Service connection is using client secret. We observed Load Testing is not supported certificate in Service connection though certificate is recommended for security standpoint .

Describe the solution you'd like
Please provide support for Service Connection backed up by Certificate to run load test via Azure pipeline.

Additional context
We are getting below error if we use Service Connection backed up by Certificate.

`(node:1600) UnhandledPromiseRejectionWarning: Error: Endpoint auth data not present: cd80e187-3750-431f-8d01-f70e51cba41b

at Object.getEndpointAuthorizationParameter (/home/vsts/work/_tasks/AzureLoadTest_b61b360c-31ac-4389-922b-303ea8befde7/1.1.87/node_modules/azure-pipelines-task-lib/task.js:357:15)

at /home/vsts/work/_tasks/AzureLoadTest_b61b360c-31ac-4389-922b-303ea8befde7/1.1.87/models/mappers.js:264:42

at Generator.next ()

at /home/vsts/work/_tasks/AzureLoadTest_b61b360c-31ac-4389-922b-303ea8befde7/1.1.87/models/mappers.js:27:71

at new Promise ()

at __awaiter (/home/vsts/work/_tasks/AzureLoadTest_b61b360c-31ac-4389-922b-303ea8befde7/1.1.87/models/mappers.js:23:12)

at getAuthInputs (/home/vsts/work/_tasks/AzureLoadTest_b61b360c-31ac-4389-922b-303ea8befde7/1.1.87/models/mappers.js:260:12)

at Object.getInputParams (/home/vsts/work/_tasks/AzureLoadTest_b61b360c-31ac-4389-922b-303ea8befde7/1.1.87/models/mappers.js:215:5)

at /home/vsts/work/_tasks/AzureLoadTest_b61b360c-31ac-4389-922b-303ea8befde7/1.1.87/index.js:48:17

at Generator.next ()

(node:1600) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)

(node:1600) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.`

AB#1464126

[Feature Request] Publish service tag to allow load testing through NSG

Is your feature request related to a problem? Please describe.

When running a script, all traffic is blocked because it's coming from unknown IPs. We would like to be able to allow load testing through the NSGs via a service tag such that traffic can pass through from load testing without needing to open up all IPs for incoming traffic

Describe the solution you'd like
Publish a Service Tag (https://docs.microsoft.com/en-us/azure/virtual-network/service-tags-overview) that corresponds to Azure Load Testing

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Allow all traffic in
Log IPs used by Azure Load Testing and add them to NSG as a group (i.e. manually maintain IPs for incoming requests)

Additional context
N/A

[BUG] test issue public

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

AB#1452415

[BUG] Load Test description is not displaying when running load test via Azure Pipeline

Describe the bug
Created Azure Pipeline which uses configuration.yml file. This file has description.
while running load test, Observed Description is not updated in portal. Screenshot is attached for reference.

configuration.yml file
version: v0.1 testName: NotificationXLoad testPlan: Notification.jmx description: 'Notification XLoad Test Run' engineInstances: 1

Expected behavior
Unable to find Description value -> 'Notification XLoad Test Run' in below screeshots

Screenshots

AB#1463625

[Feature Request] Load test configuration yaml file not supporting variable to run Azure Load testing task via Azure Pipeline

Is your feature request related to a problem? Please describe.
We are having four applications which we want to load test via single Azure pipeline so tried to create variables in load test yaml file but it didn't work for me. To make it working created 4 separate yaml files corresponding to each service.

Describe the solution you'd like
Created below load test config yaml file

variables:
- template: variables.yaml 

version: v0.1
testName:  ${{ testname }}
testPlan: Notification.jmx
description: 'Notification Load Test Run'
engineInstances: 1
secrets:
  - name: notificationAuthCode
    value: https://abcd.azure.net/secrets/authCode

Created varaible.yml file

variables:
  testname: 'TestName'

Azure Load Testing execution task failed.

Please provide a way where we can make our load test config yaml file configurable

AB#1465662

[Feature Request] JMeter Templates for Azure Services / Protocols + Documentation

Is your feature request related to a problem? Please describe.
The documentation comes with a single JMeter file example for calling an HTTP endpoint.

Describe the solution you'd like
Is there any possibility of obtaining JMeter templates for different services in Azure, or is JMeter strictly an HTTP endpoint test tool (hopefully not)?

JMeter files are not user-friendly to configure, and most parameters are unclear. Is there clear documentation or a tool that can be used? The JMeter documentation says that there is a GUI tool but that it can only be used for non-load-testing. JMeter document is not completely user friendly, would be better if parameters were documented in the Azure Load Test documentation and that additional sample templates were supplied.

Templates, their configuration, and overall guidance in this area is much needed. Thanks!

Describe alternatives you've considered

Additional context

AB#1467986

Azure Load TestinLoad Tests Return 'Not Applicable' Failed status despite showing as successful in Exported Result Files & AppInsights[BUG]

Customer has raised an issue in Microsoft Q&A - https://docs.microsoft.com/en-us/answers/questions/684415/index.html

We are currently running a POC on the new Azure Load Testing feature (currently in preview) as a resource to run our jMeter tests against an in development API.

Unfortunately we are hitting a few problems with the output results displayed within a test run, namely the tests always fail with no start, end and duration time along with a 'Not Applicable' status when running against the APIs despite all tests running successfully directly within the jMeter GUI. The same tests when run in the Azure Load Testing tool also appear to show as 'Passed' when exporting results from the test screen.

Has anyone experienced this before and would be able to help, as this tool could be exactly what we are looking for if we can get it to output the correct results.

Our jMeter test is fairly simple and obtains a validation token which is used in the latter request, I have manually verified this is all authenticating correctly and the variables we are using in these tests are being correctly populated when run in Azure. Note this issue appears in both our GraphQL and REST API tests.

See attached screenshots:

Test Result
wD6KP.png

Exported Result
aTjYQ.png

We have also attempted to use the 'Ultimate Thread Group' plugin following a suggestion on StackOverflow, but unfortunately this also appears to return the same results (passes when run locally or through Blazemeter) but returns no result in Azure Load Testing despite the requests returning a successful result in AppInsights.

If anyone would be able to help at all this would be greatly appreciated. Unfortunately at the moment this is preventing us from reccomending Azure as a tool for our upcoming Load Testing strategy
.

AB#1461304

[BUG] Load Test: Run Test and then change references: previous results inaccessible

Describe the bug
Load Test: Run Test and then change references: previous results inaccessible

To Reproduce

Create Load Test
Add References
Run Load Test
Modify References (add additional, remove, etc. Any combination.)

Results from all previous test runs inaccessible (black banner saying that there was an error retrieving results).

Expected behavior
Previous runs should not be inaccessible when changing references.

Screenshots

Desktop (please complete the folowing information):

OS: Windows 10 (latest, Microsoft internal)
Browser: Edge
Version: Latest

Submitted by Conrad to Nick Trough in December 2021, who sent to product team.

AB#1467982

[Feature Request] Test Metrics Configuration scope limited: Can only be configured inside test run. Not reusable.

Is your feature request related to a problem? Please describe.

Metrics can only be configured in a test run, which means that Load Test must be run before configuring metrics.
This also means that metrics cannot be configured at Load Test level and used across test runs.

Describe the solution you'd like
Users should have the ability to configure test metrics at the load test level so that the configuration can be reused across test executions.

Describe alternatives you've considered
N/A

Additional context
If a large customer adopts Azure Load Test they might wish to put standards in place. For example, they might want to specify the metrics collected on Azure App Service load tests across the company. Therefore, and additional enhancement is to create portable test metric profiles: Scenario: Create load test for an app service; apply an existing test metric profile (this profile is used across the company for app services).

AB#1467980

Use own testing tool instead of JMeter [Feature Request]

Is your feature request related to a problem? Please describe.
JUnit is limited in what kinds of tests it can perform.

Describe the solution you'd like
I'd like to be able to plug in my own testing tool instead of JUnit, and otherwise take advantage of this solution.

microsoft / azure-load-testing Goto Github PK

azure-load-testing's Introduction

Azure Load Testing

Frequently Asked Questions

Known Issues for Public Preview

Contributing

Trademarks

azure-load-testing's People

Contributors

Stargazers

Watchers

azure-load-testing's Issues

##[debug]Evaluating condition for step: 'AzureLoadTest' ##[debug]Evaluating: SucceededNode() ##[debug]Evaluating SucceededNode: ##[debug]=> True ##[debug]Result: True Starting: AzureLoadTest

Task : Azure Load Testing Description : Automate performance regression testing with Azure Load Testing Version : 1.1.87 Author : Microsoft Corporation Help : https://docs.microsoft.com/azure/load-testing/tutorial-cicd-azure-pipelines#azure-load-testing-task

Recommend Projects

Recommend Topics

Recommend Org

##[debug]Evaluating condition for step: 'AzureLoadTest'
##[debug]Evaluating: SucceededNode()
##[debug]Evaluating SucceededNode:
##[debug]=> True
##[debug]Result: True
Starting: AzureLoadTest

Task : Azure Load Testing
Description : Automate performance regression testing with Azure Load Testing
Version : 1.1.87
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/load-testing/tutorial-cicd-azure-pipelines#azure-load-testing-task