microsoftedge / browserefficiencytest Goto Github PK

View Code? Open in Web Editor NEW

65.0 65.0 45.0 605 KB

BrowserEfficiencyTest is a Selenium WebDriver based web browser test automation project.

License: Other

C# 100.00%

browserefficiencytest's People

Contributors

Stargazers

Watchers

Forkers

nolanlawson brendyna brheenan zkcaj ekhavana calebrouleau bjjones tychosnose zeroinfinite charlieyqin snped f7en rodrickmakore imhomos drbasic jjlee9 paulhuan raget sooraj77 ondrejsotolar majkl47 xsustek stefan-malcek sbojnak paulritzman dahaoniuniu jewilder jefferyq connectionmaster saranshgupta20-zz huachunbo ykankaya johnarthur1 ninovalboskoni howmanysmall magicandre1981 isabella232 mikehoffms qqzsxyz1 seanpm2001

browserefficiencytest's Issues

CNN Scenario doesn't always find headline

The last couple days, the CNN headline link was able to be found by finding the zn-banner class. Today it is no longer working. Today it is under a different class name. We need to find a more robust way to find the top headline on CNN or possibly settle for any article on CNN.

Edge Fails to Start (Chrome Works)

@TychosNose This is following your most recent commit. I pulled down the changes, rebuilt the BrowserEfficiencyTest solution, and attempted to run the tests.

Chrome ran without issue. Once an Edge window opens, the executable gets terminated. Perhaps this is an issue with the 15063 MicrosoftWebDriver executable, rather than the BrowserEfficiencyTest solution?

PS C:\Users\jjdsa\Desktop\Battery Tests\BrowserEfficiencyTest\BrowserEfficiencyTest\bin\Debug> ./BrowserEfficiencyTest.exe -b edge chrome -i 2 -rp C:\Traces -ms cpuUsage -s FastScenario WikipediaUnitedStates
BrowserEfficiencyTest Version: 11.1.0.0
[2017-05-25 10:21:50] --- OS Version ------
[2017-05-25 10:21:50]   BuildLabEx: 15063.0.x86fre.rs2_release.170317-1834
[2017-05-25 10:21:50]   EditionID: Professional
[2017-05-25 10:21:50]   ProductName: Windows 10 Pro
[2017-05-25 10:21:50] ---------------------
[2017-05-25 10:21:50] Pausing before starting first tracing session to reduce interference.
Attempting to connect with the trace controller...
Successfully connected to trace controller.
[2017-05-25 10:22:02] Starting Test Pass
[2017-05-25 10:22:02] Iteration: 0 ------------------
[2017-05-25 10:22:03]  Launching Browser Driver: 'chrome'
Starting ChromeDriver 2.29.461591 (62ebf098771772160f391d75e589dc567915b233) on port 52474
Only local connections are allowed.
[2017-05-25 10:22:08]   Executing - Scenario: FastScenario  Iteration: 0  Attempt: 0  Browser: chrome  MeasureSet: cpuUsage
[2017-05-25 10:22:10]     Scenario FastScenario returned in 1.5099187 seconds. Sleep for remaining 8.4900813 seconds.
[2017-05-25 10:22:18]   Completed - Scenario: FastScenario  Iteration: 0  Attempt: 0  Browser: chrome  MeasureSet: cpuUsage
[2017-05-25 10:22:18]  New Tab: Attempting to create a new tab using the javascript method window.open()
[2017-05-25 10:22:20]   Executing - Scenario: WikipediaUnitedStates  Iteration: 0  Attempt: 0  Browser: chrome  MeasureSet: cpuUsage
[2017-05-25 10:22:41]     Scenario WikipediaUnitedStates returned in 22.8342525 seconds. Sleep for remaining 7.1657475 seconds.
[2017-05-25 10:22:48]   Completed - Scenario: WikipediaUnitedStates  Iteration: 0  Attempt: 0  Browser: chrome  MeasureSet: cpuUsage
[2017-05-25 10:22:48]  SUCCESS!  Completed Browser: chrome  Iteration: 0  Attempt: 0  MeasureSet: cpuUsage
[2017-05-25 10:22:48]   Pausing between tracing sessions to reduce interference.
[2017-05-25 10:23:08]  Launching Browser Driver: 'edge'
[10:23:08.195] - Listening on http://localhost:52534/

Unhandled Exception: System.IO.FileNotFoundException: MicrosoftWebDriver.exe
   at System.Diagnostics.FileVersionInfo.GetVersionInfo(String fileName)
   at BrowserEfficiencyTest.RemoteWebDriverExtension.CreateDriverAndMaximize(String browser, String browserProfilePath) in C:\Users\jjdsa\Desktop\Battery Tests\BrowserEfficiencyTest\BrowserEfficiencyTest\RemoteWebDriverExtension.cs:line 327
   at BrowserEfficiencyTest.ScenarioRunner.Run() in C:\Users\jjdsa\Desktop\Battery Tests\BrowserEfficiencyTest\BrowserEfficiencyTest\ScenarioRunner.cs:line 213
   at BrowserEfficiencyTest.Program.Main(String[] args) in C:\Users\jjdsa\Desktop\Battery Tests\BrowserEfficiencyTest\BrowserEfficiencyTest\Program.cs:line 46
PS C:\Users\jjdsa\Desktop\Battery Tests\BrowserEfficiencyTest\BrowserEfficiencyTest\bin\Debug>

YahooNews scenario fails frequently

I've observed recently that YahooNews scenario causes the workload to abort quite frequently, the most common exception thrown being related to Timeout:

and another exception sometimes being:

I'm using windows public build 15063, and the latest web driver available.

Refset peak column is difficult to use

The result column for peak refset is the exact string from WPA, which is "###### at XYZs", stating both the peak and the time it occurred at. While useful to know the time of the peak, it also makes aggregation difficult as you need to split and convert the string

No Metrics found for measure set cpuUsage on ETL

I believe I have followed all setup instructions correctly, however none of the cpuUsage data appears to be collected. Also, the generated CSV is empty:

Command

BrowserEfficiencyTest.exe -b edge chrome -i 2 -rp C:\Traces -ms cpuUsage -s FastScenario WikipediaUnitedStates

The tests opened the two browsers, and appeared to run successfully.

Output

Completed Test Pass
- Starting performance processing. -
- Processing ETL C:\Traces\chrome_fastscenario-wikipediaunitedstates_0_cpuUsage_20170509_172604.etl using profile .\MeasureSetDefinitionAssets\CpuUsage.wpaProfile
WPA 10.0.15063.137 (WinBuild.160101.0800)
No Metrics found for measure set cpuUsage on ETL C:\Traces\chrome_fastscenario-wikipediaunitedstates_0_cpuUsage_20170509_172604.etl.
- Processing ETL C:\Traces\chrome_fastscenario-wikipediaunitedstates_0_cpuUsage_20170509_172933.etl using profile .\MeasureSetDefinitionAssets\CpuUsage.wpaProfile
WPA 10.0.15063.137 (WinBuild.160101.0800)
No Metrics found for measure set cpuUsage on ETL C:\Traces\chrome_fastscenario-wikipediaunitedstates_0_cpuUsage_20170509_172933.etl.
- Processing ETL C:\Traces\chrome_fastscenario-wikipediaunitedstates_1_cpuUsage_20170509_173146.etl using profile .\MeasureSetDefinitionAssets\CpuUsage.wpaProfile
WPA 10.0.15063.137 (WinBuild.160101.0800)
No Metrics found for measure set cpuUsage on ETL C:\Traces\chrome_fastscenario-wikipediaunitedstates_1_cpuUsage_20170509_173146.etl.
- Processing ETL C:\Traces\edge_fastscenario-wikipediaunitedstates_0_cpuUsage_20170509_173038.etl using profile .\MeasureSetDefinitionAssets\CpuUsage.wpaProfile
WPA 10.0.15063.137 (WinBuild.160101.0800)
No Metrics found for measure set cpuUsage on ETL C:\Traces\edge_fastscenario-wikipediaunitedstates_0_cpuUsage_20170509_173038.etl.
- Processing ETL C:\Traces\edge_fastscenario-wikipediaunitedstates_1_cpuUsage_20170509_173251.etl using profile .\MeasureSetDefinitionAssets\CpuUsage.wpaProfile
WPA 10.0.15063.137 (WinBuild.160101.0800)
No Metrics found for measure set cpuUsage on ETL C:\Traces\edge_fastscenario-wikipediaunitedstates_1_cpuUsage_20170509_173251.etl.

Generated CSV

EtlFileName,Scenario,Iteration,Browser,DateStamp,TimeStamp,MeasureSet,Measure,Result

Scroll function sometimes times out on ad heavy sites

The scroll function sometimes times out causing the webdriver to throw and exception. This seems to happen most often on ad heavy sites such as TechRadar. CNN seems induce this issue regularly as well.

Powertest disconnects from the NamedPipeServerStream uncleanly

When PowerTest completes, it does not disconnect properly from the NamedPipeServerStream resulting in a broken pipe. This prevents another instance of PowerTest from reconnecting to the NamedPipeServer.

Unable to run from elevated command prompt with some platforms

As Todd noted in pull request #4, not all WebDriver platforms support being run from an elevated command prompt.
Here's what I've found works from an elevated command prompt:

Edge Insider Preview
FireFox

Here's what doesn't work from an elevated command prompt

Chrome
Edge Fall Release

I haven't tested Opera.
The case for being able to run from an elevated command prompt is to support controlling the Windows Performance Recorder which requires elevated privileges. There workarounds for this such as Todd mentions. This issue is opened to track this issue.

CSV files produced when using -tc and -ms are always stored in same directory as BrowserEfficiencyTest.exe

According to the instructions, ETLs and processed files go to the folder specified in the -tc option. However the processed csv output is always stored in the same directory as BrowserEfficiencyTest.exe.

PowerTest only processes ETLs that are in the current working directory

PowerTest only processes energy data from ETLs that are in the same directory as PowerTest.exe. This is a problem since the ElevatorServer currently only saves ETL files to the directory that ElevatorServer.exe is in. This causes PowerTest to not process any ETLs that were captured during the currently run test pass.

Firefox webdriver crashes when launched

Issue is with Firefox 47.0 and Webdriver 2.53
selenium-2-53-not-working-on-firefox-47

Need to figure out the right columns to export

Some columns don't make a lot of sense as it currently works