adiralashiva8 / robotframework-metrics

Generation of metrics report based on robotframework output.xml

Robot Framework Metrics Report

Creates custom HTML report (dashboard view) by parsing robotframework output.xml file

---

• Sample Report link

• Whats new in v3.3.1 link

• v3.5.0 supports RF7 (need to be installed as mentioned in Step 1, case 3)

---

How it Works:

1. Read output.xml file using robot.api - ResultVisitor, ExecutionResult API Link

2. Get Suite, Test Case , Keyword , Status and Elapsed time values

3. Convert data to html report

---

How to use in project:

Step 1 Install robotmetrics

Case 1: Using pip

pip install robotframework-metrics==3.3.3

Case 2: Using setup.py (clone project and run command within root)

python setup.py install

Case 3: For latest changes (pre-release or changes in master: RF7 support added in master)

pip install git+https://github.com/adiralashiva8/robotframework-metrics

Step 2 Execute robotmetrics command to generate report

Case 1: No change in output.xml file name (assumig user is in same folder)

robotmetrics

Case 2: output.xml under 'Result' folder

robotmetrics --inputpath ./Result/ --output output1.xml

For more info on command line options use:

robotmetrics --help

---

Generate robotframework-metrics after execution

Execute robotmetrics command after suite or test execution as follows:

• Create .bat (or) .sh file with following snippet

  robot test.robot &
  robotmetrics [:options]
  

  & is used to execute multiple command's in .bat file

• Modify robotmetrics command as required and execute .bat file

• Robotframework metrics will be created after execution

---

If you have any questions / suggestions / comments on the report, please feel free to reach me at

• Email: [email protected]

---

Special Thanks To:

Idea, Guidance and Support:

• Steve Fisher
• Goutham Duduka

Contributors:

1. Pekka Klarck [Author of robotframework]

2. Ruud Prijs

3. Jesse Zacharias

4. Bassam Khouri

5. Francesco Spegni

6. Sreelesh Kunnath

Feedback:

1. Mantri Sri
2. Prasad Ozarkar
3. Suresh Parimi
4. Amit Lohar
5. Robotframework community users

---

⭐ repo if you like it

---

robotframework-metrics's Issues

Email: Generate email with pie charts similar as dashboard

As a user I want to generate email similar as robotframework-metrics dashboard like

    Suite   Total   Pass  Fail
    Test    Total   Pass  Fail
    Kw      Total   Pass  Fail
 
   Pie charts
   Suite   Test   KW

So that I can share my email with supervisor

Note: I dont want to include top 10 perfromance stats in execution status email

MetricsReport: Generate report with time-stamp

As I user I want to generate report with timestamp so I can refer next time

EmailStatistics: Modify email template

Change UI
Provide more information in email

calling robotmetrics without any arguments raises an exception

When calling robotmetrics with no arguments, an exception is raised.

$ robotmetrics 
Traceback (most recent call last):
  File "/Users/bassam.khouri/venv3/bin/robotmetrics", line 11, in <module>
    sys.exit(main())
  File "/Users/bassam.khouri/venv3/lib/python3.7/site-packages/robotframework_metrics/runner.py", line 102, in main
    generate_report(opts)
  File "/Users/bassam.khouri/venv3/lib/python3.7/site-packages/robotframework_metrics/robotmetrics.py", line 92, in generate_report
    result = ExecutionResult(output_name)
  File "/Users/bassam.khouri/venv3/lib/python3.7/site-packages/robot/result/resultbuilder.py", line 50, in ExecutionResult
    return _single_result(sources[0], options)
  File "/Users/bassam.khouri/venv3/lib/python3.7/site-packages/robot/result/resultbuilder.py", line 75, in _single_result
    raise DataError("Reading XML source '%s' failed: %s" % (unic(ets), error))
robot.errors.DataError: Reading XML source './output.xml' failed: No such file or directory

Here is my environment

$ python --version
Python 3.7.2

$ robot --version
Robot Framework 3.1 (Python 3.7.2 on darwin)

$ pip freeze | grep robotframework-metrics
robotframework-metrics==3.1.1

I'm not familiar with the source code so I don't know what files are required by the tool, but a more human readable error message should be displayed.

Dashboard: Show log of my company in report

As a user I want to show my company logo in report

Tabs: Suite name should display as link

Not able to identify suite or test name as link. Modify font color to look likes link

Font style as well?

Datatable: Missing show no. of entries

As a user I want to view no. of entries in data table ( show no. of entires is missing.)

Report: Send email when robotmetrics file is executed

As a user I want system to send email with basic result info when .py file is executed

Sending mail via Mailgun API

It would be nice to be able to use an API such as Mailgun in order to send e-mails.
For instance Mailgun does accept html formatted e-mails, addition to plain text e-mails.

https://documentation.mailgun.com/en/latest/quickstart-sending.html#send-with-smtp-or-api

Report Generation: Read output.xml irrespective of folder location in project

Expected Result: As a report generator I should be able to generate report irrespective of output folder location.

In my case my .xml file will be like merged.xml | ouput123.xml etc.,

Move the html, css and js content into a separate variables

Hi,

I think that the robotframework-metrics is a great tool!

Do you have plan to move out the java scripts, html and css parts into a separate modules?
Example variables:

• head_content
• icons_txt

Example js sources

• <script src="https://www.gstatic.com/charts/loader.js" type="text/javascript"></script>
• <script type="text/javascript">google.charts.load('current', {packages: ['corechart']});</script>

At this moment you have to have internet connection do gain access to external js sources.
And those long html variables make the code very hard to read

If there's no such plan at the moment i can do it by myself

Regards
Patryk Walaszkowski

Support Data -driven testing

While reviewing the results I have noticed some of our cases are data-driven cases and hence appearing as a single test case in your report. Since I have included a tag in robot framework report I am able to see them separately. can we include tags as well in our report so that data-driven testing would also be supported.? Please share your thoughts.
Given below is the format of a data-driven test case.
*** Settings ***
Test Template Example keyword

Variables
&{dictionary1} argument=first tag=first tag
&{dictionary2} argument=second tag=second tag
&{dictionary3} argument=third tag=third tag
&{dictionary4} argument=fourth tag=fourth tag

*Keywords
Example keyword
[Arguments] ${dictionary}
log ${dictionary.argument}
Set tags ${dictionary.tag}

*** Test Cases **
Templated test case
[Template] Example keyword
${dictionary1}
${dictionary2}
${dictionary3}
${dictionary4}

Dashboard: Display charts on page load in dashboard

Expected:
Display charts in dashboard on page load, Instead of clicking on buttons for chart

Keyword Metrics may be configurable

1. It good if you can make keyword metrics as configurable to the user it will help to improve performance.It is taking long time when we have more test cases.
2. Its good if the metrics can be accessible from any remote machine, if I share only the metrics.html to anyone.

Dashboard: Show more info

As a user I want to know the report generated time,Eexcution time, browser and system info

MetricsReport: Handle content-security exceptions

Add content of report to unsafe-inline

Parse the error message when my XML contains Chinese

Parse the error message when my XML contains Chinese，The error message:

Converting .xml to .html file. This may take few minutes...
1 of 6: Capturing dashboard content...
2 of 6: Capturing suite metrics...
3 of 6: Capturing test metrics...
4 of 6: Capturing keyword metrics...Traceback (most recent call last):
File "robotmetrics.py", line 849, in
result.visit(KeywordResults())
File "/Library/Python/2.7/site-packages/robot/result/executionresult.py", line 120, in visit
visitor.visit_result(self)
File "/Library/Python/2.7/site-packages/robot/result/visitor.py", line 44, in visit_result
result.suite.visit(self)
File "/Library/Python/2.7/site-packages/robot/model/testsuite.py", line 168, in visit
visitor.visit_suite(self)
File "/Library/Python/2.7/site-packages/robot/model/visitor.py", line 87, in visit_suite
suite.tests.visit(self)
File "/Library/Python/2.7/site-packages/robot/model/itemlist.py", line 72, in visit
item.visit(visitor)
File "/Library/Python/2.7/site-packages/robot/model/testcase.py", line 74, in visit
visitor.visit_test(self)
File "/Library/Python/2.7/site-packages/robot/model/visitor.py", line 108, in visit_test
test.keywords.visit(self)
File "/Library/Python/2.7/site-packages/robot/model/itemlist.py", line 72, in visit
item.visit(visitor)
File "/Library/Python/2.7/site-packages/robot/model/keyword.py", line 118, in visit
visitor.visit_keyword(self)
File "/Library/Python/2.7/site-packages/robot/model/visitor.py", line 129, in visit_keyword
if self.start_keyword(kw) is not False:
File "robotmetrics.py", line 834, in start_keyword
table_td.string = str(test_name)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)

Externalize Configuration

Configuration of the report (i.e. disabling emails, link to report logo) is done in the robotmetrics.py script directly. Which means, you would have to redo all configuration every time a newer version of the script is published.

It would be nice, if the configuration could be done outside the script in a configuration file (properties file, xml, json or anything). That way, you could upgrade the script to a newer version repeating customization.

Report Generation: Improve performance of report generation time

Expected: Improve performance of report generation using Thread or Multiprocess

Metrics Report: Show Suite Metrics in report

Expected: As a viewer I want to see Suite metrics in metrics report

How to get metrics.html without timestamp?

Would be great to have the timestamp in generated result file optionally.

Remove send email functionality

Remove send email functionality of robotframework-metrics. As a user I will you other options to send email

Email: Metrics stats are not included in email

When there are no keywords in project, Metrics stats are not included in generated email

Logo gets distorted

Logo gets distorted due to the both width and height being defined in the style.
<img src="%s" style="height:18vh!important;width:95%;"/>

If the width is omitted, then the logo aspect ratio remains correct.
<img src="%s" style="height:18vh!important;"/>

This would work fine when logo is provided as a square image. (ex. 512x512 pixels.)

Dashboard: Pie charts show only ten record data when metrics tab is selected first and view pie chart

Describe the bug
Only ten records data Pie chart is shown when I select 'Test Metrics' tab before clicking on "Click Here For Tests Status Chart".

To Reproduce
Steps to reproduce the behavior:

1. Generate Metrics Report ( more than 10 records)
2. Click on Tests Metrics Tab
3. Click on Dashboard
4. Click "Click Here For Test Status Chart"

Expected behavior
Pie chart should display based on all row status data

Report: Use bootstrap dashboard example

Use bootstrap dashboard example, so user dont need to use other css links

Not able to create robotmetrics report

Hi Shiva, I have downloaded and copy the python file in my project folder and trying to run python robotmetrics.py but nothing is responding.

Anything to install?

right now we are displaying the Test case name in our Dashboard, can we display the Test case ID of behalf of Test case Name ? Because name contains very large text and it is not fitting properly in Dashboard.

Actual: Lengthy x-axis label names are not auto fitting in dashboard
Expected: Show ids instead of name in bar graph x-axis label

Test Metrics: Include error message in table

As a user I want to have error message in Test Metrics table, so when I share my report with others this helps other to figure out the issue instead of looking into log.html

Parse error when output.xml contains latin characters

It seems that the code is not encoding free (Given #41)

table_td.string = str(kw.kwname)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe0' in position 36: ordinal not in range(128)

On solution may be to sanitise the xml input to make robotframework-metrics error free. It is frustrating that the script does not go throught for 1 unrecognised character. However it would not matter if that character was replace by ?.

u.encode('ascii', 'replace')

Dashboard: Show top 5 suite failures

As a user I want to view top 5 suite failure which helps me to analyze top failures area.

This feature requested by one of the robotframework-user.

Writing unit tests

As a developer, I would like the ability to make changes, and run some tests so I can gain confidence my changes did not introduce any regression issues.

Note: I tried following the Conextra card template as this is more of a Story than a defect

ZeroDivisionError

Traceback (most recent call last):
File "robotmetrics.py", line 411, in
kwpp = round(passed_keywords*100.0/total_keywords,2)
ZeroDivisionError: float division by zero

This error is coming when robot file don't have any keywords but only test cases.

Dashboard: Use Open Source Charts Instead of CanvasJs

Since CanvasJs is licenced use open source charts

Suite Metrics: Show log when click on suite name

As a 'Metrics Report' user I want to view suite logs when I click on suite name in 'Suite Metrics' Table

Report: Show loading indicator while loading report is loading

Show loading indicator while loading report

Metrics: Show colors of PASS and FAIL texts

For better view show colors for PASS and FAIL texts

The Dashboard with the Top 10 Performance of all suites and Test case it is overriding with something can you move the Text of Top 10 Performance at top of the Table ?

Actual: Chart title "Top 10 Performance" are overlaped with bar graph.
Expected: Show Chart title outside of chart same as Status chart

MetricsReport: Allow user to specific ouput file location

As I user i want to specify my .xml file name and location instead of hotcode location

Encoding error during log generation (character maps to <undefined>)

Hi!
I'm getting an encoding error during log generation (probably because of cyrillic characters).
I tried setting the sysdefaultencoding to UTF-8 but it does not work because of python 3+.

Python: 3.7.4
robotmetrics: 3.1.4 (pulled from github)
OS: Win10

Security Query

hi,
does the robotmetrics, upload any data to the following sites or any 3rd party sites apart from using the metrics query? In short does robotmetrics sends any client data outward to internet, if yes, then what kind of data?

<!doctype html><html lang="en">
    <head>
        <link rel="shortcut icon" href="https://png.icons8.com/windows/50/000000/bot.png" type="image/x-icon" />
        <title>RF Metrics</title>
        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width, initial-scale=1">
        <link href="https://cdn.datatables.net/1.10.19/css/jquery.dataTables.min.css" rel="stylesheet"/>
        <link href="https://cdn.datatables.net/buttons/1.5.2/css/buttons.dataTables.min.css" rel="stylesheet"/>
        <link href="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/4.1.3/css/bootstrap.min.css" rel="stylesheet"/>
        <link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css" rel="stylesheet">
       <script src="https://code.jquery.com/jquery-3.3.1.js" type="text/javascript"></script>
        <!-- Bootstrap core Googleccharts -->
       <script src="https://www.gstatic.com/charts/loader.js" type="text/javascript"></script>
       <script type="text/javascript">google.charts.load('current', {packages: ['corechart']});</script>
       <!-- Bootstrap core Datatable-->
        <script src="https://cdn.datatables.net/1.10.19/js/jquery.dataTables.min.js" type="text/javascript"></script>
        <script src="https://cdn.datatables.net/buttons/1.5.2/js/dataTables.buttons.min.js" type="text/javascript"></script>
        <script src="https://cdn.datatables.net/buttons/1.5.2/js/buttons.flash.min.js" type="text/javascript"></script>
        <script src="https://cdnjs.cloudflare.com/ajax/libs/jszip/3.1.3/jszip.min.js" type="text/javascript"></script>
        <script src="https://cdnjs.cloudflare.com/ajax/libs/pdfmake/0.1.36/pdfmake.min.js" type="text/javascript"></script>
        <script src="https://cdnjs.cloudflare.com/ajax/libs/pdfmake/0.1.36/vfs_fonts.js" type="text/javascript"></script>
        <script src="https://cdn.datatables.net/buttons/1.5.2/js/buttons.html5.min.js" type="text/javascript"></script>
        <script src="https://cdn.datatables.net/buttons/1.5.2/js/buttons.print.min.js" type="text/javascript"></script>

Utf8 issue on robotframework-metrics

I encountered an utf8 issue after the generation of my test suite metrics (126 tests).

python robotmetrics.py -email false -inputpath
"MYPATH" -output output.xml -report report.html -log l
og.html
Converting .xml to .html file. This may take few minutes...
1 of 6: Capturing dashboard content...
2 of 6: Capturing suite metrics...
3 of 6: Capturing test metrics...
4 of 6: Capturing keyword metrics...Traceback (most recent call last):
File "robotmetrics.py", line 848, in
result.visit(KeywordResults())
File "C:\Python27\lib\site-packages\robot\result\executionresult.py", line 118
, in visit
visitor.visit_result(self)
File "C:\Python27\lib\site-packages\robot\result\visitor.py", line 44, in visi
t_result
result.suite.visit(self)
File "C:\Python27\lib\site-packages\robot\model\testsuite.py", line 161, in vi
sit
visitor.visit_suite(self)
File "C:\Python27\lib\site-packages\robot\model\visitor.py", line 86, in visit
_suite
suite.suites.visit(self)
File "C:\Python27\lib\site-packages\robot\model\itemlist.py", line 76, in visi
t
item.visit(visitor)
File "C:\Python27\lib\site-packages\robot\model\testsuite.py", line 161, in vi
sit
visitor.visit_suite(self)
File "C:\Python27\lib\site-packages\robot\model\visitor.py", line 86, in visit
_suite
suite.suites.visit(self)
File "C:\Python27\lib\site-packages\robot\model\itemlist.py", line 76, in visi
t
item.visit(visitor)
File "C:\Python27\lib\site-packages\robot\model\testsuite.py", line 161, in vi
sit
visitor.visit_suite(self)
File "C:\Python27\lib\site-packages\robot\model\visitor.py", line 86, in visit
_suite
suite.suites.visit(self)
File "C:\Python27\lib\site-packages\robot\model\itemlist.py", line 76, in visi
t
item.visit(visitor)
File "C:\Python27\lib\site-packages\robot\model\testsuite.py", line 161, in vi
sit
visitor.visit_suite(self)
File "C:\Python27\lib\site-packages\robot\model\visitor.py", line 87, in visit
_suite
suite.tests.visit(self)
File "C:\Python27\lib\site-packages\robot\model\itemlist.py", line 76, in visi
t
item.visit(visitor)
File "C:\Python27\lib\site-packages\robot\model\testcase.py", line 74, in visi
t
visitor.visit_test(self)
File "C:\Python27\lib\site-packages\robot\model\visitor.py", line 108, in visi
t_test
test.keywords.visit(self)
File "C:\Python27\lib\site-packages\robot\model\itemlist.py", line 76, in visi
t
item.visit(visitor)
File "C:\Python27\lib\site-packages\robot\model\keyword.py", line 118, in visi
t
visitor.visit_keyword(self)
File "C:\Python27\lib\site-packages\robot\model\visitor.py", line 129, in visi
t_keyword
if self.start_keyword(kw) is not False:
File "robotmetrics.py", line 837, in start_keyword
table_td.string = str(kw.kwname)

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 40:
ordinal not in range(128)

Dashboard: Remove pie charts

Pie charts are duplicate content of stats. Instead of pie chart display percentage within tiles

process multiple output.xml

Would it be possible to take as input multiple output.xml files? Similarly to what rebot does. At the moment the only thing I was able to do was to merge the several output.xml into a single file (with a single <robot ...> element) and pass it to robotmetrics. The nasty part is removing the multiple copies of the <robot ...> element (one per output.xml file), then merge the children elements, then wrap all the children with a single outer <robot ...> element. It would be nice if the tool could take care of that.

Dashboard: Dashboard text should link to respective tab

When I click Suite text in statistic it should redirect to Suite Tab

Performance: Remove Start And End time from metrics tables

Remove start and end time to decrease datatable content to improve page loading performance

Make report offline compatiable

Accepatance Criteria: Decrease usage of internet calls for robotframework-metrics report

Requirement: As a user I should be able to generate | open robotframework metrics without or with less usage of internet call to increase performance of report loading (offline compatiable)

Reference: May be this issue going to implement the requirement #53

Disable Email Notification

I could not figure out how to disable sending the report in an email.

I excluded all parts regarding emails from the script. Maybe you could provide a switch for enabling/disabling email notification.

Make the library pip installable.

Instead of downloading the file, library should be installed via pip.
The HTML content can be separated out within variables as constants separately.

The file can be made an executable and can be run using the executable name. Ex:
robot-metrics [:options]

Metrics: Provide option to export metrics

Datatabel supports to export table content.
For more info see: https://datatables.net/extensions/buttons/examples/initialisation/export.html