getgauge / html-report

HTML report generation plugin for Gauge

html-report

Features

• A comprehensive test results report template prepared in a html format providing the overall summary with drill down of the test cases executed and effort spent during the testing for each stage and feature.
• It provides the details for the defects found during the run.
• It indicates the tests by color code - failed(red), passed(green) and skipped(grey).
• The failure can be analyzed with the stacktrace and screenshot(captures unless overwritten not to).
• The skipped tests can be analyzed with the given reason.
• Custom Messages allows users to add messages at runtime.

Sample HTML Report documemt

Installation

gauge install html-report

• Installing specific version

gauge install html-report --version 4.3.1

Offline installation

• Download the plugin from Releases

gauge install html-report --file html-report-4.3.1-linux.x86_64.zip

Build from Source

Requirements

• Golang

Compiling

Download dependencies

go get -t ./...

Compilation

go run build/make.go

For cross-platform compilation

go run build/make.go --all-platforms

Installing

After compilation

go run build/make.go --install

Installing to a CUSTOM_LOCATION

go run build/make.go --install --plugin-prefix CUSTOM_LOCATION

Creating distributable

Note: Run after compiling

go run build/make.go --distro

For distributable across platforms: Windows and Linux for both x86 and x86_64

go run build/make.go --distro --all-platforms

New distribution details need to be updated in the html-report-install.json file in the gauge plugin repository for a new version update.

Configuration

The HTML report plugin can be configured by the properties set in the env/default.properties file in the project.

The configurable properties are:

gauge_reports_dir

• Specifies the path to the directory where the execution reports will be generated.

• Should be either relative to the project directory or an absolute path. By default it is set to reports directory in the project

overwrite_reports

• Set to true if the reports must be overwritten on each execution maintaining only the latest execution report.

• If set to false then a new report will be generated on each execution in the reports directory in a nested time-stamped directory. By default it is set to true.

GAUGE_HTML_REPORT_THEME_PATH

• Specifies the path to the custom theme directory.

• Should be either relative to the project directory or an absolute path. By default, default theme shipped with gauge is used.

gauge_minify_reports

• Set to true if the generated HTML files needs to be minified. This helps avoid creating huge reports if the project suite is huge.

Report re-generation

If report generation fails due to some reason, we don't have to re-run the tests again.

Gauge now generates a last_run_result file in the .gauge folder under the Project Root. There is also a symlink to the html-report executable available in <gauge_reports_dir>/html-report.

To regenerate the report

• Navigate to the reports directory
• move the html-report file to .gauge directory
• Navigate to the .gauge directory
• run ./html-report --input=last_run_result --output="/some/path"

Note: The output directory is created. Take care not to overwrite an existing directory. The html-report executable and last_run_result will be generated only if the property save_execution_result is set to true. While regenerating a report, the default theme is used. A custom can be used if --theme flag is specified with the path to the custom theme.

License

This program is licensed under:

Apache License, Version 2.0

Copyright

Copyright 2015 ThoughtWorks, Inc.

html-report's Issues

Is it possible to add spec name in the header?

Hi,

Is it possible to add a spec name (filename) next to the specHeading and timestamp. It would help in finding out where exact to go when debugging especially when there's a lot of specs with similar heading names.

Rendering breaks if beforeSuite hooks fail

It should instead show the error response / stack trace from beforeSuite.

Error message in case of failure doesn't honor newlines

The result.js has the error message including newlines. But this isn't shown properly in report.

Header contents overlap on small screens (e.g. iPad)

The header at the top of the HTML report has elements that overlap when the browser window is small, or when shown on a device with a small screen. For instance, this is what it looks like when shown with a width of 1024 (the width of an iPad in landscape mode):

As you can see, the number skipped overlaps with a number of gray lines. Also, the number of total specs are beoverlapping with the pie chart.

This is what it looks like with a width of 768 (the width of an iPad in portrait mode):

At this point, the Skipped row has gone to a new line, and the total specs are completely overlapping the pie chart. Also, note that the space to the left of the pie chart hasn't changed.

This should be fixed so that we don't have any overlap at any screen size and appears better when reduced width is available. Also, vertical space should be considered as well.

gauge message and error messages not honoring line endings.

With the new html-report 2.0.0, when a step fails, the error message that displays in the error-message class does not honor the line endings placing the entire message on a single line. This doesn't appear to affect the stacktrace portion of the output.

This also affects the custom messages.

Example:
Gauge.writeMessage("Hello: \n World:");
Should be:
Hello:
World:

Appears as:
Hello: World:

I don't know if the custom message worked before, but the error messages definitely did.

Show number of scenarios, instead of specs, in summary

Currently, the top panel shows the summary of count of specs passed/failed.

Each scenario is a test. I would want to know how many of the total scenarios passed/failed.
When I see the total count of passes/failures, it makes more sense to have the number of scenarios.

Html-report aligning issue with stacktrace and screenshot

When stacktrace is wide, the screenshot isn't shown. so will have to wrap it.

Wrong colors in reports

Expected behavior

The circular icon for scenario should be shown green / red if the scenario has executed. It should be greyed out only if scenario is skipped.

Actual behavior

If any one scenario in the spec is skipped, all the scenarios are greyed out.
For instance, in the screenshot below, all the steps in the scenario are successfully executed. But it is still shown as grey instead of green.

Steps to reproduce

1. Create a spec with multiple scenarios.
2. Add an unimplemented step in any one of the scenario so that the scenario is skipped.
3. Run gauge specs and open reports.

Gauge version

Gauge version: 0.4.1

Plugins
-------
html-report (2.1.0)
java (0.3.4)
ruby (0.1.1)

Tests are executed with sub-set of table rows should be indicated in the report

Expected behavior

If a sub-set of table rows are executed, the result html should highlight only the executed rows with green/red. The ones which are not-executed should be greyed out and not clickable.

Actual behavior

If a sub-set of table rows are executed, the result html highlights the ones which are not-executed also in green and are clickable.

Steps to reproduce

• Create a spec with table data
• Create a step using the table data.
• Implement the step
• Run the test with sub-set of table rows as parameters.
• The report highlights the rows that were not executed as green.

Gauge version

Gauge version: 0.4.0

Plugins

html-report (2.1.0)
java (0.3.4)

Add support for viewing nested concepts

open a failed spec by default

Currently when spec failures occurs no spec is opened in the report and you need to click on the spec name to open it up.
It should open up a failed specification by default.

Add overall start and end date/time to report

If someone looks at a Gauge report, it is not clear when the report was generated. Last night? A week ago? An hour ago? It's not clear.

I'd like to request that the start time of the run (i.e. the time when gauge specs was executed) and the end time of the run (i.e. the time when gauge specs is completed) be added to the report.

Left hand pointing arrow is still there.

Screenshot:

Steps to reproduce:

• Generate reports
• Click on the failed count (or skipped count)

ability to pass gauge_reports_dir as a parameter

This is currently blocked by getgauge/gauge#282

Give option for downloading test report as excel or other file format

it would be good if user can download test results into an excel or other file.

HTML report improvements

Hi there —

A couple of handful of things that stand out as the gauge report downloads from our CI server.

• Pie charts — they are really bad once you start adding more stuff to it, and humans are generally bad at understanding relative sizing of pies. Consider using a simple bar graph. More here and here

• There's a flash of html "template" that's visible as your report and it's data renders data.

  

• Also, it appears that the html report loads things from an external CDN, is not compressed or concatenated which causes irritably slow page loads on slow networks, by making too many http requests.

  <link rel="stylesheet" type="text/css"
        href="https://fonts.googleapis.com/css?family=Open+Sans:300,400,400italic,700,700italic">
  <link rel="stylesheet" type="text/css"
        href="https://maxcdn.bootstrapcdn.com/font-awesome/4.4.0/css/font-awesome.min.css">
  <link rel="stylesheet" type="text/css" href="css/normalize.css"/>
  <link rel="stylesheet" type="text/css" href="css/angular-hovercard.css"/>
  <link rel="stylesheet" type="text/css" href="css/style.css"/>
  

• Stack traces in a serif fonts make it hard to read, code SHOULD always preferably in a monospace font

• Screenshots cannot be dismissed on hitting the ESC key, which feels more natural than pressing x.

• The small press x to close at the bottom that's may not exactly be visible in all cases.

• Report names in sidebar should use css ellipsis to prevent being garbled (instead of trimmed to a fixed length). The advantage of using css ellipsis is that you can still copy the entire report name even though it does not show up.

  

Comments are not shown at the expected location.

Since upgrading from Gauge 0.2.? to 0.3.0, the HTML report is not showing comments that are in a spec in-line with the steps. Instead, all comments are being shown at the top of the scenario.

Example:

MySpec

tags: MyTags

My comments about the spec as a whole.

My Scenario

setup the environment

• Step 1
• Step 2
  
  check stuff
• Step 3
  
  clean up environment
• Step 4

Gauge 0.3.0, the HTML report shows

.....
setup the environment
check stuff
clean environment
Step 1
Step 2
Step 3
Step 4

In Gauge 0.2.?, the HTML report showed

....
setup the environment

• Step 1
• Step 2
  
  check stuff
• Step 3
  
  clean up environment
• Step 4

Not having the comments "in-line" in the reports has created confusion in understanding specs that have a large number of steps.

Current version
Gauge 0.3.0
html-report 2.0.0
java 0.3.4
xml-report 0.1.2

Unfortunately, I don't have the exact version numbers prior to upgrading, but the reports in the old format have comments in-line.

Add Timestamps to the HTML-Reports

On behalf of @AWDAS:

Timestamps should be added to the HTML-Reports, containing the following information:

When the step started; when the step ended; how long the step took
Total execution time (incl. when spec started, when spec ended)

Data table doesnt show all the rows when duplicated

When a cell value is repeated in a row, the data table skips that row and doesnt show it in report. But execution works fine.

Eg:

 |Product|Description       |
 |-------|------------------|
 |Gauge  |Gauge             |
 |Automation |Automation    |

Missing screenshot in chrome

Expected behavior

Expect to see a screenshot on failure in Chrome

Actual behavior

We are not seeing screenshots in Chrome. The screenshots appear fine in Firefox and IE

Steps to reproduce

reports.zip
1.Unzip and open the attached report in Chrome
2.Go to the failed step and scroll to the far right. No screenshot
3.Open same report in IE or Firefox scroll to the far right. Screenshot is there.

Gauge version

Gauge version: 0.4.0

Plugins

html-report (2.1.0)
java (0.3.4)

Chrome version: 49.0.2623.110
Firefox version: 43.0.1
IE version: 11.0.9600.18230

HTML report not created when reports are overwritten in windows.

Occurring due to 2 reasons:

• When reports are overwritten, the path for the HTML report contains forward slashes instead of backward slashes in windows.
• The ":" character is not allowed in a file name in windows.

Log text goes beyond expected area.

Clicking "Failed" icon shows passed specs if there are no failures

When there are 0 failures, click the failed tests icon after clicking the passing tests icon. This still shows the tests that are passing

index.html.zip

Show tags and environment executed against in report

table at spec level should appear clickable

For scenario result of each data table row, currently we need to click on each row. But its not easily implied that its clickable.

The default message about all tests green is missing

With the following spec, the html report generated does not show the details by default
html-report.zip

Add html-report plugin to gauge repository

Spec panel is resizing when spec is clicked

Expected behavior

Specification panel on the left should not be resized when user clicks on spec.

Actual behavior

Specification panel on the left is getting resized.

This is how the screen looks like when we open the reports.

The screen turns to this when we click on the Execution hooks failing specification in the left hand side panel.

Gauge version

Gauge version: 0.4.1

Plugins
-------
html-report (2.1.0)
java (0.3.4)
ruby (0.1.1)

reports-ruby.zip

Add a chart to display test summary

This chart should represent how many tests passed, failed etc.

Alignment of data table and first context step

Looks like the first cotext step is a row of the data table.

Specification names are not shown properly if the name exceeds certain limit

Search for spec tags does not list scenarios

Actual

In the html-report, searching by a tag lists specifications, when there is a match at spec level tag. However, the search result filters out all the scenarios, assuming that there are not scenario matches, thus lists a blank spec.

Expected

Search for a tag should list

(scenarios with matching tags) U (all scenarios from specs with matching tags)

tldr: spec tags apply to all scenarios that the spec contains and should be treated as such when searching.

Formatting issues on long exceptions

With HTML-Report v2.1.0, when there is a long exception the formatting is broken with different results across different browsers.

In Chrome v49 - the screenshot is not visible even if scrolled all the way to the right.
Screenshot from Chrome.

In IE v11, the left sidebar is not visible, but the screenshot is available if you scroll all the way to the right

In Firefox v42, the left sidebar is visible and the screenshot is available if you scroll all the way to the right. So Firefox technically is functioning but having to scroll to the right is cumbersome. I think inline scrolling like is done for the stacktrace would be better.

Gauge version: 0.3.0
Plugins
html-report (2.1.0)
java (0.3.4)
xml-report (0.1.2)

Report js: Failed step execution time is always zero

In the report js, which includes the execution time for each step, whenever a step fails, the execution time is always zero.

Green icon for failed test

With the spec, even though the errors are indicated, the html report has the green icon against the step descriptions
html-report.zip

Users should be able to start the game
======================================

User should be able to access the site
--------------------------------------
tags: join

User should be able to join an existing game
--------------------------------------------
tags: search

* Green icon for failed test

Failure screenshot gets displayed over logs in report

Nested concepts need to have visual distinction

Currently nested concepts seem to appear on the same indentation level, making it difficult to understand whether they are nested at all. There should proper UI/UX feedback to identify nested concepts in the reports.

Support for non-asterisk lists

Asterisk lists are used to define scenario steps, but sometimes there is a need to use lists for documentation purpose within scenario/specification. It would be great, if you can render non-asterisk lists (http://daringfireball.net/projects/markdown/syntax#list) properly.

Show failed scns on top against a failed spec

Currently, the order is the same as it was executed in. But in case of failure, it's useful to show this on top.

Template is shown instead of actual values til result.js loads

Add custom messages to reports added to ProtoExecutionResult

As a part of custom data in HTML report feature, the messages added to ProtoExecutionResult need to be added into the Report.

no contents display when open report with chrome on windows

chrome version: 45
env: windows 8.1

Markdown should be rendered in the report

Gauge specializes in reading specifications as markdown, it's very common for other markdown to be present in the specifications Gauge doesn't care about that is there for rendering to nice looking HTML easily.

I feel html-report should make an attempt to parse any of this extraneous markdown into HTML to preserve the pretty rendering in the report.

Pegdown is an open-source markdown-to-HTML library I am personally using and can vouch for that is incredibly simple to adapt into any code-base. Parsing timeouts, extensions, etc. all are supported and should be configurable in the report generation as well.

Filter by tag in reporting

From @harinee on November 18, 2015 11:46

There is currently no way to list the scenarios based on tags from my tests, in my report.
I would like to see how my tests are split based on the different tags that exist, across spec files.

Things that can help in reporting:

1. Count of scenarios across tags
2. Scenario heading under each tag
3. Ability to drill into the scenario to see the steps

Copied from original issue: getgauge/gauge#254

Data driven spec reports show all rows for the spec, not just the rows its executed for

Executing a spec for a specific row of the data table creates a report with all the rows from the data table with incorrect marking of failed row. If the spec is executed for n-th row and it fails, in the report all the rows are displayed and failure is marked on the first row of the data table.

Failed afterhook is displayed at the start of next scenario

If a scenario has a afterhook step and fails.. then the failed afterhook step is displayed at the start scenario's results page.

Scale screenshot, proportional to the monitor

@redcue implemented this via pull request: #24

Need to implement this in the new design.

It reports with an empty table

When create a step that takes a table with some empty cells, it reports with an empty table

|shopName |city |location |description |category |phoneNumber|
| | | | Any thing |Classic |0655744451 |

like this :

I tried to replace white spaces with some value like : None or Empty, but it stills give the same result.
apparently, it doesn't accept consequent equal values in the same row

Search doesn't show the spec in specification list, if the search string is in context step

Formatting is broken for custom messages of a step