GitHub Step Summary about compliance-tests

[LarsAsplund]

I was thinking about a JSON+XUnit export and have a VUnit attribute on tests referencing relevant LRM section. The output would be a table with test case name + pass/fail + LRM link. Now that we have two simulators that we can use online we should have a column for each. Just like in that SystemVerilog example. We should also generate a HTML report for those running offline with their own simulator. Pandoc on the Markdown maybe. My current client uses GitHub and there I use this technique to create a test to spec mapping which the CI pushes to the GitHub Wiki. I didn't know about this summary feature so let's give it a try. Will you? I better focus on the VUnit support for NVC.

[umarcor]

We are talking about different use cases. The GitHub Step Summary is useful for someone watching a CI run through the web. So clicking on any of the runs listed in https://github.com/VHDL/Compliance-Tests/actions.

Generating JSON, XUnit or HTML cannot be done there. That needs an additional step, which might be a) downloading an artifact and/or b) pushing it somewhere (gh-pages, wiki, anywhere else).

In ghdl/extended-tests, we use pytest pytest-reporter pytest-reporter-html1 and we upload the HTML report to gh-pages (see ghdl.github.io/extended-tests).
Moreover, in GHDL we pregenerate the content of a subdir and then include it in the doc: https://github.com/ghdl/ghdl/blob/master/doc/conf.py#L114-L115. Hence, we might convert VUnit's XUnit output to HTML and include it in the documentation.