Compliance tests dependent on vunit about compliance-tests

[bpadalino]

I tend to agree, but I think @LarsAsplund said there was a way around the vunit requirement using a dummy library that would "act" like vunit but was just a placeholder.

I believe it is #39 and I am unsure why it hasn't been pulled in yet.

@sckoarn would #39 help you out to alleviate the vunit requirement?

[umarcor]

@sckoarn since VHDL does not have built-in or canonical build and test execution management features (for context, see https://gitlab.com/IEEE-P1076/VHDL-Issues/-/issues/13), additional code is required. With VHDL 2019, a solution might be coded using VHDL only, but such solution does not exist as far as I am aware. All the projects I know do use some other language (shell/bash, powershell, tcl, python...) or they use some GUI tool which hides the implementation.

Therefore, "not depending on VUnit" is an invalid request because it's incomplete. An alternative needs to be provided. Do you suggest to use some other third-party project/codebase or are you thinking about reimplementing those features from scratch using custom code tailored to this repo? Please explain the alternative and the benefits it would provide.

EDIT

Note that the main feature we need is: "execute this or that subset of testbenches using this or that tool". We need it to work on multiple systems (at least GNU/Linux and Windows), and on multiple architectures (at least x86, arm, ppc, s390x and riscv). We also need to support as many tools as possible; not only simulators.

[bpadalino]

@umarcor I think @sckoarn is saying that the requirement for any type of "framework" should be optional for a Compliance Test suite. I think he wants something more like the vests that were developed for VHDL-93, as you're well aware, which do not require any external frameworks for the VHDL itself.

Using vunit as a runner outside of all VHDL would probably be acceptable for @sckoarn, but the fact it's inside the VHDL itself is the problem for him. I, too, believe it sullies the tests in general.

Does that make sense?

[umarcor]

@bpadalino I understand the theory, but I find it to fall short in practice. What solution would replace the functionality that VUnit provides?

VUnit requires 5 lines of VHDL code per testbench. If we remove those, the Python functionality of VUnit (support for multiple tools, filtering tests, getting config options from either envvars or CLI args, etc.) cannot be used. So, what are we going to use as an alternative? Until we can answer that question, I shall deny the argument that 5 lines make the tests full of over head.

We might argue about removing all other VUnit features (info/error, checks, etc.). I would be open to that, if there was an alternative to VUnit for test build and management. However, since we need VUnit for now, those feature come for free.

So, let's focus on the vests as a reference. How would you run those in CI with GHDL and NVC, without using VUnit?

[LarsAsplund]

See #39

[bpadalino]

VUnit requires 5 lines of VHDL code per testbench. If we remove those, the Python functionality of VUnit (support for multiple tools, filtering tests, getting config options from either envvars or CLI args, etc.) cannot be used. So, what are we going to use as an alternative? Until we can answer that question, I shall deny the argument that 5 lines make the tests full of over head.

It isn't about 5 lines of VHDL, but to be fair it's more than 5 lines of VHDL. The fact that #39 exists means it's more than just 5 lines of trivial code. Just look at #39 and the size of the pull request for the mock stuff.

We might argue about removing all other VUnit features (info/error, checks, etc.). I would be open to that, if there was an alternative to VUnit for test build and management. However, since we need VUnit for now, those feature come for free.

You don't need vunit to be part of the tests themselves. It's convenient, but you don't need it.

So, let's focus on the vests as a reference. How would you run those in CI with GHDL and NVC, without using VUnit?

Think of this like separation of church and state. Church is vunit and state is VHDL-2019 Compliance Test. Let's take a random test - tb_032.vhd. Due to the way the tests were originally written, they didn't care about vunit so the entire vunit design unit is added at the end.

In this case, a simple thing to do is to separate the vunit aspect from the test itself. If we separated it into vunit_032.vhd which held the contents below line 44, and kept tb_032.vhd as strict standards-only VHDL with no extra libraries, wouldn't that solve the situation? Folks who don't want to or can't use vunit aren't forced to do so.

vunit being an optional convenience versus a necessity seems doable. Yes?

[sckoarn]

Ok, I am sorry I have not been replying. To me this is very simple. The test set should only contain feature test code. As in VHDL code that tests some feature. No other code is needed. No environment to run it, no methods of simulation should be part of this repository. The compliance tests should only contain tests in VHDL that present code that will demonstrate the target feature. The compliance test set should contain "pure" VHDL, nothing else.

It is completely up to the user how they will use this. If you want to use vunit, then by all means go ahead. If I want to use some bash script to run my regression set, I am free to do so, and will not have the encumbrance of vunit, or have to deal with some ones fav language like Python.

It is not the job of a compliance test set to provide or enforce some testing environment.

Also, who is the target of this test set? Do we feel like the "Big 3" will jump onto this and test against their product? I will guess that they may not, and may not like the fact that there is a dependency on vunit. Many of the tool providers have their own compliance tests. I would expect that the Big 3 may have a look, then take what they find to be of use and integrate that into their own testing system.

I have looked over the tests. When I go to use them, I will be writing a script to remove all vunit stuff and make each test stand alone. There is for sure useful code and one way or another I will use this output.

[sckoarn]

I also had a look at the #39 and all though it might work, IMO it is still huge overhead. It also still requires the use of a specific package. Regardless of how much it is dummied down, it is still a system which some may not want to use.

I have been doing verification since 1995. I have seen many ways of doing simulation and testing. I have my way of doing things and do not like to be forced down a particular path, specially when we are talking about adding tests to my regression set. My way is of course the best way.

[LarsAsplund]

The reason the mock is large is that more features then the minimal 5 lines are used. If only 5 lines are used the mock would be 50 lines of code or so. Basically just subprogram declaration "overhead".

The reason more than 5 lines are used is that this project started out in a separate repository written by me and Philippe Faes (one of the founders of Sigasi) targeting compliance for VHDL-2008 which was the focus many years ago. We were biased towards VUnit. However, I claim no ownership of this code and suggest the following:

1. Approve #39
2. Reduce all testbenches to 5 lines of VUnit
3. Reduce the mock package accordingly
4. Decide if VUnit should be removed entirely or not

To be honest I didn't have any ambitions with this project other than showing a path for how it can be done. This means that I won't be a driving developer or a maintainer for it. Those that eventually become the core of the project have to decide on the path forward. These are the people in charge of maintaining the project and making sure that the test suite can be executed in all simulators. That can be a significant commitment and it is reasonable that they can have it their way. I will not interfere in this process and try to have it my way. I think I have provided the options to the best of my knowledge and the rest will be for others to decide.

[sckoarn]

I have a test set which is a code coverage suite for 93 standards files. These tests are very simple and are a derivative of the work done by Bishop. I would like to add those tests to this repository. Once done, they could be reviewed and a possible direction for other tests. Another way to compare against and evaluate for improving the path forward. I am all for a good reference set for proving correctness.

I believe I have some time allotment to assist with this effort. Creating a test set to prove a language implementation is not a small undertaking and will be a continuous effort.

[nickg]

For older standards we have the "Billowitch" tests from VESTs. The old CVS repository is down but I have a mirror here and I believe there's a copy in the GHDL source tree.

There is also "vi_suite", another old test suite which is unfortunately proprietary although the source can be found on the internet (I think I got my copy from @diogratia). I believe Accellera is the current copyright holder so if someone has a contact there they might be able to release it under a more permissive license.

Both of those could probably be adapted to run under VUnit with some effort, although given the number of tests it would have to be partially automated.

Re. the use of VUnit itself... it is possible to treat it as a set of VHDL packages and execute the test benches directly without the Python harness.

[sckoarn]

What I am saying is that a test suite should not contain code which dictates how the code will be used. If vunit functions and procedures are part of the test code then the test code is dependent on vunit. IMO this is just wrong.

The Billiwitch and Ashenden tests are good, but they did not cover all that was intended. I updated and finished those tests so that the ieee standards they test, fully cover those standards. I uncovered a couple "bugs" in the standard files.

The tests I have are not dependent on any method of running them. It is up to the user to make the regression happen and collect the data to show the coverage. This is tool dependent. Since the tests I created / modified are part of our dsim regression I do not care what might happen to them when passed over to the standards group.

I also believe in simplicity where ever possible, vunit is not simple. The whole ieee regression set I use is run from a simple shell script.

The tests in this repo do not contain the additions I have made. Therefore this repo is incomplete regardless of what environment you choose to run it under. The differences between my updated tests and the original is significant. So, if it is intended that the current repo be continued, I suggest someone finish the testing and complete the test set.

[nickg]

The Billiwitch and Ashenden tests are good, but they did not cover all that was intended. I updated and finished those tests so that the ieee standards they test, fully cover those standards. I uncovered a couple "bugs" in the standard files.

Regardless of whether they end up in this repository or not, I think the community would be very grateful if more tests could be shared under a similar license to the original Billowitch tests.

[sckoarn]

Also, just so it is known, I have been doing RTL verification since 1995. I have seen and used many methods of verification from free python based systems, VHDL and SV. I have not used vunit, but have published my own version of a VHDL testing system, in the VHDL testbench package, 2007.

[sckoarn]

I also have been granted some time to work for the VHDL group to implement / share the IEEE coverage we did. Nothing attached, just a contribution back to the community, to be licensed as the group sees fit.
But we have to get over this issue. I am not willing to mod my additions to make it run under vunit. In fact I started from a zip file sent to me by some one which was working on the standards testing. So all my work is derived from an original copy of the work done in 2008. The test cases I looked over match that which was given to me but have been modified to run under vunit.

Making it so that my test set will run generically under several simulators will be very easy in it's current state. The tests are fully driven from shell scripts. Those scripts would just have to be modified / copied for the specific simulator. Then someone could test them that has access. I am restricted to dsim and free simulation like GHDL.

[nickg]

Do you mean the FPHDL tests or something else? I'm aware of a set of tests from David Bishop here: https://github.com/FPHDL/fphdl/blob/master/vhdl2008test.zip

[sckoarn]

Yes, I got a direct copy of what David had. I am unsure where it originated from. But he stated that he had to dig it up.

Correction, those look like the tests I started from.