riscv-software-src / riscv-ctg Goto Github PK

Summary

The csr coverpoints (csr_comb node defined here) in the CGF correspond to the values present in the CSR. These kinds of tests would try to set the CSR to a particular value either by writing to them or by executing code which causes the CSR to change due to a side effect, interrupts and exceptions. The tests will have to propagate the CSR values to the signature. CTG does not support generating tests for these coverpoints in its current state. Each csr(or csr type) will have to be handled independently.

Changes required

• Add support to test generator.
• Add relevant assembly macros necessary to handle most common cases.

Basic example

A coverpoint corresponding to the C bit in misa register:

c_misa:
  opcode:
    c.add: 0
  csr_comb:
    'misa & 0x4 == 0': 0

Assembly Test

la x4, signature_addr
addi x1,x1,0
csrr x2,misa
li x3,~0x4
and x2,x2,x3
csrw misa,x2
c.add x1,x3
c.nop
nop
sw x1, 0(x4)

Dear All,
I have been trying to add csr instruction support in riscv-ctg by following these instructions. I have made the following updates in riscv-ctg:

1. Added csr registers in cgf.yaml datasets to add a few csr and pmp registers.
2. Added a new instruction node for csr instruction "csrw" in cgf.yaml according to the following template
3. Update template.yaml to add csr registers in metadata and added csrw node)
4. Added a new function in arch_test.h) to add support for csrw instruction.
5. Updated rv32i_priv.cgf to create nodes for new tests to verify the csrw.

Although I have followed all instructions given in RISC-V ISA Coverage Documentation but my riscof command is still not working. After running the command

$ riscof coverage --config=config.ini --cgf-file $PATH_TO_RISCV_CTG/sample_cgfs/dataset.cgf --cgf-file PATH_TO_RISCV_CTG/sample_cgfs/rv32i_priv.cgf --suite=PATH_TO_RISCV_ARCH_TESTS/riscv-test-suite/rv32i_m/privilege/ --env=PATH_TO_RISCV_ARCH_TESTS/riscv-test-suite/env/

My terminal gets stuck at line "RUNNING TESTS ON REFERENCE" without giving any error.
The tests are correct and have been verified by riscof with dummy nodes in rv32i_priv.cgf. Can someone please help in rectifying any mistake here?

Hi

I followed the instructions in the doc to generate test cases, but I encountered the following problem(*1).

cmd: riscv_ctg -v debug -d ./tests/ -r -cf ./sample_cgfs/dataset.cgf -cf ./sample_cgfs/rv32i.cgf -bi rv32i -p2

Version Information:
os:Linux geralt 5.15.0-69-generic #76-Ubuntu SMP Fri Mar 17 17:19:29 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Python Package:
riscv-ctg-0.11.0 (installed via 'pip install riscv-ctg')
ruamel.yaml: ruamel.yaml 0.17.32

(*1) Error message:
INFO | ****** RISC-V Compliance Test Generator 0.11.0 *******
INFO | Copyright (c) 2020, InCore Semiconductors Pvt. Ltd.
INFO | All Rights Reserved.
INFO | Copying env folder to Output directory.
Traceback (most recent call last):
File "/home/msyu/workspace/venv/bin/riscv_ctg", line 8, in
sys.exit(cli())
File "/home/msyu/workspace/venv/lib/python3.8/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/home/msyu/workspace/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/home/msyu/workspace/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/msyu/workspace/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/home/msyu/workspace/project_rvv_testsuite/riscv-ctg/riscv_ctg/main.py", line 29, in cli
ctg(verbose, out_dir, randomize ,xlen, int(flen), cgf,procs,base_isa,inst)
File "/home/msyu/workspace/project_rvv_testsuite/riscv-ctg/riscv_ctg/ctg.py", line 126, in ctg
op_template = utils.load_yamls(const.template_files)
File "/home/msyu/workspace/project_rvv_testsuite/riscv-ctg/riscv_ctg/utils.py", line 65, in load_yamls
return dict(yaml.load(fp))
File "/home/msyu/workspace/venv/lib/python3.8/site-packages/ruamel/yaml/main.py", line 426, in load
return constructor.get_single_data()
File "/home/msyu/workspace/venv/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 111, in get_single_data
node = self.composer.get_single_node()
File "/home/msyu/workspace/venv/lib/python3.8/site-packages/ruamel/yaml/composer.py", line 70, in get_single_node
document = self.compose_document()
File "/home/msyu/workspace/venv/lib/python3.8/site-packages/ruamel/yaml/composer.py", line 92, in compose_document
node = self.compose_node(None, None)
File "/home/msyu/workspace/venv/lib/python3.8/site-packages/ruamel/yaml/composer.py", line 128, in compose_node
node = self.compose_mapping_node(anchor)
File "/home/msyu/workspace/venv/lib/python3.8/site-packages/ruamel/yaml/composer.py", line 209, in compose_mapping_node
item_value = self.compose_node(node, item_key)
File "/home/msyu/workspace/venv/lib/python3.8/site-packages/ruamel/yaml/composer.py", line 128, in compose_node
node = self.compose_mapping_node(anchor)
File "/home/msyu/workspace/venv/lib/python3.8/site-packages/ruamel/yaml/composer.py", line 209, in compose_mapping_node
item_value = self.compose_node(node, item_key)
File "/home/msyu/workspace/venv/lib/python3.8/site-packages/ruamel/yaml/composer.py", line 104, in compose_node
if self.parser.check_event(AliasEvent):
File "/home/msyu/workspace/venv/lib/python3.8/site-packages/ruamel/yaml/parser.py", line 139, in check_event
self.current_event = self.state()
File "/home/msyu/workspace/venv/lib/python3.8/site-packages/ruamel/yaml/parser.py", line 617, in parse_block_mapping_value
if self.scanner.check_token(ValueToken):
File "/home/msyu/workspace/venv/lib/python3.8/site-packages/ruamel/yaml/scanner.py", line 1728, in check_token
self._gather_comments()
File "/home/msyu/workspace/venv/lib/python3.8/site-packages/ruamel/yaml/scanner.py", line 1768, in _gather_comments
self.fetch_more_tokens()
File "/home/msyu/workspace/venv/lib/python3.8/site-packages/ruamel/yaml/scanner.py", line 292, in fetch_more_tokens
return self.fetch_literal()
File "/home/msyu/workspace/venv/lib/python3.8/site-packages/ruamel/yaml/scanner.py", line 675, in fetch_literal
self.fetch_block_scalar(style='|')
File "/home/msyu/workspace/venv/lib/python3.8/site-packages/ruamel/yaml/scanner.py", line 686, in fetch_block_scalar
self.tokens.append(self.scan_block_scalar(style))
File "/home/msyu/workspace/venv/lib/python3.8/site-packages/ruamel/yaml/scanner.py", line 1930, in scan_block_scalar
return Scanner.scan_block_scalar(self, style, rt=rt)
File "/home/msyu/workspace/venv/lib/python3.8/site-packages/ruamel/yaml/scanner.py", line 1120, in scan_block_scalar
breaks, max_indent, end_mark = self.scan_block_scalar_indentation()
File "/home/msyu/workspace/venv/lib/python3.8/site-packages/ruamel/yaml/scanner.py", line 1309, in scan_block_scalar_indentation
raise ScannerError(
ruamel.yaml.scanner.ScannerError: more indented follow up line than first in a block scalar
in "", line 1484, column 5

To support multiple ISA strings per instruction, the isa field in each node of template.yaml and the generation of the RVTEST_ISA macro in the tests need to be modified.

Summary

Coverpoints spanning across multiple instructions help identify interesting instruction sequences which have architectural significance such as structural hazards and data hazards. CTG lacks the infrastructure necessary to generate such tests. This involves defining the relevant coverpoints using the custom format( cross_comb node defined here) for specifying such coverpoints and employing a constraint solver to produce tests with the relevant instruction sequences.

Changes required

• Add support to test generator.
• Add relevant assembly macros necessary to handle most common cases.

Basic example

WAW hazard between an add and a sub instructions which are 3 instructions apart. The instructions in-between are consuming i.e use the result of the add operation as one of the operands.

Coverpoint Definition

add_sub_cwaw:
  cross_comb:
    - "[add : ? : ? : ? : sub] :: [a=rd : ? : ? : ? : ?] :: [? : rs1==a or rs2==a : rs1==a or rs2==a : rs1==a or rs2==a : rd==a]"

Possible assembly sequence

li x3, 152;
li x4, 169;
add x3,x3,x4;
addi x5,x3,1;
mulh x6,x4,x3;
slli x4,x3,25;
sub x3,x5,x6;

The sample cgf file for RV32E tests contains the following ISA checks
check ISA:=regex(.*I.*)
which needs to be changed to following
check ISA:=regex(.*E.*)
because the cgf file is for E base extension.

Also the generation.py and contants.py file need update for the generation of RV32E tests.

For forward jumps imm/2 - 3 number of nop instructions are inserted here , that is due to the 2 instructions between jal and 3f. When the target address has to be 2 byte aligned, 2 additional nop instructions after jal are present (due to the trap handler return constraint), which means the number of extra nop needed is reduced by two. But that is not accounted for in the adjustment.

This does not manifest as a bug currently, because the coverpoint for a target of 2 byte alignment, does not depend on the immediate value. But if the coverpoint is updated to something where the immediate value is also tested with the alignment of the effective addres, then this macro will fail.

The variable 'xlenlim' referenced in the abstract combination of a few bitmanip instructions within the configuration files rv32i_b.cgf, rv32e_b.cgf, and rv64i_b.cgf appears to be "not defined", leading to an error. One instance of this error is in the 'bclr' instruction, where the exact error message is:
ERROR | Error evaluating abstract comb: xlenlim("rs1_val", xlen) in bclr: name 'xlenlim' is not defined

Hello.
I don't see any Vector configuration file in the sample, and I don't see anything about vector instruction format or register in the CTG source code. Does CTG currently have any plans to support the expansion?
Looking forward to reply, thanks.

The following instruction attributes are not documented (see https://riscv-ctg.readthedocs.io/en/latest/overview.html#attributes):

• std_op
• sig and its subfields stride and sz

Further, the example has the following template:

  template: |

    // $comment
    // opcode: $inst ; op1:$rs1; op2:$rs2; dest:$rd; op1val:$rs1_val;  op2val:$rs2_val
    TEST_RR_OP($inst, $rd, $rs1, $rs2, $correctval, $rs1_val, $rs2_val, $swreg, $offset, $testreg)

However, a |- should be there instead of the |.

Additionally, the order of attributes does not match the order used in most templates.

Additional coverpoints/scenarios for FP extensions:

• In RISC-V, the same register is used for storing both the single and double precision numbers. The single precision numbers are NaN boxed when stored in a 64 bit wide register. In such a configuration, if the source to a sp operation is incorrectly NaN boxed, the input should be a canonical NaN. This can be tested by a fld -> sp op -> fsw sequence. Capturing coverage for this requires further thought.
• Certain operations propagate the NaN payloads, but the arithmetic operations canonicalize. In case a NaN boxed sp value is stored using a dp store operation, the nan payload should be identical. Similar argument can also be made for a flw -> fsjn* -> fsd sequence.

Both the aforementioned scenarios are also applicable for the mv operations to/from the integer register file.

    This is not true in all cases. When `xlen<flen`, the registers are paired and use of odd numbered registers is reserved at this point. It would probably be best to modify this to ensure that the same is captured.

Originally posted by @pawks in #49 (comment)

In the attached clmul-01.S file,

inst_1:
// rs1 == rs2 == rd, rs1==x29, rs2==x29, rd==x29, rs2_val == 9223372036854775807, rs1_val == 18446744073709550591
// opcode: clmul ; op1:x29; op2:x29; dest:x29; op1val:0xfffffffffffffbff; op2val:0xfffffffffffffbff
TEST_PKRR_OP(clmul, x29, x29, x29, 0x0000000000000000, 0xfffffffffffffbff, 0xfffffffffffffbff, x29, x1, 8, x2)

rs2_val and its hex equivalent in op2val are not same

Similarly in
inst_3:
// rs1 == rs2 != rd, rs1==x27, rs2==x27, rd==x30, rs2_val == 16140901064495857663,
// opcode: clmul ; op1:x27; op2:x27; dest:x30; op1val:0xfffffffffffffbff; op2val:0xfffffffffffffbff
TEST_PKRR_OP(clmul, x30, x27, x27, 0x0000000000000000, 0xfffffffffffffbff, 0xfffffffffffffbff, x27, x1, 24, x2)
rs2_val and its hex equivalent in op2val are not same

###########
riscv_ctg --version
RISC-V Compliance Test Generator, version 0.6.3

python3 --version
Python 3.6.9
clmul.cgf.txt
clmul-01.S.txt

Hello,
I didn't see vector ISA in sample_cgfs. What should I do if I want to test vector ISA in this test architecture?
Looking forward to answering, thanks！

The command to generate floating point assembly tests in the fadd.d_b1-01.S filehttps://github.com/riscv-non-isa/riscv-arch-test/blob/main/riscv-test-suite/rv32i_m/D/src/fadd.d_b1-01.S does not work when invoked in the riscv-ctg repository via the corresponding cgf file https://github.com/riscv-software-src/riscv-ctg/blob/master/sample_cgfs/sample_cgfs_fext/RV32D/fadd.d.cgf.

The command in the comment is missing some arguments(--base-isa) which had to be passed along with --flen instead of --xlen.
Even after passing the arguments as requested by riscv-ctg, the following error pops up :

When I use 'riscv_ctg -v debug -d ./tests/ -r -cf ./sample_cgfs/dataset.cgf -cf ./sample_cgfs/rv64i.cgf -bi rv64i -p2,' I can generate tests for RV64I. However, I couldn't find the provided rv64d.cgf and rv64f.cgf. Now I want to generate tests for D and F extensions. How can I do that? Looking forward to your guidance. Thank you very much.

I am trying to generate RV32I_K tests using riscv-ctg but tests are not generated with this error:

File "/home/eman/riscof/riscv-ctg/riscv_ctg/generator.py", line 800, in eval_inst_coverage
var_dict[key] = int(instr[key])
ValueError: invalid literal for int() with base 10: ' 0x03020100'

Looking forward for a solution, Thanks!

I am using the following command to generate floating point tests;
riscv_ctg -v debug -d ./tests/32/F/ -r -cf riscof/riscv-ctg/sample_cgfs/dataset.cgf -cf riscof/riscv-ctg/sample_cgfs/sample_cgfs_fext/RV32F/fsw-align.cgf -bi rv32i -p2
But no tests are generated with these warnings;
WARNING | mnemonics node not found in covergroup: datasets
WARNING | Skipping fsw since its not supported in current FLEN(0):
Its same for all floating point tests.
I think am using wrong covergroup or Is there any problem with the command or something else please elaborate?
riscv-ctg version: 0.10.0

Hello,
when I use riscv-ctg to generate tests, there is a problem that fld,fsd,fsw,flw cannot generate tests. The specific error log is 'TEST_ STORE_ F 'and' TEST_ LOAD_F' Missing a parameter .
for example:
error：macro "TEST_STORE_F" pass 12 arguments, but takes just 11.
What additional parameters are required for this error, thanks!
And, when I try to generate RV64i_ B During the test, bclr,bclri,bext,binv,binvi,bset,bseti will report an error: xlenlim is not defined.
clz, clzw, cpop,cpopw....will report an error: object of type 'int' has no len()
Do you have any of these problems？Looking forward to answering, thanks！

The use of key 'label' here is not allowed, as the the key is not yet assigned. With the usual understanding the key 'label' refers to the cover group name, where there isn't any in the name of '1b' as of i'm aware. Hence unsure of this line, instr['label'] == '1b' what is exactly verifying.

The current key and sample values of a dictionary

'inst': 'c.beqz'
'comment': 'rs1==x13,rs1_val < 0 and imm_val < 0, rs1_val == -2'
'index': '0'
'rs1': 'x13'
'rs1_val': '-2'
'imm_val': '-12'

Trying to run rv32ic, to ensure the present code supports the existing compressed instructions.

• The checks to detect whether an instruction belongs to the F/D extension is currently based on whether the first letter of the instruction is f. Instead it should be based on the isa node in the template node for that instruction. Similar to the checks for P extension.
• The check strings for the D extension cgfs only check for the F extension in the isa. They should be modified to check for D extension. Otherwise the tests will be selected for configurations where they aren't applicable in riscof.