sifive / riscv-vector-intrinsic-fuzzing

A random fuzz generator for the RISC-V vector extension intrinsics

RVV Intrinsic Fuzzing (RIF)

This repository aims to create fuzz testing for the RVV C intrinsic.

Usage

How To Use?

rif-test is the current driver to fuzz all existing intrinsics in include/CustomOperator.def.

./rif-test --help
usage: rif-test [-h] [--gen {random_gen,fused_gen}] [--sim SIM] [--cc CC] 
                [--mode {full,fast}] [--node NODE] [--arch ARCH] [--abi ABI]
                [--cflags CFLAGS] [--seed SEED] [--random {on,off}] [--has-policy]

optional arguments:
  -h, --help            show this help message and exit
  --gen {random_gen,fused_gen}
                        Random gen pattern
  --sim SIM             Path to simulator
  --cc CC               Path to compiler
  --mode {full,fast}    Testing mode
  --node NODE           Minimal # of node
  --arch ARCH           Target arch config, default is rv64gcv_zvfh
  --abi ABI             Target ABI config, default is lp64d
  --cflags CFLAGS       Default compilation flag
  --seed SEED           Fixed random seed
  --random {on,off}     Feed random seed
  --has-policy          Enable policy if set

RIF already have linked a pre-built simulator from metal-run. So the minimum requirement for testing it to specify a compiler that is able to compile RISC-V vector intrinsics. To specify your own simulator (QEMU), please add ${PATH_TO_QEMU_BINARY} ${CPU_CONFIGS} of how you wish to run ${PATH_TO_QEMU_BINARY} ${CPU_CONFIGS} ${PATH_TO_ELF} under --sim.

$ git clone https://github.com/sifive/rvv-intrinsic-fuzzing
$ mkdir test
$ cd test
$ ../rvv-intrinsic-fuzzing/rif-test --cc=<YOUR-COMPILER-PATH>
$ make

How To Build?

$ git clone https://github.com/sifive/rvv-intrinsic-fuzzing
$ cd rvv-intrinsic-fuzzing
$ mkdir build; cd build
$ cmake ..
$ make
$ ./tool/random_gen --help
Usage: random_gen [OPTION...]

  -c, --code=FILE            C code file, default filename: 'output.c'
  -d, --dot=FILE             Graphviz visualization file, default filename is
                             'output.dot'
  -h                         Give this help list
  -m, --march=ARCH_STRING    Arch string for testing, default is rv64gcv
  -n, --nodes-to-gen=NODES_TO_GEN
                             The number of nodes to generate for the graph.
  -r, --root=OPERATOR_ENUM   Initial node for the graph generation, default
                             root is 'AddVV32'
  -s, --seed=RANDOM_SEED     Seed for random number generator, default seed is
                             '0xdeadbeef'
  -v, --verbose              Produce verbose output
  -?, --help                 Give this help list
      --usage                Give a short usage message

Mandatory or optional arguments to long options are also mandatory or optional
for any corresponding short options.

riscv-vector-intrinsic-fuzzing's Issues

Add code gen category enum for operator

Currently under Operator.cpp code gen for different operations are done by a bunch of branching statements, and the branching statements are based on the OP_ID of the operations (e.g. _vv, _vx, or even specialized for a certain operator if (op->opType == "msbf_m")).

This is rather troublesome and bugs may hide inside type-o of these statements.

A better implementation would be have enums that represents categories of these code gen and appending the enun as a parameter inside the definition file of the operator.

Moreover, if the operator already contains codegen category enum, then the function generateOperatorCode can be template-ized.

template<enum CodeGenType>
void generateOperatorCode() { /* ... */ }

This issue can be solved by:

1. build a chart mapping operators to specific code gen pattern
2. adjust generation of definition file
3. rewrite code under Operator.cpp to utilize the appended information

Configurable lmul for FuseGraph

Currently for a FuseGraph we are setting the smallest lmul possible for each operator. Add parameter to configure it. Check the legality too.

Stable-ize the way to interact with riscv-isa-sim

Discussions have came up in multiple places ([1], [2]) that we should stable-ize how we should find a fixed place for linking to riscv-isa-sim, to both containing the refactoring from upstream and downstream SiFive custom instruction headers (under riscv/insns). This issue wants to track this so we don't forget this in the future.

[1] https://github.com/sifive/rvv-intrinsic-fuzzing/pull/93/files#r866516767
[2] https://github.com/sifive/rvv-intrinsic-fuzzing/pull/77

We can clone Spike only and not the whole federation repo using sparse checkout (Looks like a new feature since 2012)

https://stackoverflow.com/questions/600079/how-do-i-clone-a-subdirectory-only-of-a-git-repository
https://stackoverflow.com/questions/6238590/set-git-submodule-to-shallow-clone-sparse-checkout

Can't compile the generate testcase

I use the toolchain provided by GNU but can't compile the testcases. The reason may be the latest version is incompatible with the fuzzing tool. Could you provide the zip file of the environment please? I have tried to use the tar file of toolchain in your another git release but still can't work. I would appreciate it if you can help

Filter operator with provided arch string

There is a parameter [--arch ARCH] inside the test driver rif-test. The purpose of it is to test possible intrinsics provided by the architecture, which has not been implemented yet in RIF.

To restrict intrinsics based on provided architecture string, you can simply examine the value (sew, value type) that should be used by an operator.

Idea for introducing control flow dependency into RIF Graph

This is an idea from the discussion with @arcbbb.

The compiler may need to do some work on side effect instructions like vesetround. RIF mainly operates on a data flow graph, therefore we need to find a way to blend in control flow dependencies to test if the compiler recognizes interactions across basic blocks.

An immediate example is shown in the following snippet. If the if condition is true and both vaadd is executed, we would need an instruction to set the rounding mode before vaadd in line 3. If the else condition is executed, then we would need an instruction to set the rounding mode before vaadd in line 7. The compiler would have to recognize this sort of control dependency.

Update:
@kito-cheng gave a better idea, we can simply use volatile bool.

1: for (...) {
2:   if () {
3:     vaadd(..., rm_A);
4:   } else {
5:     /* clean, some code un-related to rounding mode */
6:   }
7:   vaadd(..., rm_A)
8: }

---

This gives us a new way of testing. RIF can generate some program that is guaranteed to receive certain input from runtime. With the guaranteed input, RIF can still pre-compute the expected results and the compiler will see value_0 as some runtime-determined variable and won't optimize out the control flow. The code may look like the following.

01: int main(int argc, char **argv) {
02:   bool value_0 = parseFromArg(argc, argv);
03:   assert(value_0 == true); // input that is runtime 
04:   for (...) {
05:     if (value_0) {
06:       vaadd(..., rm_A);
07:     } else {
08:       /* clean, some code un-related to rounding mode */
09:     }
10:     vaadd(..., rm_A)
11:   }
12: }

Use a namespace or enum class to protect OpAttr

As topic.

Generate array with distinct indexes for unordered index store

To guarantee deterministic execution for unordered index store, the indexing array has to contain distinct indexes.

Additionally, this also requires the simulator (e.g. QEMU) to also express the indeterminism for us to verify the result.

https://en.wikipedia.org/wiki/Random_permutation