luchobazz / quicktest

CLI for stress testing in competitive programming contest

Quick Test CLI

Command Line Interface (CLI) for Stress Testing for Competitive Programming

📖 Docs

Demo

Documentation

Table of Contents

• Getting Started
  • Installation
• Introduction
• Commands
• How to Contribute
• Bug reports
• Supported Languages
• Compilation and Execution Commands
• License
• Credits

Getting Started

Installation

Installation tutorial for Linux, Windows and macOS can be found here on the docs website

Introduction

Quick Test CLI is a project to perform stress testing in competitive programming contests in an easy and fast way, focusing only on the contest.

Currently, Quick Test CLI supports three types of tests which are listed below:

quicktest cmp	quicktest stress	quicktest check

• quicktest cmp | qt cmp: It checks the correctness of the algorithm we want to verify by comparing it with a brute force solution which is usually very slow, but is 100% sure to provide the correct solution.

  • Sample:

    quicktest cmp --target-file=main.cpp --correct-file=correct.cpp --gen-file=gen.cpp
    # Or shorter:
    qt cmp -t main.cpp -c correct.cpp -g gen.cpp --tout 1000 --tc 1000

• quicktest stress | qt stress: Verify that the code execution time does not exceed what is allowed, using a random generator for multiple test cases.

  Note: In this scenario there is no slower solution with which to compare the correction.

  • Sample:

    quicktest stress --target-file=main.cpp --gen-file=gen.cpp
    # Or shorter:
    qt stress -t main.cpp -g gen.cpp --tout 1000 --tc 1000

• quicktest check | qt check: In some problems more than one answer is accepted, so the quicktest cmp command would not work correctly, in this case a script checker is used to verify the correctness of the algorithm.

  • Sample:

    quicktest check --target-file=main.cpp --checker-file=correct.cpp --gen-file=gen.cpp
    # Or shorter:
    qt check -t main.cpp -c check.cpp -g gen.cpp --tout 1000 --tc 1000

• quicktest output | qt output: run all test cases that match a prefix and save the response to an output file

  • Sample:

    quicktest output --target-file=main.cpp --prefix=testcase_ac
    # Or shorter:
    qt output -t main.cpp -p test_cases/testcase_ac --tout 1000

Commands

Note: you can use the long command quicktest or the alias qt

• quicktest cmp | qt cmp

  Required Options

  • -t=<value> | --target-file=<value>
  • -c=<value> | --correct-file=<value>
  • -g=<value> | --gen-file=<value>

  Other Options

  • --test-cases=<value> | --tc=<value> [default: 1000]
  • --timeout=<value> | --tout=<value> [default: 2000] Unit of time: ms
  • --memory-limit=<value> | --ml=<value> [default: 1000000000 - 1GB] Unit of time: bytes
  • --prefix=<value> | -p=<value> conflict with --gen-file (Only one can be used at a time)
  • --diff Show differences between the expected file and the output file

---

• quicktest stress | qt stress

  Required Options

  • -t=<value> | --target-file=<value>
  • -g=<value> | --gen-file=<value>

  Other Options

  • --test-cases=<value> | --tc=<value> [default: 1000]
  • --timeout=<value> | --tout=<value> [default: 2000] Unit of time: ms
  • --memory-limit=<value> | --ml=<value> [default: 1000000000 - 1GB] Unit of time: bytes
  • --prefix=<value> | -p=<value> conflict with --gen-file (Only one can be used at a time)

---

• quicktest check | qt check

  Required Options

  • -t=<value> | --target-file=<value>
  • -c=<value> | --checker-file=<value>
  • -g=<value> | --gen-file=<value>

  Other Options

  • --test-cases=<value> | --tc=<value> [default: 1000]
  • --timeout=<value> | --tout=<value> [default: 2000] Unit of time: ms
  • --memory-limit=<value> | --ml=<value> [default: 1000000000 - 1GB] Unit of time: bytes
  • --prefix=<value> | -p=<value> conflict with --gen-file (Only one can be used at a time)

---

• Flags of the cmp, stress and check subcommands

  • --break-bad | --break Break if WA, TLE or RTE states occurs
  • --run-ac Run test cases Accepted
  • --run-all Run all test cases
  • --run-rte Run test cases Run Time Error
  • --run-tle Run test cases Time Limited Exceeded
  • --run-wa Run test cases Wrong Answer
  • --save-all Save all test cases
  • --save-bad Save only bad cases with WA, TLE or RTE states

---

• quicktest output | qt output

  Required Options

  • -t=<value> | --target-file=<value>
  • -p=<value> | --prefix=<value>

  Other Options

  • --timeout=<value> | --tout=<value> [default: 2000] Unit of time: ms
  • --memory-limit=<value> | --ml=<value> [default: 1000000000 - 1GB] Unit of time: bytes
  • --break-bad | --break Break if WA, TLE or RTE states occurs
  • --save-out Save the output of the target file for each test case

---

• quicktest setup | qt setup

  Subcommand

  • config Subcommand that allows to change C++ settings

    Options

    • --label=<value> Label with the path of the configuration that you want to change
    • --value=<value> value you want to change a selected label to

---

• quicktest example | qt example

  Flags

  • --check Show examples of the check subcommand
  • --cmp Show examples of the cmp subcommand
  • --stress Show examples of the stress subcommand
  • --output Show examples of the output subcommand
  • --setup Show examples of the setup subcommand

  Nota: can only use one flag at a time

How to Contribute

If you are interested in contributing to the Quick Test CLI project, please take a look at the Contribute guide

Bug reports

You can report any bugs here.

Supported Languages

Language
C++
Java
Python
Rust Lang
Go Lang
GNU C
Kotlin

Compilation and Execution Commands

Language	Compile / Interpreter	Execution Command
C++17	g++ -std=c++17 -Wall -DONLINE_JUDGE=1 -o .qt/main main.cpp	./.qt/main
Java	javac -d .qt/ Main.java	java -cp .qt/ Main
Python3		python3 main.py
Rust Lang	cp main.rs ~/.quicktest/rust/src/main.rs && cargo build --release --quiet --manifest-path ~/.quicktest/rust/Cargo.toml && cp ~/.quicktest/rust/target/release/rust .qt/main	./.qt/main
Go Lang	cp main.go ~/.quicktest/go_mod/main.go && go build -buildmode=exe -o ./.qt/main ~/.quicktest/go_mod/main.go	./.qt/main
GNU C	gcc -std=gnu11 -lm main.c -o .qt/main	./.qt/main
Kotlin	kotlinc main.kt -include-runtime -d .qt/main.jar	java -jar .qt/main.jar

License

Licensed under either of these:

• MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT)

---

Credits

_{Logo image based on the one made by Freepik for www.flaticon.com}

_{Documentation was based on cpbooster}

_{Installation scripts were based on deno_install}

quicktest's Issues

[housekeeping] remove extension enum

If it is not used anywhere, remove
Lines: https://github.com/LuisMBaezCo/quicktest/blob/main/src/runner/types.rs#L64-L70

Resolve Warnings generated by `cargo clippy`

See Cargo Clippy

• cargo clippy
• cargo clippy -- -D warnings
• cargo clippy --all-targets --all-features -- -D warnings
• cargo test

issue: test c++ build with -02 and -Wall

g++ -std=c++17 -02 main.cpp -o .qtest/main.o -Wall

Test RTE

enhancement: Proposed change of the command `quicktest tle` to `quicktest stress`

issue: No such file or directory

quicktest --version

[email protected]

thread 'main' panicked at 'Unable to open file: Os { code: 2, kind: NotFound, message: "No such file or directory" }', /home/sorkierkegaard/.cargo/registry/src/github.com-1ecc6299db9ec823/quicktest-0.9.0/src/config/load_config.rs:39:71
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

How can I test for multiple inputs

How stress test if there multiple inputs like graph of n nodes a array permutation of n nodes and Binary string of of length n

2.How can I see failed test case when I use qt cmp ?

Improve performance by adding async

Tasks

• Add object-oriented structure + async to the output command (v1.0.2)
• Add object-oriented structure + async to the setup command (v1.0.3)
• Add object-oriented structure + async to the stress command (v1.0.4)
• Add object-oriented structure + async to the check command (v1.0.5)
• Add object-oriented structure + async to the cmp command (v1.0.6)

---

Adding Async

• replace std::path::PathBuf to async_std::path::PathBuf
• Migrate to Async File Handler

---

Add new dependencies

async-std = "..."
futures = "..."
tokio = { version = "...", features = ["full"] }

Issues

# Don't works, v1.0.3
qt stress -t main.cpp -g gen.cpp --tout 1000 --tc 2 --run-ac 

Add reference of Logo

• add reference of Logo from flaticon.com

chore(ci): Add ci config for Windows

enhancement: add `setup` Subcommand Examples

Command

$ quicktest example --setup

issue: emogi in powershell and windows cmd don't work

chore(ci): Add ci config for MacOS

enhancement: add examples in shortcode

• Add examples to the quicktest example | qt example subcommand

[housekeeping] move the file `src/util/file.rs` to `src/file_handler/file.rs` and remove `util` folder

[housekeeping] move the file `src/diff/diff_line_by_line.rs` to the folder `views` and remove `diff` folder

enhancement: Proposal to add setup command

to change the default values of compiling and running python and c ++

quicktest setup cpp --program=g++ --standard=-std=c++20
quicktest setup python --program=python3

enhancement: example of Interactive Problem Tester

Reference

chore(ci): Update ci config for Linux

colors in the windows terminal are not working

Colors in the windows terminal are not working

Windows

• cmd windows (don't work)
• powershell (don't work)

quicktest --version

quicktest 0.5.1

.memory_limit method not found on Mac OS v1.0.0

QuickTest Version

$ quicktest --version

quicktest 1.0.0
CLI for stress testing in competitive programming contest

Error:

error[E0599]: no method named `memory_limit` found for struct `process_control::control::OutputControl` in the current scope
  --> src/runner/cmd.rs:66:14
   |
66 |             .memory_limit(memory_limit as usize * memory_factor_needed) // bytes
   |              ^^^^^^^^^^^^ method not found in `process_control::control::OutputControl`

For more information about this error, try `rustc --explain E0599`.
error: could not compile `quicktest` due to previous error

issue: default values in the `quicktest setup [cpp | python] --help` command is not working

$  quicktest setup cpp --help

quicktest-setup-cpp 0.7.0
Subcommand that allows to change C++ settings

USAGE:
    quicktest setup cpp [OPTIONS]

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
-        --flags <flags>          Change build command flags arguments in C++ [default: ]
-        --program <program>      Change the program argument of the compile command in C++ [default: ]
-        --standard <standard>    Change the standard argument of the compile command in C++ [default: ]

enhancement: Proposal to change the command to compile a program in c ++

See
g++ -static -DONLINE_JUDGE -Wl,--stack=268435456 -O2 -std=c++17 program.cpp

Support for Memory Limited Exceeded

Reference

[chore] Update rust version to 1.64

update dependency versions

enhancement: Add demos of each of the commands in the website documentation

[housekeeping]: refactor execute_program method in runner/cmp.rs file

eliminate nested conditionals