fdefelici / clove-unit Goto Github PK

Add commandline options for choosing:

• report format: like pretty print and json (options: -r, --report)
• output stream: stdout or file (output: -o, --output)

Currently, if some test fail, the process exit with error code 1.

This could be misleading when you want to know if the process ended gracefully, or exit by applicative or system error.

The idea could be to make the behaviour to return error code, in case of test fail configurable using a specific flag on command line.

For example:
<exec> -x will exit with error in case of test failure.
otherwise it will exit with 0, if the process end gracefully.

NOTE: This is a breaking change with vscode-clove-unit extension

Hello,

when compiling with sanitizers (I used memory, leaks, and undefined), I get a memory leak during the call to the following macro call (I guess it returns early and some temporary malloc is not freed)

__CLOVE_VECTOR_FOREACH(&cmd_handlers, __clove_cmdline_handler_f, handler, {
        __clove_cmdline_errno_t result = (*handler)(&cmdline);
        if (result != __CLOVE_CMD_ERRNO_UNMANAGED) {
            cmd_result =  result;
            break;
        }
    });

I tried to fix it by using various free functions but did not manage to make it pass the CI tests. I'm probably missing some understanding in here. In particular I tried adding

__clove_vector_free(&cmd_handlers);
__clove_cmdline_free(&cmdline);

on the other branch of the if but I guess it frees too much (it indeed removes the memory leak but..... then no test is executed anymore in the CI...).

To add the sanitizers you can add the following lines in the CMakeLists.txt files:

add_compile_options(-fsanitize=address,leak,undefined -g)
add_link_options(-fsanitize=address,leak,undefined)

Edit: formatting.

For pretty report, in case of skipped test, print suite file path followed by test case line number.

NOTE:

• check if for some reason this information could be useful also for csv and json report.
• update image in readme with new skipped test output

Feature that could be helpful for the one running clove-unit tests from a Terminal within an ide (e.g. vscode).
Basically the idea is to produce a path-line format so that the ide succeed to create an hyperlink, making the failining testcase easyly reachable with a click (or CTRL+click depending on ide).

Currently in pretty print report, on test with failure the path line are printed like this:

<AbsolutePathToTestFile>@<LineNumber>

Convert this format as c/cpp compilers do, so that will allow an ide with terminal hyperlink feature to create the link with the real file. Possible choises:

• <AbsolutePathToTestFile>(<LineNumber>)
• <AbsolutePathToTestFile>:<LineNumber>

NOTE: Compilers format is (<LineNumber>,<ColumnNumber>) or <LineNumber>:<ColumnNumber>, but for this usecase, ColumnNumber could be omitted and hyperlink would be created anyway

This an example of the hyperlink created by vscode from its terminal:

Introduce an 'in memory' stream to favor unit testing for pretty report and reduce the number of integration testing (when not really needed) introduced with #42

Compiling with msvc with warning flag /W4 emits some warnings.

produce test coverage in the build pipeline and integrate with codecov service

consider this (not fully complete) implementation:

#define AS_STRING(x) #x

#define __CLOVE_SUITE_CALL_METHOD_AUTO_2(suite, title, param) __clove_sym___##suite##___##title(param)
#define __CLOVE_SUITE_CALL_METHOD_AUTO_1(suite, name, param) __CLOVE_SUITE_CALL_METHOD_AUTO_2(suite, name, param)
#define __CLOVE_SUITE_PRINT_METHOD_AUTO(suite, name) printf("%s %s %s %d\n", AS_STRING(suite), #name, __FILE__, __LINE__)

#define __CLOVE_TEST_AUTO(title) \
	__CLOVE_SUITE_METHOD_AUTO_1( CLOVE_SUITE_NAME, 30_ ## title, __clove_test_t *_this); \
	__CLOVE_SUITE_METHOD_AUTO_1( CLOVE_SUITE_NAME, 20_ ## title, __clove_test_t *_this) { \
		if (print_tests) \
			__CLOVE_SUITE_PRINT_METHOD_AUTO(CLOVE_SUITE_NAME, title); \
		else \
			__CLOVE_SUITE_CALL_METHOD_AUTO_1(CLOVE_SUITE_NAME, 30_ ## title, _this); \
	} \
	__CLOVE_SUITE_METHOD_AUTO_1( CLOVE_SUITE_NAME, 30_ ## title, __clove_test_t *_this)

when print_tests is true we will get print output of all the tests and when print_tests is false the tests will run as usaul.

Split automation in 2 action: CI and RELEASE

Release CLove Unit as a Conan package on https://conan.io/center/

Currently, all the executed tests are listes on the console with PASS, SKIP, FAIL status.

Allow to launch test in "Detail Mode" where only tests with FAIL status are listed.
The "Detail Mode" will provide stats like: total tests, passed/skip/fail count, execution time.

"Detail Mode" could be activated on run tests execution using options:

<program> [-d , --run-detail <detail_mode>]

where <detail mode> could be one of this values:

• 1 : will report only failed tests
• 2 : will report only failed and skipped tests
• 3 : will report all tests (passed, failed and skipped). This is also the default.

NOTE: Need also update:

• README update to expose command line commands
• list option in help command

Could be useful for integration testing when you have no control over logic preparing the string and exists some parts that are not deterministic.

This also could be used for improving clove-unit integration testing where the non-deterministic part is the test duration.

Api example:

CLOVE_STRING_REGEX(expected_with_regex, actual)

If expected_with_regex doesn't contain any regex, this should behave like CLOVE_STRING_EQ.

Related to #44

Now all the function are static so are copied for every inclusion of the header file.
To reduce binary size:

• split function definition from implementation
• use CLOVE_IMPLEMENTATION macro to let the user to inclue only once the implementation.
  Something like this

#define CLOVE_IMPLEMENTATION
#include "clove-unit.h"

Hello,

first let me say that this library looks really awesome. Thanks for the great work.

I have nevertheless a slight problem when compiling with gcc or clang (with -Wall, -Wextra, and -pedantic). There are a very large amount of warnings (unknown-pragmas only with gcc, unused parameters, defined but not used variables, unused variables, etc).

I tested with gcc 11.4 and clang 14.0 on Ubuntu.

Unfortunately this is problematic for use for using in a testing framwork since it pollutes a lot the possible warnings of the actual library one wants to test.

Most problems are related with using linux I think and could be corrected with a lot of pre-processing directives spread all over the place.

What would you think about the idea of creating a MR about this?

Evaluate if add assertion for size_t type:

• CLOVE_SIZET_EQ(exp, act)
• CLOVE_SIZET_NE(exp, act)

Make the command line accept list of test to be run by file.

This could help in some automation/plugin scenario, where the test list is very huge (and command line characters are limited. Eg. windows around 8000 char) to be passed directly on the command line using multiple -i (--include) option.

To improve semantic add CLOVE__CHAR_EQ and CLOVE__CHAR_NE (instead of using INT version), and have a better output message on console in case of failure

Add test duration on the test report

Need to introduce test suite concept so that can have:

• SetUp / TearDown: executed for each test
• A List of test to be run
• Nice to have: SetUpOnce / TearDownOnce: executed before/after all the test in the list

Conan release process foresees to open a pull request agains conan-center-index.

Understand if could be automatized.

update report: pretty, csv, json to support:

• new datatype size_t
• GT/GTE/LT/LTE assertions

the idea is to make possibile to implement a custom runner.

This will allow the user to execute custom logic before/after clove-unit test execution

Idea is to provide additional assertions when dealing with numeric types: Greater Than, Greater than or Equal, Less Than, Less than or Equals ...

Understand if make sense or not.

Currently aligment (with dot character) is hardcoded.

Eventually allow the user to specify the amount of aligment in case where some "Suite.Test" are not entirely visible.

Or eventually evaluate the possibility to automatically align to the longest "Suite.Test" string

When using CLOVE_FAIL() assertion, issue type is null and appear in the json report like this:

Potential fix is to set issue type to "" (empty string)

Implement help command showing possibile options for a clove-unit executable.

README console api need to be updated

idea is to provide assertion helper when dealing with strings like: Contains, StartWith, EndWith....

Understand if make sense or not.

NOTE: Breaking change for VSCode Plugin because adding more string assertion type

Receiving this warning during build

actions/checkout@v2 needs to be upgraded to actions/checkout@v2

Current pretty print behaviour for list test command (-l, --list-test options)

The idea is to return a format that allow ide hyperlink for suite and test cases (as done in #75 )

Consider the floating-point precision configuration. Currently, the following assertion returns true with:

CLOVE_FLOAT_EQ(0.00000198765435, 0.00000112345678);

Should it be specific to each suite or should it be global?

On Windows cmd console has not Ansi codes enabled so you will obtain something like this:

Understand if it possible to check at runtime if console support coloring, or eventually enable Ansi

Now when a test fail, the full file path is printed.

Make configurable for the user the path to print a relative path instead by command line option: -b or --base-path

This option works for the commands: run tests and lists tests and for all report format (pretty, csv, json)

Currently -i / -e options accepts expressions involving Suite name and Test name.

When running test using pretty report a test index is shown:

So eventually could be usefull allow include/exclude options to accept also test indexes, like this:

• <program> -i 1 will run the test with "01" index
• <program> -e 2 will run the test with "01" and "03" index

This kind of change will have a huge impact on test filtering process, because filtering happen while retrieving test symbols while test index is assigned after when test run (depending on the number of test executed)

NOTE:

• to update commandline api documentation
• to update help command

In the current implementation, all public assertions end with semicolons, leading to empty statements when assertions are followed by semicolons, as shown in the sample code and screenshot below:

To address this issue, one solution would be to remove semicolons from public assertions, requiring developers to use semicolons after assertions. However, this change would have an impact on developers who have not previously used semicolons after their assertions in order to avoid empty statements.

Another potential solution is to update the documentation and sample code to indicate that semicolons are not required after assertions.

Allow to launch or exclude suites/tests from command line

Add assertion CLOVE__PASS() e CLOVE__FAIL() to make test to pass or fail rispectively

add support for travis-ci and release automation creation.

Try compiling / executing with different compiler / os

The idea could be to make cmdline arguments available within the test case, in order to implement custom logic.

Eventually from this args could be filtered out the one related to clove-unit command line, and just keep the user defined parameters.

Potential use cases:

• Execute tests assertion on a dynamic collection for performances. So from commandline, one can input the number of elements
• Dynamically change some assertion value
• Even skip test execution following custom logic (instead of using -include -exclude params)

__clove_path_basepath uses strdup and calloc. This results in two separate memory allocations, which is inefficient and may cause memory fragmentation. A better approach would be to use only one allocation as mentioned in the function itself.
See: clove-unit.h#L818

Add test function line number in both csv/json test execution report.

This to open better intergration with plugins and automation.

NOTE:

• For CSV format could have some impact in naming properly column names (already exits 'line' column referred to the assert withing the test)
• Maybe better to split this issue in 2, one related to the json (without impact) and one related to csv.

shield.io api for build badge has been changed and is now broken

See issue: badges/shields#8671

Dropping manual mode feature. Autodiscovery implementation is good enough, so no need to keep the manual one.

Currently test counter is like this:

[INFO] 9) UNIT_StringViewTest.Contains.............................[PASS] (0.011 ms)
[INFO] 99) UNIT_StringViewTest.AsString............................[PASS] (0.011 ms)
[INFO] 100) UNIT_StringViewTest.BeginAndEnd........................[PASS] (0.001 ms)

It could become like this:

[INFO] 009) UNIT_StringViewTest.Contains...........................[PASS] (0.011 ms)
[INFO] 099) UNIT_StringViewTest.AsString...........................[PASS] (0.011 ms)
[INFO] 100) UNIT_StringViewTest.BeginAndEnd........................[PASS] (0.001 ms)

So '0' left padding to reach same length of the highest test number.

NOTE: Update reference image (pretty print) in Readme