doctest / doctest

The fastest feature-rich C++11/14/17/20/23 single-header testing framework

master branch
dev branch

doctest is a new C++ testing framework but is by far the fastest both in compile times (by orders of magnitude) and runtime compared to other feature-rich alternatives. It brings the ability of compiled languages such as D / Rust / Nim to have tests written directly in the production code thanks to a fast, transparent and flexible test runner with a clean interface.

The framework is and will stay free but needs your support to sustain its development. There are lots of new features and maintenance to do. If you work for a company using doctest or have the means to do so, please consider financial support. Monthly donations via Patreon and one-offs via PayPal.

A complete example with a self-registering test that compiles to an executable looks like this:

There are many C++ testing frameworks - Catch, Boost.Test, UnitTest++, cpputest, googletest and others.

The key differences between it and other testing frameworks are that it is light and unintrusive:

• Ultra light on compile times both in terms of including the header and writing thousands of asserts
• Doesn't produce any warnings even on the most aggressive warning levels for MSVC/GCC/Clang
• Can remove everything testing-related from the binary with the DOCTEST_CONFIG_DISABLE identifier
• thread-safe - asserts can be used from multiple threads spawned from a single test case - example
• asserts can be used outside of a testing context - as a general purpose assert library - example
• No global namespace pollution (everything is in doctest::) & doesn't drag any headers with it
• Portable C++11 (use tag 1.2.9 for C++98) with over 100 different CI builds (static analysis, sanitizers..)
• binaries (exe/dll) can use the test runner of another binary => tests in a single registry - example

This allows the framework to be used in more ways than any other - tests can be written directly in the production code!

Tests can be a form of documentation and should be able to reside near the production code which they test.

• This makes the barrier for writing tests much lower - you don't have to: 1) make a separate source file 2) include a bunch of stuff in it 3) add it to the build system and 4) add it to source control - You can just write the tests for a class or a piece of functionality at the bottom of its source file - or even header file!
• Tests in the production code can be thought of as documentation/up-to-date comments - showcasing the APIs
• Testing internals that are not exposed through the public API and headers is no longer a mind-bending exercise
• Test-driven development in C++ has never been easier!

The framework can be used just like any other without mixing production code and tests - check out the features.

doctest is modeled after Catch and some parts of the code have been taken directly - check out the differences.

This table compares doctest / Catch / lest which are all very similar.

Checkout the CppCon 2017 talk on YouTube to get a better understanding of how the framework works and read about how to use it in the JetBrains article - highlighting the unique aspects of the framework! On a short description on how to use the framework along production code you could refer to this GitHub issue. There is also an older article in the february edition of ACCU Overload 2017.

Documentation

Project:

• Features and design goals - the complete list of features
• Community driven roadmap - upcoming features
• Benchmarks - compile-time and runtime supremacy
• Contributing - how to make a proper pull request
• Changelog - generated changelog based on closed issues/PRs

Usage:

• Tutorial - make sure you have read it before the other parts of the documentation
• Assertion macros
• Test cases, subcases and test fixtures
• Parameterized test cases
• Command line
• Logging macros
• main() entry point
• Configuration
• String conversions
• Reporters
• Extensions
• FAQ
• Build systems
• Examples

Contributing

Support the development of the project with donations! There is a list of planned features which are all important and big - see the roadmap.

If you work for a company using doctest or have the means to do so, please consider financial support.

Contributions in the form of issues and pull requests are welcome as well - check out the Contributing page.

Stargazers over time

Logo

The logo is licensed under a Creative Commons Attribution 4.0 International License. Copyright © 2019 area55git  

doctest's Issues

[question] Is there a way to always have 0 as the exit code regardless of test results?

I need an option to define exit code value. I am personally interest in setting it to zero, independent from test results, but would be happy to be able to set it for success and failure separately.

Background:
I am currently using this framework within emacs org-mode in an source code block, which can be executed within emacs, adding the result into the document. This, however, does not work if the exit code is not zero, because noting is added. But exactly that case would be of interest.

Example of what I need to do:

#+HEADER: :exports both :results output
#+HEADER: :includes <iostream> <unistd.h>
#+HEADER: :main no
#+HEADER: :flags -I. -std=c++03 -pedantic -Werror
#+BEGIN_SRC C++
#define DOCTEST_CONFIG_IMPLEMENT//_WITH_MAIN
#include "doctest.h"

static int factorial(int number)
{
   return number <= 1 ? number : factorial(number - 1) * number;
}

TEST_CASE("testing the factorial function")
{
   CHECK(factorial(0) == 1);
   CHECK(factorial(1) == 1);
   CHECK(factorial(2) == 2);
   CHECK(factorial(3) == 6);
   CHECK(factorial(10) == 3628800);
}

int main(int argc, char** argv) {
    doctest::Context context; // initialize

    // defaults
    context.addFilter("test-case-exclude", "*math*"); // exclude test cases with "math" in their name
    context.setOption("abort-after", 5);              // stop test execution after 5 failed assertions
    context.setOption("sort", "name");                // sort the test cases by their name

    context.applyCommandLine(argc, argv);

    // overrides
    context.setOption("no-breaks", true);             // don't break in the debugger when assertions fail

    int res = context.run(); // run

    if(context.shouldExit()) // important - query flags (and --exit) rely on the user doing this
        return res;          // propagate the result of the tests
    
    int client_stuff_return_code = 0;
    // your program - if the testing framework is integrated in your production code
    
    return 0;//res + client_stuff_return_code; // the result from doctest is propagated here as well
}
#+END_SRC

#+RESULTS:
#+begin_example
[doctest] doctest version is "1.1.4"
[doctest] run with "--help" for options
===============================================================================
/tmp/babel-6771qy1/C-src-6771uQC.cpp(16)
testing the factorial function

/tmp/babel-6771qy1/C-src-6771uQC.cpp(18) FAILED! 
  CHECK( factorial(0) == 1 )
with expansion:
  CHECK( 0 == 1 )

===============================================================================
[doctest] test cases:    1 |    0 passed |    1 failed |    0 skipped
[doctest] assertions:    5 |    4 passed |    1 failed |
#+end_example

pointer stringification not working for compilers different from MSVC

[feature request] Thread-safety for asserts and logging facilities

Hi,

I would like to know if this framework supports tests that are multithreaded. To clarify, I am not talking about running multiple tests in parallel, but about running one single test that involves multiple threads doing assertions in parallel. I know that Catch and Boost.Test crash when we do that, but I didn't see anything in doctest's feature page nor in its roadmap about this. Is this supported? Will it be?

Add an exception translation mechanism + the ability for users to extend it with custom exception types

In some cases a test will fail because an exception is thrown. Right now it just says
"TEST CASE FAILED! (threw exception)"
It would be very helpful if it says what exception is thrown and the message.

No red when all tests pass

A red "failed 0" looks very scary. Please make the report more peaceful.

[support] identical names for test suites?

I recently ran into a problem while compiling, and it turned out it was caused by having multiple TEST_SUITEs with the same name. I kept getting compilation errors like this:

mysubclass.o: In function `DOCTEST_ANON_FOR_SEMICOLON_3()':
mysubclass.cpp:(.text+0x0): multiple definition of `DOCTEST_ANON_FOR_SEMICOLON_3()'
myclass.o:myclass.cpp:(.text+0x370): first defined here

I now know that TEST_SUITEs need to be uniquely defined by design, but I had to investigate the documentation to realize this.

So the biggest issue is that the compilation errors aren't really helpful for figuring out the cause of the errors. Would it be possible to add some functionality that warns against multiply defined TEST_SUITEs?

Same test runs multiple times when written in a header and included with different unnormalized paths

This happens for me due to "bool TestData::operator<(const TestData& other)" differentiating between files "a/../a/b.h" and "a/b.h", thus including a test twice in "testArray".

Normalizing the paths should help I guess.

Single-letter options active even without leading '-' (dash)

Options -v, -c,, -s, -e are activated even if used without the leading dash.

C++11 nullptr

Howdy,

I'm converting a test setup using Catch to doctest. Most of the process was super easy -- just a global find and replace for catch --> doctest and SECTION -> SUBCASE. But, I'm getting a couple errors.

1. I'm using a check like:

REQUIRE(ptr == nullptr);

and getting an error like:

/home/tbent/projects/taskloaf/taskloaf/lib/doctest.h:267:63: error: ambiguous overload for ‘operator<<’ (operand types are ‘std::ostream {aka std::basic_ostream<char>}’ and ‘std::nullptr_t’)
             static const bool value = sizeof(testStreamable(s << t)) == sizeof(yes);

It looks like this was an issue in Catch previously: catchorg/Catch2#80
and for me it's solved by adding this:

namespace doctest {
template <>
struct StringMaker<std::nullptr_t> {
    static String convert(std::nullptr_t) { return "nullptr"; }
};
}

2. Comparing the size of a collection like:

REQUIRE(my_vector.size() == 4);

produces

comparison between signed and unsigned integer expressions

warnings. While, this is less of a problem than #1, it's always nice to get rid of warnings!

Cheers!

Tests that accompany code run and produce output at default

In the case that unit tests accompany code, user may not expect, or like that at default

• the tests are run,
• the test run produces output (even) if all tests pass.

If tests take a negligible amount of time, running the tests at default may be fine, provided they produce no output on success.

If tests take a non-negligible amount of time, I can imagine people like to skip the tests at default.

Provided running the test at default is retained, it would be nice if the default non-running case could be achieved through a user-supplied main() in combination with a new command line option --run, or --unittest (the flag used by D).

int main(int argc, char** argv) {
    doctest::Context context(argc, argv); // initialize

    // overrides
    context.addFilter("no-run", !context.flag("run")); // control --no-run from --run
    ...
}

Note that currently configuring the context can only occur after processing the commandline options in Context's constructor, hence the comment mentioning overrides. The lack of being able to set defaults before commandline processing leads to the convoluted example above.

Supporting configuring defaults, e.g.:

int main(int argc, char** argv) {
    doctest::Context context;
    context.addFilter("no-run", true);  // default

    // --run (--unittest) on the commandline now overrides the --no-run default
    context(argc, argv);  // or context.apply(argc, argv);, or ...
    ...
}

How to force the use of colors in the terminal?

Defining DOCTEST_CONFIG_COLORS_ANSI is not working, only using ./main --force-colors=true works

My project source code files are like these:

doctest_main.cpp

/**
 * This tells to provide a main() - Only do this in one cpp file.
 */
#define DOCTEST_CONFIG_COLORS_ANSI
#define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN
#include "doctest.h"

If here on doctest_main.cpp or everywhere else I tried, I defined the DOCTEST_CONFIG_COLORS_ANSI, it does nothing. It keeps not showing any colors. The only way to force color is to call the main application as ./main --force-colors=true.

---

doctest_tests.cpp

#include "project_sources_with_tests.cpp"
//...

---

project_sources_with_tests.cpp

#include "debug.h"

unsigned int factorial( unsigned int number )
{
    return number <= 1 ? number : factorial(number-1)*number;
}

TEST_CASE("testing the factorial function")
{
    CHECK(factorial(1) == 1);
    CHECK(factorial(2) == 2);
    CHECK(factorial(3) == 6);
    CHECK(factorial(10) == 3628800);
}

---

debug.h

/**
 * This is to view internal program data while execution. Default value: 0
 *
 *  0  - Disables this feature.
 *  1  - Basic debugging.
 *  2  - Run the `doctest` Unit Tests.
 */
#define DEBUG_LEVEL 1



#define DEBUG_LEVEL_DISABLED_DEBUG 0
#define DEBUG_LEVEL_BASIC_DEBUG    1
#define DEBUG_LEVEL_RUN_UNIT_TESTS 2

#if DEBUG_LEVEL > DEBUG_LEVEL_DISABLED_DEBUG

    #define DEBUG

    /**
     * Disables the `doctest` Unit Tests from being included/compiled to the binary output file.
     *
     * See:
     * https://github.com/onqtam/doctest/blob/master/doc/markdown/configuration.md
     */
    #if !( DEBUG_LEVEL & DEBUG_LEVEL_RUN_UNIT_TESTS )
        #define DOCTEST_CONFIG_DISABLE
    #endif

    #include "doctest.h"

#endif

---

I am running the project through a Sublime Text build system -> shell script -> Makefile:

project.sublime-project

{
    "folders":
    [
        {
            "path": ".",
        },
    ],
    "build_systems":
    [
        {
            "working_dir": "$project_path/source",
            "file_regex": "^(..[^:]*):([\\d+]):?(\\d+)?:? (.*)$|^\\(.*\\)\\((\\d+)\\)(.*)$",

            "name": "Doctest Driver Class",
            "cmd": ["sh", "make_run.sh", "doctest_tests"],

            "target": "ansi_color_build",
            "syntax": "Packages/ANSIescape/ANSI.tmLanguage"
        }
    ]
}

---

make_run.sh

FIRST_COMMAND_ARGUMENT=$1

if [[ $FIRST_COMMAND_ARGUMENT == "main" ]]
then
    make clean
    make

elif [[ $FIRST_COMMAND_ARGUMENT == "veryclean" ]]
then
    make $FIRST_COMMAND_ARGUMENT
    make

else
    make clean
    make $FIRST_COMMAND_ARGUMENT
fi

wait $!

---

Makefile

PROGRAM_MAIN_DEBUGGER = debug

DOCTEST_TESTS_FILE   = doctest_tests
DOCTEST_DRIVER_CLASS = doctest_main


doctest_tests: $(DOCTEST_DRIVER_CLASS).o $(PROGRAM_MAIN_DEBUGGER).h
	g++ --std=c++11 $(DOCTEST_DRIVER_CLASS).o $(DOCTEST_TESTS_FILE).cpp -o main
	./main

$(DOCTEST_DRIVER_CLASS).o: $(DOCTEST_DRIVER_CLASS).cpp
	g++ $(DOCTEST_DRIVER_CLASS).cpp -c -o $(DOCTEST_DRIVER_CLASS).o

Add the include file to releases

Would you mind adding the single header include file to releases, which can easily be done by uploading via Github online release page. This will make it very easy to integrate with some other build systems and would save some time on low-speed connections too. I know we can download/clone the whole source anyways, but it would be much helpful if a direct download link to main file is provided :)

Thankyou

Add support for un-parenthesized expressions containing commas in asserts

The aim is for this to be possible:

CHECK(std::vector<int>{1, 2} == std::vector<int>{1, 3});
CHECK_THROWS_AS(throws(true), std::pair<int, int>);

for more info see this Catch issue

DOCTEST_BREAK_INTO_DEBUGGER undefined under OSX

Hi,

When I use doctest with my Mac (up-to-date XCode :-), DOCTEST_BREAK_INTO_DEBUGGER is not defined.

Steps to reproduce :
Clone https://github.com/pthom/DocTest_LibTest and compile on a mac (cmake / make)

See for example
https://github.com/pthom/DocTest_LibTest/blob/master/MyLibrary/lib1.cpp
where I had to define it in order to avoid an error

Thanks !

Add option --force-colors - for when a tty is not detected for stdout

this is the issue for #50 which didn't get merged - issue needed for the automatic changelog generation

restrict expressions in assertion macros to binary comparisons at most with a static assert

as discussed here

Tests inside a static library

Hi,

(This is more a question than an issue)
What is the recommended way to add tests inside a static library project ?

If I compile a library (which includes TEST_CASE(s)) as a static library,
and then I try to run the tests from this library by creating a target that links
to it, the test suite is actually empty (i.e I guess the tests were stripped at link time)

Am I doing something wrong, or is there a way to circumvent this?

Below is an example of what I am trying to achieve (using cmake):

Project         <- main project
  CMakeLists.txt
  doctest/
    doctest.h
    doctest_main.cpp
  MyLibrary/
    lib1.cpp    <- these files includes actual library code and TEST_CASE(s)
    lib2.cpp    
    lib3.cpp    
    CMakeLists.txt <- will produce two targets : library and its test
  MyMainProgram/
    main.cpp
    CMakeLists.txt

MyLibrary/CMakeLists.txt :

    set(sources lib1.cpp lib2.cpp lib3.cpp)
    add_library(MyLibrary STATIC ${sources})
    target_include_directories(MyLibrary PUBLIC ${CMAKE_SOURCE_DIR}/doctest )

    # This does not work : it links but the test list is empty !
    add_executable(MyLibrary_DocTest ${CMAKE_SOURCE_DIR}/doctest/doctest_main.cpp)
    target_link_libraries(MyLibrary_DocTest MyLibrary)

    # This works, but it requires to re-compile all the source files and to re-define
    # exactly the same compile options as for the library (linked library, compile definition, etc); 
    # this can be tedious
    add_executable(MyLibrary_DocTest ${sources} ${CMAKE_SOURCE_DIR}/doctest/doctest_main.cpp)

MyMainProgram/CMakeLists.txt :

    add_executable(MyMainProgram main.cpp)
    target_link_libraries(MyMainProgram MyLibrary)

A working example can be found at https://github.com/pthom/DocTest_LibTest

Add API for reporting failures

It would be marvellous if there was a supported way to report failures from 3rd party tools. As mentioned, you can throw exceptions, but that is not always a good idea (during stack rollback, for example.) There is also the option to just use REQUIRE_FALSE(message), but it does not give as nice messages and attributes file/line to the function that does the reporting.

In a proof of concept reporter for trompeloeil mocking framework I successfully used:

struct initializer {
  initializer() {
    trompeloeil::set_reporter([](trompeloeil::severity s,
                                 char const *file,
                                 unsigned long line,
                                 std::string const& msg)
                              {
                                doctest::detail::logAssert(false,
                                                           msg.c_str(),
                                                           false,
                                                           s == trompeloeil::severity::fatal
                                                           ? "unexpected call"
                                                           : "unfulfilled expectation",
                                                           "trompeloeil violation",
                                                           file,
                                                           line);
                                if (s == trompeloeil::severity::fatal)
                                  throw std::runtime_error("fatal error in trompeloeil expectation");

                              });
  }
} initializer_obj;

This works fine, but relying on undocumented internal functions is never a good idea.

GCC 5.3.1 Compiler warning: sign compare

Compiler:
g++ (Debian 5.3.1-21) 5.3.1 20160528

Code:

#define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN
#include "doctest/doctest.h"

#include <map>

TEST_CASE("x")
{
    std::map<int, int> m;
    CHECK(m.size() == 0);
}

Command line (unimportant flags like include paths removed):

/usr/bin/c++ -std=c++14 -Wall -Wextra -Wshadow -Wnon-virtual-dtor -pedantic -Wcast-align
-Woverloaded-virtual -g -fprofile-arcs -ftest-coverage -fno-omit-frame-pointer -O0 
-std=gnu++14
-o CMakeFiles/x.dir/tests/x.cpp.o

Result (full paths removed):

In file included from /somewhere/tests/x.cpp:2:0:
/somewhere/tests/doctest/doctest.h: In instantiation of ‘typename doctest::detail::enable_if<(doctest::detail::can_use_op<T>::value || doctest::detail::can_use_op<R>::value), bool>::type doctest::detail::eq(const L&, const R&) [with L = long unsigned int; R = int; typename doctest::detail::enable_if<(doctest::detail::can_use_op<T>::value || doctest::detail::can_use_op<R>::value), bool>::type = bool]’:
/somewhere/tests/doctest/doctest.h:593:82:   required from ‘doctest::detail::Result doctest::detail::Expression_lhs<L>::operator==(const R&) [with R = int; L = const long unsigned int&]’
/somewhere/tests/x.cpp:9:2:   required from here
/somewhere/tests/doctest/doctest.h:557:127: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
  || can_use_op<R>::value, bool>::type eq (const L& lhs, const R& rhs) { return lhs == rhs; }
                                                                                    ^

Slower than Catch in realistic test cases

Hi,

I did some benchmarking of doctest on my machine (Linux, gcc-4.9.3). I adapted a bit your Python script to work on Linux (with_gcc = 1 and removed the forced CMake generator).

First, I can confirm that in the simple benchmark case, it is indeed faster to compile than Catch by a large margin. However, this is not a very realistic test case.

I have modified the benchmark to generate 50 files, each with 25 test cases and each with 10 CHECK (using a variable and not two constants). Here are the results:

Catch: 1 minute 56 seconds
Doctest: 2 minutes 32 seconds

In a realistic test case, doctest is significantly slower than Catch. I can see the same thing when I replace Catch by doctest in one of my large test case of my ETL library.

Do you have any idea why this happen ?

I can only guess that the overhead comes from the templates used to extract lhs and rhs or by the macros generating the test cases.

clang warnings when using C++11 or newer

Clang complains about Result and Subcase requiring implicit copy constructors when they have user defined destructors (deprecated in C++11).
doctest/doctest.h:727:9: warning: definition of implicit copy constructor for 'Result' is deprecated because it has a user-declared destructor [-Wdeprecated]
...
doctest/doctest.h:847:89: note: implicit copy constructor for 'Result' first required here template <typename R> Result operator==(const DOCTEST_REF_WRAP(R) rhs) { return Result(lhs == rhs, stringifyBinaryExpr(lhs, " == ", rhs)); }

doctest/doctest.h:2312:14: warning: definition of implicit copy constructor for 'Subcase' is deprecated because it has a user-declared destructor [-Wdeprecated] ...
This one was caused by pushing a Subcase onto a vector (line 2303).

(doc)test doctest?

Will you dogfeed doctest?

Compile time awful slow with Visual Studio with optimization

Today I tried to migrate a bunch of legacy tests to doctest. I have to say that I was really satisfied with the outcome and the neat usage of this excellent testing framework. Migration made it necessary to compile hundreds of cpp files.

Firstly, I build the debug version of my tests and the compilation time taking 55s was very short considering the amount of tests I had to compile.

Secondly I build the release version of my library and the compilation really took ages. After 487s the compilation was finished.

After I used the compilation switches /Od /Ob0 instead of /O2 and /Ob2 the compilation speed up to amazing 43s. Unfortunately, the Executable became slow.

I'm wondering what I did wrong?

Issue with using lambdas in tests in gcc

I have come across an issue where two test cases in separate source files, that happen to acquire the same name (due to COUNTER macro being per compilation unit), and using lambdas inside them are compiled (or linked) incorrectly. The issue is certainly in either gcc or the linker (or both) and relates to how it handles inline functions.
In this particular case the issue is in macro DOCTEST_CREATE_AND_REGISTER_FUNCTION. It forwards declares the function as static, and then implements it as inline, but without "static". The fix is in either defining the function as static inline, or just forget about inline (after all the name is going ot be unique within a compilation unit, inline seems unnecessary here).

I have managed to reproduce the compiler(or linker) bug without doctest, I can supply more details if interested. But in this particular case, when two modules declare static functions of the same name, and then define them as inline without "static", and also both declare and call a lambda inside, both functions will end up calling the same lambda (from one of the functions, the other one will not be in the executable at all).

I can create and submit a PR with a fix, but it is as simple as changing
inline void f()
to either
static inline void f()
or
static void f()

This is line 1188 in the latest version at the time of writing.

creative formatting of chars

I acknowledge that in many contexts people use chars to represent - well - characters. However, in many cases they are actually holding things that are not "printable" characters. And even when they are printable it is hard to know just how they should be printed.

The current behaviour is surprising (imho) - eg:

char one = 1;
char another_one = '1';
REQUIRE(one == another_one);

FAILED!
REQUIRE( one == another_one )
with expansion:
REQUIRE ( 1 == 1 )

Options seem to be

Print a 2 character expansion of "reserved" chars eg output one as '\1' and of course, '' for the dreaded \

Or concede that chars are just very short integers that may well be used to represent characters (which ones is highly dependent on locale) and "promote" them (in effect or literally) by printing them as integers. This is conventional/expected in C/C++ anyway? Far more expected than this:

PASSED!
CHECK( one+'0' == another_one )
with expansion:
CHECK( 49 == 1 )

drop VC++6 support

Add option to print which test suites/cases are run

It would be nice to have options to enable output which test cases are running (similar to the -s option) and also show whether they passed or failed.

Add clearFilters() to doctest::Context

I just wanted to let you know that doctest works great for me running on an Analog Devices ADSP21479 SHARC MCU. For that to work I had to add this macro to my build: DOCTEST_CONFIG_COLORS_NONE and _LIBCPP_VERSION.

Since I am running the tests menu driven (The Analog Devices Eclipse based CrossCore Embedded Studio 2.3.0 allows printf / scanf through the Eclipse Console.) I also created my own main function (DOCTEST_CONFIG_IMPLEMENT). Because doctest is not geared towards menu driven test I added one function to the code which allows me to run only one test the user has chosen from the menu. If you like to incorporate it into the next version, please go ahead. The function just clears the filter vectors.

BTW, it takes a few seconds to compile doctest under the Analog Devices compiler (_ADI_COMPILER) on my decently fast Windows 7 PC.

Thanks for the great work,

Günter Fuchs
[email protected]

doctest.txt

Add support for templated test cases - parameterized by type

I cannot figure out if this framework supports or plans to support parametric test cases, where I define a test parametrized by some value, and then instantiate a number of sibling tests with different values; or I define a test parametrized by a type (like a template) and then instantiate a number of sibling test cases using a different type.

Frameworks like Gtest or Boost.Test support them. For reference see:

1. http://www.boost.org/doc/libs/1_61_0/libs/test/doc/html/boost_test/tests_organization/test_cases/test_case_generation.html
2. http://www.boost.org/doc/libs/1_61_0/libs/test/doc/html/boost_test/tests_organization/test_cases/test_organization_templates.html#ref_BOOST_TEST_CASE_TEMPLATE
3. https://github.com/google/googletest/blob/master/googletest/docs/AdvancedGuide.md#value-parameterized-tests
4. https://github.com/google/googletest/blob/master/googletest/docs/AdvancedGuide.md#typed-tests

Can we expect a similar functionality in this library?

re-enable valgrind on OSX

before the 1.0.0 release I disabled the use of valgrind even for 64 bit builds because 2/3 of all OSX builds failed after I added better examples testing more of the features of the framework (mainly what happens when an assert/test case throws unexpectedly) - this is the commit that disables the use of valgrind.

I'm 100% sure the problem isn't in doctest - so I'm hoping for some workaround (or a suppression file for valgrind might be the correct solution)

here is the travis build that failed... Check out the specific builds that failed and search for "error"

False positive test

This simple program:

#define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN
#include "doctest.h"

TEST_CASE("This is doctest") {
    CHECK(1==9 || 1==8);
}

Compiled with g++ 5.4.0 or clang++ 3.8.0 generates a program that, when executed, shows:

[doctest] doctest version is "1.0.0"
[doctest] run with "-dt-help" for options
===============================================================================
[doctest] test cases:    1 |    1 passed |    0 failed
[doctest] assertions:    1 |    1 passed |    0 failed

But obviously "1==9 || 1==8" is false.

(A similar program with Catch produces an error saying "STATIC_ASSERT_Expression_Too_Complex_Please_Rewrite_As_Binary_Comparison")

I don't known if this is a known limitation, but it is a disturbing one.

Ability for a binary (executable / shared object) to use the test runner implementation of another binary - with exported symbols - so tests end up in a single registry

Exception handlers cause warnings when exceptions are disabled

C++ exceptions in my opinion (and in the opinion of others) are subpar and I usually have them turned off along with RTTI. However,
the presence of try/catch blocks in doctest causes warnings (at least on MSVC - C4530) which can't be suppressed (and I'm building with /W4 /WX so this breaks my build).

I simply deleted / commented out all usages of try/catch in the my copy of the library but it will be helpful to only use exceptions handling if the user's exceptions are enabled. This can be detected by checking for

• __cpp_exceptions on gcc
• _HAS_EXCEPTIONS on MSVC

Add custom failure messages with lazy stringification

Is it or will it possible to output custom messages only when REQUIRE or CHECK fails? In Catch, we can capture certain variables before doing the check, but that unnecessarily creates a string and then discards it when the test succeeds (which should be more common than failures).

Add ability to filter subcases with filters

add configuration preprocessor identifier for passing by value in assertions instead of by reference

add <= and >= operators to Approx (and also maybe < and >)

UNIX line feedings on GitHub please

By using .git/config, you can control the line feeding for your local clone, but I hope to see UNIX line feedings on the public site.

A way to refrain from polluting “#define” space for users of tested code?

I’m writing a templated geometry library based on rational numbers, and I’m having a great time using doctest in it.

However, I’d like to allow users of my library to also use doctest without them inadvertently re-testing my code or having to worry about include order. At first, I thought I might just use a preprocessor guard to allow me to ignore the defined-ness state of DOCTEST_CONFIG_DISABLE in the body of my code (turn it on/off at the start of each of my files, put it back how it was at the end), but then, of course, I realized this couldn't possibly work, since once doctest.h is included, it would necessarily define all the macros I’m trying not to pollute the “#define” space with.

I then tried custom-named macros, with DOCTEST_CONFIG_NO_SHORT_MACRO_NAMES defined, followed by another #include "doctest.h", but even when I put a #define DOCTEST_CONFIG_DISABLE before include, the effects off a previous include’s macro definition were felt and my tests still registered.

Is there already support for what I’m trying to do? Could the functionality of DOCTEST_CONFIG_NO_SHORT_MACRO_NAMES be extended to not affecting the state of already defined short macros, but still allowing another include to define the long macros in a different DOCTEST_CONFIG_DISABLEed (on or off) state, ready for namespace-esque custom redefinition?

Only one SUBCASE per line is executed

I found an important difference with Catch: Only one SUBCASE per line is executed.

For instance, this test case:

TEST_CASE("lines") {
    SUBCASE("A"){ std::cout << "a";} SUBCASE("B"){ std::cout << "a"; }
}

Will only print "a" and not "aa". Catch (with SECTION) executes correctly both sections.

This makes a big difference when you are using SUBCASE in a macro. In which case, you'll have several SUBCASE per line once the preprocessor is done. I'm using this to create templated test case. This work well with Catch, but does not work with doctest.

This probably comes from the fact that pairs of files and lines are stored in subcasesPassed instead of triple of files, lines and names.

Add ability to capture variables from test scope

I'm in the position where I have to generate random combinations of a fixed length string and then run a parser function on them and compare the result of the function with a known value. However, if the assertion fails, I don't know what was in the string that failed to be parsed.

I remember that in Catch there was a macro called CAPTURE or something like that. Which allowed me to see the contents of that variable in case an assertion macro failed in that scope.

Is there something similar in doctest or a way to implement it? I can't seem to find anything related in the reference or tutorials.

Rename doctest::detail::Result res; in DOCTEST_ASSERT_IMPLEMENT

doctest::detail::Result res; takes up the quite common symbol name res and makes a test like CHECK(res == x); impossible.

Please rename it to something which would be much more unlikely to clash like __doctest__res__

Interaction with --gc-sections ?

When compiling a testcase with -ffunction-sections and -fdata-sections, and linking with --gc-sections (or alternatively, -flto entirely), do the test cases get removed if there is no runner in the executable, or if it is not linked in?

Enhance BDD support

I really enjoy writing tests using the BDD syntax - the popular chai.js library is a good example (http://chaijs.com/api/bdd/) and although doctest has some support for that (SCENARIO, THEN, WHEN) it would be great to have this support further enhanced.

How can I mix production code with the Unit Tests?

How can I mix production code with the Unit Tests?

Initial page says:

This allows the framework to be used in more ways than any other - tests can be written directly in the production code!

• This makes the barrier for writing tests much lower - you don't have to: 1. make a separate source file 2. include a bunch of stuff in it 3. add it to the build system and 4. add it to source control - You can just write the tests for a class or a piece of functionality at the bottom of its source file - or even header file!
• Tests in the production code can be thought of as documentation or up-to-date comments - showing how an API is used
• Testing internals that are not exposed through the public API and headers becomes easier!

How can I mix production code with the Unit Tests?

I am thinking about just doing the #include "doctest.h", and writing the tests down, but how the Unit Tests code will not end up on the production binary after compiled from the main program, instead of the Unit Tests Driver Class:

#define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN
#include "doctest.h"

I found out this macro DOCTEST_CONFIG_DISABLE. Therefore how would it work, I would have to define it on every production source code file, to disable the Unit Tests?

char* comparison uses the contents, not the pointer

char* a = strdup("Hello");
char* b = strdup("Hello");
CHECK(a == b); // passes, even though it shouldn't

Looking at the code, it seems like 'eq' is transforming the value to a string, and then doing the comparison, which doesn't do the right thing in this particular case. Casting the pointers to void* seems to solve it.

Add option to query if code is being ran inside of a test - doctest::is_running_in_test

Turn off coloring after tests are finished?

Howdy,

Simple issue. When doctest runs, it uses some colored text. If further information is printed after doctest runs, then the rest of the text is all red. Not a huge issue, but I thought I'd report it.

Ben

[doctest] run with "-dt-help" for options
===============================================================================
[doctest] test cases:    2 |    2 passed |    0 failed
[doctest] assertions:   10 |   10 passed |    0 failed
------ everything below here will still be red ------ 
.
running tests/test_somemorestuff
running tests/test_evenmorestuff

Add example how to remove doctest options from the command line for the program after the tests run

When test code is embedded with the application, the application code would preferably not have to deal with commandline options that belong to the test code. See example below.

Currently there seems to be no provision to only provide the application code with the non-doctest options.

#define DOCTEST_CONFIG_IMPLEMENT
#include "doctest.h"

int program(int argc, char** argv);

int main(int argc, char** argv) 
{
    doctest::Context context;  

    context.setOption("--exit", false); // continue after tests, unless --exit

    context.applyCommandLine(argc, argv);

    int test_result = context.run();    // run queries, or run tests unless --no-run

    if (context.shouldExit())           // honor query flags and --exit
        return test_result;

    int app_result = program(argc, argv);

    return test_result + app_result;
}

TEST_CASE("Fail") { REQUIRE(0); }
TEST_CASE("Pass") { REQUIRE(1); }

int program(int argc, char** argv) 
{
    printf("\nProgram code, %sarguments received:\n", argc > 1 ? "":"no ");
    while ( *++argv )
        printf("%s\n", *argv);
    return EXIT_SUCCESS;
}

Compile and run:

prompt> g++ -Wall -Wextra -o main.exe main.cpp && main.exe --no-colors=true --my-option=abc 123
[doctest] doctest version is "1.0.0"
[doctest] run with "--help" for options
===============================================================================
main.cpp(44)
Fail

main.cpp(44) FAILED!
  REQUIRE( 0 )
with expansion:
  REQUIRE( 0 )

===============================================================================
[doctest] test cases:    2 |    1 passed |    1 failed |    0 skipped
[doctest] assertions:    2 |    1 passed |    1 failed |

Program code, arguments received:
--no-colors=true
--my-option=abc
123

!!! BREAKING CHANGE !!! - Make TEST_SUITE work with blocks and add TEST_SUITE_BEGIN

I think making the following code valid would be a good idea

TEST_SUITE("math")
{
    TEST_CASE("") {} // part of the math test suite
}

and adding the macro/typedef TEST_SUITE_BEGIN, to match TEST_SUITE_END

TEST_SUITE_BEGIN("gui")

TEST_CASE("") {} // part of the gui test suite

TEST_SUITE_END();