stablecoder / cmake-scripts Goto Github PK

Hi, right now the ccov-clean target uses rm to remove the binaries.list and profraw.list files.

It would be nice not to have to jump in WSL to generate code coverage reports.

I am using cmake-scripts through CPM, which accepts only tags (or semantic versioning tags, v.X.Y.Z) to specify which version of the project to retrieve.
I wanted to use tools.cmake in my project and use both cppcheck and cppcheck_reset as shown in the readme, however the reset function is not available in the last tag (i.e. 23.06).

Hi. Running CMake with the Xcode generator seems to fails the compilation for Address sanitizer on macOS:

❯ cmake .. -G "Xcode" -DUSE_SANITIZER=Address
-- The C compiler identification is AppleClang 13.1.6.13160021
-- The CXX compiler identification is AppleClang 13.1.6.13160021
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode13.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode13.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Testing with Address sanitizer
CMake Error at /Users/jedumas/external/git/cmake-scripts/sanitizers.cmake:77 (message):
  Address sanitizer not available for
  /Applications/Xcode13.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++
Call Stack (most recent call first):
  CMakeLists.txt:10 (include)


-- Configuring incomplete, errors occurred!

Furthermore it looks like the -fsanitize=address flag should also be passed to linker options. I would generally recommend setting global options via add_compile_options() and add_link_options() rather than mucking around with CMAKE_CXX_FLAGS or CMAKE_EXE_LINKER_FLAGS directly though, but that's probably a separate issue.

I'm using your code-coverage script and is working great! (than you)

I wanted to know if it is possible to pass arguments to the executable when running the coverage? I'm using this for test-coverage and I want to, for example, see the coverage of a specific test. I would like to do something like this

ccov-unit_tests -- catch2_args

Thanks!

Most of the ccov-all commands have an if(WIN32) section to handle command substitution of 'cat ...'. The ccov-all-export does not.
Below is a snippet that fixes that, could you incorporate in next release?

# Export coverage information so continuous integration tools (e.g.
      # Jenkins) can consume it
      if(WIN32)
        add_custom_target(
          ccov-all-export
          COMMAND
            powershell -Command $$FILELIST = Get-Content
            ${CMAKE_COVERAGE_OUTPUT_DIRECTORY}/binaries.list\; llvm-cov.exe
            export $$FILELIST
            -instr-profile=${CMAKE_COVERAGE_OUTPUT_DIRECTORY}/all-merged.profdata
            -format="text" ${EXCLUDE_REGEX} >
            ${CMAKE_COVERAGE_OUTPUT_DIRECTORY}/coverage.json
          DEPENDS ccov-all-processing)
      else()
        add_custom_target(
          ccov-all-export
          COMMAND
            ${LLVM_COV_PATH} export `cat
            ${CMAKE_COVERAGE_OUTPUT_DIRECTORY}/binaries.list`
            -instr-profile=${CMAKE_COVERAGE_OUTPUT_DIRECTORY}/all-merged.profdata
            -format="text" ${EXCLUDE_REGEX} >
            ${CMAKE_COVERAGE_OUTPUT_DIRECTORY}/coverage.json
          DEPENDS ccov-all-processing)
      endif()

I have another request.
Besides lcov we're also using gcovr for some of our projects and we have our own script that generates the coverage report. For those projects I'm still using your code-coverage.cmake module as basis w.r.t. the compiler settings and I just create a custom cmake tarket to produce the coverage report analog to ccov-capture-${target_code_coverage_COVERAGE_TARGET_NAME}.
Everything works fine, but I'd like to suppress the implicitly created ccov-* targets to keep the list of available targets clean. Can you think of a simple way to support that? Perhaps by offering a cmake option like the following?

option(
  DISABLE_COVERAGE_TOOLS
  "If ON then no targets are created for the built-in supported coverage tools."
  OFF)

How to reproduce:
git clone https://github.com/TheLartians/ModernCppStarter
cd ModernCppStarter
change symbol @ at cmake/tools.cmake:14 to #
cmake -S . -B ./build -DUSE_STATIC_ANALYZER=clang-tidy
should work just fine
change tag version in cmake/tools.cmake:14 from 23.04 to 24.04
get a next error:

|| CMake Error at /home/paxu/.cache/cpm/cmake-scripts/3a2aeaa8fc08089d36c491d5e0bfc2c2b3d9e1ef/tools.cmake:88 (message):
||   include-what-you-use not found!
|| Call Stack (most recent call first):
||   /home/paxu/code/uni/matvienko/lab2/mylab/cmake/tools.cmake:58 (include_what_you_use)
||   CMakeLists.txt:8 (include)
|| 
|| 
|| CMake Error at /home/paxu/.cache/cpm/cmake-scripts/3a2aeaa8fc08089d36c491d5e0bfc2c2b3d9e1ef/tools.cmake:136 (message):
||   cppcheck not found!
|| Call Stack (most recent call first):
||   /home/paxu/code/uni/matvienko/lab2/mylab/cmake/tools.cmake:59 (cppcheck)
||   CMakeLists.txt:8 (include)
|| 

Please, tell what changed, or tell how ModernCppStarter should be reworked

Trying to use this for templates header only library. It is defined as an interface but your gcov stuff is implemented with PRIVATE. I show the following modifications get me further but I still dont see results at the end of the build..

  set(options AUTO ALL EXTERNAL INTERFACE)

...

    elseif(CMAKE_COMPILER_IS_GNUCXX)
      if (NOT target_code_coverage_INTERFACE)
        target_compile_options(${TARGET_NAME} PRIVATE -fprofile-arcs
                                                      -ftest-coverage
                                                      -fno-inline 
                                                      -fno-inline-small-functions 
                                                      -fno-default-inline)
        target_link_libraries(${TARGET_NAME} PRIVATE gcov)
      else()
        target_compile_options(${TARGET_NAME} INTERFACE -fprofile-arcs
                                                        -ftest-coverage 
                                                        -fno-inline 
                                                        -fno-inline-small-functions 
                                                        -fno-default-inline)
        target_link_libraries(${TARGET_NAME} INTERFACE gcov)
      endif()
    endif()

Thoughts on the complete solution or should these changes do the trick? Thanks in advance.

To be clear, after the compile and run I am seeing the following warnings

geninfo: WARNING: cannot find an entry for #usr#include#c++#8#array.gcov in .gcno file, skipping file!
geninfo: WARNING: cannot find an entry for #usr#include#c++#8#bits#alloc_traits.h.gcov in .gcno file, skipping file!
geninfo: WARNING: cannot find an entry for #usr#include#c++#8#bits#allocator.h.gcov in .gcno file, skipping file!
geninfo: WARNING: cannot find an entry for #usr#include#c++#8#bits#basic_string.h.gcov in .gcno file, skipping file!
geninfo: WARNING: cannot find an entry for #usr#include#c++#8#bits#basic_string.tcc.gcov in .gcno file, skipping file!
geninfo: WARNING: cannot find an entry for #usr#include#c++#8#bits#char_traits.h.gcov in .gcno file, skipping file!
geninfo: WARNING: cannot find an entry for #usr#include#c++#8#bits#cpp_type_traits.h.gcov in .gcno file, skipping file!
geninfo: WARNING: cannot find an entry for #usr#include#c++#8#bits#functional_hash.h.gcov in .gcno file, skipping file!
geninfo: WARNING: cannot find an entry for #usr#include#c++#8#bits#hashtable.h.gcov in .gcno file, skipping file!
geninfo: WARNING: cannot find an entry for #usr#include#c++#8#bits#hashtable_policy.h.gcov in .gcno file, skipping file!
geninfo: WARNING: cannot find an entry for #usr#include#c++#8#bits#ios_base.h.gcov in .gcno file, skipping file!
geninfo: WARNING: cannot find an entry for #usr#include#c++#8#bits#move.h.gcov in .gcno file, skipping file!
geninfo: WARNING: cannot find an entry for #usr#include#c++#8#bits#predefined_ops.h.gcov in .gcno file, skipping file!
geninfo: WARNING: cannot find an entry for #usr#include#c++#8#bits#ptr_traits.h.gcov in .gcno file, skipping file!

Is it possible to have code coverage reported on all targets? For example, if I run an executable with coverage I've noticed that only the compilation units in that executable get included in the coverage report. Instead, I'd like all file in my project to have 0% coverage by default, and then running the executable updates that baseline.

Is this possible and I am doing something wrong, or is this not supported?

The macro clang_tidy uses ${ARGN} variable whereas remaining include_what_you_use and cppcheck stil relay on xxx_STRING which honestly I don't know where it comes from. I've checked doc of cmake back to 3.0 and there is no mentioning of xxx_STRING. Where does it come from?

Please fix lines 32 and 40 in tools.cmake.

All the tools in tools.cmake are currently applied to only C++ code, however they all support C code as well. Supporting C should just be a fairly straightforward -- for example setting the CMAKE_C_CLANG_TIDY to match CMAKE_CXX_CLANG_TIDY

I am on Ubuntu 20.04 and this happens when I try to run:

cmake -GNinja ... -DCMAKE_BUILD_TYPE=Debug -Stest -Bbuild(test

CMake Error at .../build/test/_deps/cmake-scripts-src/code-coverage.cmake:253 (target_link_libraries):
  The plain signature for target_link_libraries has already been used with
  the target "GreeterTests".  All uses of target_link_libraries with a target
  must be either all-keyword or all-plain.

  The uses of the plain signature are here:

   * CMakeLists.txt:42 (target_link_libraries)

Call Stack (most recent call first):
  CMakeLists.txt:68 (target_code_coverage)

The relevant CMake code is:

target_code_coverage(Greeter)
target_code_coverage(GreeterTests AUTO)

It works fine with llvm-cov so there's something specific about lcov I would assume.

See https://github.com/Krzmbrzl/cmake-compiler-flags
This is a project I have started recently to implement common compiler flags in a compiler-independent way. Essentially what the compiler-flags module in this repo does as well.
Therefore, I would suggest merging my project into this one.

I'll drop this idea here. If I find the time, I'll do it myself, but until then anyone is free to do it on my behalf :)

Currently the added ccov targets execute the built target with optional parameters. Unfortunately, that doesn't work when the binary is cross-compiled for a different architecture (like arm). In that case the binary has to be executed by an emulator like qemu, e.g. /usr/bin/qemu-aarch64-static -L /usr/aarch64-linux-gnu -E LD_LIBRARY_PATH=/some/path ./test-binary

Is there a way to add support for cross-compiled targets, perhaps by adding an additional optional parameter to provide an emulator?

If that's not possible then alternatively it would be nice to have an option to to suppress the generation of the ccov targets in order to avoid having non-functional targets.

Hello,

Could you please start releasing process so that I can use specific version in the git submodule definition.

Thank you

I am using CPM for managing dependencies in my project and since it is source-based (downloads the source and then builds it), llvm-cov finds and includes it in the code coverage report.

I am using the caching feature of CPM so everything is in a .cache/CPM directory.

What do you suggest I do?

First off thanks for your code-coverage.cmake! We have been using it successfully.

Our application is structured as a "core library" with our normal executable and a Catch2 test harness executable

                 /----> app
                /
libapp-core.a -+
                \
                 \----> app-test

This works great, however we need to ensure that all of libapp-core.a gets linked into the test executable, otherwise our coverage stats are only for the code that has tests written, rather than all of the objects in libapp-core. So we use -force_load

include( code-coverage )
target_code_coverage( app-core )
target_code_coverage( app-test EXCLUDE lib/* test/* "${CMAKE_BINARY_DIR}/*")
target_link_options( app-test PRIVATE -force_load $<TARGET_FILE:app-core> )

In practice, when we run this on our CI system, the linker step can take a very long time to execute with this flag, resulting in timeouts.

Do you have any idea on how we might generate coverage for the entire static library without incurring such a penalty on the linker?

Currently on Xcode Apple LLVM version 14.0.3 (clang-1403.0.22.14.1) but this has been an issue on an off for several years

Ihave a very simple project with some dummy codes. after I integrate with codecov, it showed me the wrong rate.

[build] [----------] Global test environment tear-down
[build] [==========] 3 tests from 1 test suite ran. (0 ms total)
[build] [  PASSED  ] 3 tests.
[build] [11/12  91% :: 7.179] cmd.exe /C "cd /D D:\VS2019\quicplus\build && powershell -Command $FILELIST = Get-Content D:/VS2019/quicplus/build/ccov/profraw.list; llvm-profdata.exe merge -o D:/VS2019/quicplus/build/ccov/all-merged.profdata -sparse $FILELIST"
[build] [12/12 100% :: 7.814] cmd.exe /C "cd /D D:\VS2019\quicplus\build && powershell -Command $FILELIST = Get-Content D:/VS2019/quicplus/build/ccov/binaries.list; llvm-cov.exe report $FILELIST -instr-profile=D:/VS2019/quicplus/build/ccov/all-merged.profdata"
[build] Filename                      Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
[build] -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[build] conn.cpp                            2                 0   100.00%           2                 0   100.00%           6                 0   100.00%           0                 0         -
[build] tls.cpp                            12                 3    75.00%           3                 0   100.00%          11                 0   100.00%           0                 0         -
[build] -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[build] TOTAL                              14                 3    78.57%           5                 0   100.00%          17                 0   100.00%           0                 0         -
[cmakefileapi-parser] Read reply folder: d:\VS2019\quicplus\build\.cmake\api\v1\reply
[cmakefileapi-parser] Found index files: ["cache-v2-590d8940190b756e4d81.json","codemodel-v2
[extension] [5417] cmake.build finished (returned 0)   

conn.cpp has a dummy method:

int conn::Conn::dummy()
{
  int b = ret * 2;
  return b;
}

however, it is not tested at all in all my tests:

#include <gtest/gtest.h>

#include <quic_plus.hpp>

// NOLINTNEXTLINE
TEST(Foo, Test1)
{
  std::string hello("hellojhake");
}

// NOLINTNEXTLINE
TEST(Foo, Test2)
{
  std::string hello("hellojhake");
}

but the results still showed me 100% and 75% cover range.

anything I did wrong ? thx.

When using sanitizer.cmake with find_package (MPI REQUIRED) on a simple ubuntu environment with mpich (a docker image in reality), MPI is not found. Removing PARENT SCOPE in the definition of append solves this issue. Any idea why ?

Hey again, just thought I'd open this so there's an active issue to refer to in regards to this.

Unfortunately, the ccov-all* targets are still non-functional on Windows, and I'm still trying to figure out a solution.

from here

Hello,

From cmake-format release note:

The new --check command line option will not format the file, but exit with non-zero status if any changes would be made

I would like suggest to add that check at CI stage as well as provide used .cmake-format configuration.

error: /home/runner/work/quicplus/quicplus/build/bin/test_conn.exe, /home/runner/work/quicplus/quicplus/build/bin/test_version.exe, /home/runner/work/quicplus/quicplus/build/bin/test_quic_plus.exe: Failed to load coverage: Unsupported coverage format version
ninja: build stopped: subcommand failed.
Error: Process completed with exit code 1.

you can go to seethe workflow logs directly from my project.
https://github.com/Kiddinglife/quicplus/runs/3503532396?check_suite_focus=true

First of all, thanks for publishing this work. The code coverage library in this repo saved me a lot of trouble.

I have a large project with many, many test executables. These test executables cover code for many static library targets. Some of these static library targets have many test executables.

Then the main build artifact is an executable that links with all of these static libraries.

So the way I structured this using your library was to:

include(code-coverage)
add_code_coverage_all_targets()

And then for every library and test executable, target_code_coverage(target AUTO ALL EXCLUDE ${test_cpp_files}) is invoked.

This works fine for the report target of each of those separate library or test executable targets, but it doesn't work for the ccov-all* targets, since those exclusions aren't propagated to top-level targets.

I have a really gross patch to fix this in my project, but I thought that I'd call this out here.

https://gist.github.com/ambroff/64e9e9cc7ac1000366c05e5ed59cca95

This patch works fine for my use case but I don't think it's general enough, since I'm using this to generate a giant regular expression that matches any of the test source files provided.