r-lib / covr

Test coverage reports for R

covr

Track test coverage for your R package and view reports locally or (optionally) upload the results to codecov or coveralls.

Installation

install.packages("covr")

# For devel version
devtools::install_github("r-lib/covr")

The easiest way to setup covr on Github Actions is with usethis.

usethis::use_github_action("test-coverage")

Usage

For local development a coverage report can be used to inspect coverage for each line in your package. Note requires the DT package to be installed.

library(covr)

# If run with no arguments implicitly calls `package_coverage()`
report()

covr also defines an RStudio Addin, which runs report() on the active project. This can be used via the Addin menu or by binding the action to a shortcut, e.g. Ctrl-Shift-C.

Interactively

# If run with the working directory within the package source.
package_coverage()

# or a package in another directory
cov <- package_coverage("/dir/lintr")

# view results as a data.frame
as.data.frame(cov)

# zero_coverage() shows only uncovered lines.
# If run within RStudio, `zero_coverage()` will open a marker pane with the
# uncovered lines.
zero_coverage(cov)

Exclusions

covr supports a few of different ways of excluding some or all of a file.

.covrignore file

A .covrignore file located in your package's root directory can be used to exclude files or directories.

The lines in the .covrignore file are interpreted as a list of file globs to ignore. It uses the globbing rules in Sys.glob(). Any directories listed will ignore all the files in the directory.

Alternative locations for the file can be set by the environment variable COVR_COVRIGNORE or the R option covr.covrignore.

The .covrignore file should be added to your .RBuildignore file unless you want to distribute it with your package. If so it can be added to inst/.covrignore instead.

Function Exclusions

The function_exclusions argument to package_coverage() can be used to exclude functions by name. This argument takes a vector of regular expressions matching functions to exclude.

# exclude print functions
package_coverage(function_exclusions = "print\\.")

# exclude `.onLoad` function
package_coverage(function_exclusions = "\\.onLoad")

Line Exclusions

The line_exclusions argument to package_coverage() can be used to exclude some or all of a file. This argument takes a list of filenames or named ranges to exclude.

# exclude whole file of R/test.R
package_coverage(line_exclusions = "R/test.R")

# exclude lines 1 to 10 and 15 from R/test.R
package_coverage(line_exclusions = list("R/test.R" = c(1:10, 15)))

# exclude lines 1 to 10 from R/test.R, all of R/test2.R
package_coverage(line_exclusions = list("R/test.R" = c(1, 10), "R/test2.R"))

Exclusion Comments

In addition you can exclude lines from the coverage by putting special comments in your source code.

This can be done per line.

f1 <- function(x) {
  x + 1 # nocov
}

Or by specifying a range with a start and end.

f2 <- function(x) { # nocov start
  x + 2
} # nocov end

The patterns used can be specified by setting the global options covr.exclude_pattern, covr.exclude_start, covr.exclude_end.

NB: The same pattern applies to exclusions in the src folder, so skipped lines in, e.g., C code (where comments can start with //) should look like // # nocov.

FAQ

Will covr work with testthat, RUnit, etc...

Covr should be compatible with any testing package, it uses tools::testInstalledPackage() to run your packages tests.

Will covr work with alternative compilers such as ICC

Covr now supports Intel's icc compiler, thanks to work contributed by Qin Wang at Oracle.

Covr is known to work with clang versions 3.5+ and gcc version 4.2+.

If the appropriate gcov version is not on your path you can set the appropriate location with the covr.gcov options. If you set this path to "" it will turn off coverage of compiled code.

options(covr.gcov = "path/to/gcov")

How does covr work?

covr tracks test coverage by modifying a package's code to add tracking calls to each call.

The vignette vignettes/how_it_works.Rmd contains a detailed explanation of the technique and the rationale behind it.

You can view the vignette from within R using

vignette("how_it_works", package = "covr")

Why can't covr run during R CMD check

Because covr modifies the package code it is possible there are unknown edge cases where that modification affects the output. In addition when tracking coverage for compiled code covr compiles the package without optimization, which can modify behavior (usually due to package bugs which are masked with higher optimization levels).

Alternative Coverage Tools

• https://github.com/MangoTheCat/testCoverage (no longer supported)
• R-coverage (no longer supported)

Code of Conduct

Please note that the covr project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

covr's Issues

single line function without braces is not detected

Love this tool, but when I have a single line function without braces, e.g.

f <- function(x)
  x

the coverage tool neither detects it is covered nor uncovered by tests. (Or at least coveralls.io doesn't display it.)

Compiled code coverage not working completely on windows

Seems to be a handful of issues

• paths passed to subproccess need to use forward slashes
• does the gcc version with RTools support gcov?
• if it does support gcov are the output files in a different place than on linux/osx?

Handle primitive functions and functions with a NULL environment

as of #47 these are being skipped, but it would be better track these functions as well.

coverall Error: No description at /home/travis/build/$USER/DESCRIPTION

We get the following error after successful build with travis:

Error: No description at /home/travis/build/stnava/DESCRIPTION

The complete travis log file, associated with this error is:

* help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (ANTsR)

The command "R CMD INSTALL ANTsR" exited with 0.
$ Rscript -e 'library(covr);coveralls()'
Loading required package: methods
Error: No description at /home/travis/build/stnava/DESCRIPTION
Execution halted

Done. Your build exited with 0.

This is our .travis.yml here. I would appreciate any comments on this.

PS The coverage after this error remains "unknown"

summarize test coverage without coveralls?

Cool package! I noticed that if you run the package_coverage function, it gives you a ton of output, that I'm guessing gets uploaded to the coveralls service to give you a badge. However, when I'm trying to improve test coverage, I would want to be able to just re-run package_coverage as I add tests and see how the coverage changes.

Is there a way to generate a % coverage value from the package_coverage output? I'd be willing to write a summarize method if I knew how to interpret the values that are being returned.

Handle RC classes (might not need to do anything but add tests)

Coverage of examples

Many packages don't have any tests, but often still have examples. Would be useful to see what code is covered by examples at least.

aggregate coverage by function

Test coverage even if some tests fail

Right now if any of your tests fail you don't get any test coverage reported.

Maybe it would be better to wrap in a try block.

display coverage results using Rstudio marker API

Ala r-lib/lintr#37

Would be nice to report only the blank lines

coveralls does not work correctly if the path is not '.'

Mostly an issue with testing, you probably wouldn't call it with any other argument in practice.

Coverage of functions in base, stats packages

I am working on a tools for creating test cases for different VM implementations of R and I was thinking if it's possible to measure coverage of functions from default packages with covr.

For example,

function_coverage("colMeans", colSums(UCBAdmissions), env=getNamespace("base"))

or

ctl <- c(4.17,5.58,5.18,6.11,4.50,4.61,5.17,4.53,5.33,5.14)
trt <- c(4.81,4.17,4.41,3.59,5.87,3.83,6.03,4.89,4.32,4.69)
group <- gl(2, 10, 20, labels = c("Ctl","Trt"))
weight <- c(ctl, trt)
lm.D9 <- lm(weight ~ group)
function_coverage("lm", lm(weight ~ group), env=getNamespace("stats"))

But now, it seems that trace_calls does not insert any trace points into those functions. How can it be changed?

Have package with some zero lines

Add a test package that has some zero lines of coverage to properly test the percent_coverage and zero_coverage functions

Handle S4 classes and methods properly

Make a note about cross-compatability to the readme.

@wrathematics mentioned that covr fails with clang 3.4 and icc (due to gcov issues). This should be noted, and if possible a minimum version of gcc (which one TBD) should be added as well.

Shiny app to view coverage results.

Ideal features would be

Must haves

• select multiple files
• highlight lines that are not covered

Like to haves

• Select files using file system tree
• Count of line coverage for each covered line
• Toggle between test/example/vignette coverage

Does not need to track coverage over time as that is what coveralls.io is mainly for.

test_coverage function

1. Load all functions with traced calls
2. Run
   1. tests (like CRAN would, should support at least testthat and RUnit, preferably everything)
   2. examples
3. Return test coverage per line for each file

Error: In file(con, "r") : cannot open file '<text>': Invalid argument

If a package uses tools(*) that "dynamically" create functions based on eval(parse(text=...)) during build, then such functions will have a srcfile attribute that is "<text>". For example:

> eval(parse(text='foo <- function(...) cat("foo()\n")'))
> attr(attr(foo, "srcref"), "srcfile")
<text>

This will cause covr::coveralls() to throw an error:

> covr::coveralls()
Error in file(con, "r") : cannot open the connection
In addition: Warning message:
In file(con, "r") : cannot open file '<text>': Invalid argument

6: file(con, "r")
5: FUN(c("<text>")[[1L]], ...)
4: lapply(filenames, readLines)
3: per_line(x)
2: to_coveralls(package_coverage(path, relative_path = TRUE, ...))
1: covr::coveralls()

One quick fix is to have covr::key() do gsub("<text>", ".text", ...) such that they file name, which base on this key, is a valid file name.

Footnotes:
(*) Such tools are setGenericS3() and setMethodS3() in R.methodsS3. Yep, that's my package; I could maybe update those functions to set the srcfile attribute such that it points to some kind of source file. Regardless, there may be other code out there that relies on eval(parse(text=...)) for creating functions.

Files in subdirectories of src cause errors again

After issue #33 was fixed, covr correctly worked with source files in subdirectories of src, but there seems to be another problem in the current version. After updating I see this when I try to check coverage in package ore:

> library(covr)
> package_coverage()  # from "ore" package root directory
Regular expression compilation : .....
Matching and related operations : .................
Compatibility with rex : .

Error in file(con, "r") : cannot open the connection
In addition: Warning message:
In file(con, "r") :
  cannot open file './src/iso8859_1.c': No such file or directory

The correct path is actually ./src/onig/enc/iso8859_1.c. If it's relevant, Makevars says

OBJECTS_ENC = onig/enc/ascii.o onig/enc/iso8859_1.o onig/enc/unicode.o onig/enc/utf8.o

OBJECTS_ONIG = onig/regcomp.o onig/regenc.o onig/regerror.o onig/regexec.o onig/regext.o onig/reggnu.o onig/regparse.o onig/regposerr.o onig/regposix.o onig/regsyntax.o onig/regtrav.o onig/regversion.o onig/st.o

OBJECTS = main.o $(OBJECTS_ONIG) $(OBJECTS_ENC)

PKG_CPPFLAGS = -Ionig

Integrate with Travis and Coveralls

Blank lines within functions should not be included in coverage results.

Ex. https://coveralls.io/files/410126873#L68

Need to tweak the logic to check if a given line is blank.

compiled code coverage is not displaying in travis or coveralls

I am not sure if the issue is in travis not running gcov properly or an issue with the json to coveralls, but digest contains mostly c code and it is not being reported. https://coveralls.io/jobs/3980390

Actually there is a bug locally as well, the source coverage only works the second time you run it.

Error in aggregate.data.frame

This might very well be operator error, but ...

ff = function(x) identity(x)
function_coverage("ff", ff(1))

Results in

Error in aggregate.data.frame(mf[1L], mf[-1L], FUN = FUN, ...) : 
  no rows to aggregate

What am I missing? Thanks

Missing .gcov files

Hi Jim,

I've tried out the latest version of covr on my ore package, and I see

> package_coverage()
Error in file(con, "r") : cannot open the connection
In addition: Warning message:
In file(con, "r") :
  cannot open file '/Users/jon/Development/Git/ore/src/onig/enc/ascii.c.gcov': No such file or directory
> traceback()
7: file(con, "r")
6: readLines(file)
5: parse_gcov(paste0(file, ".gcov"))
4: FUN(c("/Users/jon/Development/Git/ore/src/main.c", 
<snip>
   "/Users/jon/Development/Git/ore/src/onig/regversion.c", "/Users/jon/Development/Git/ore/src/onig/st.c"
   )[[2L]], ...)
3: lapply(sources, run_gcov)
2: unlist(lapply(sources, run_gcov))
1: package_coverage()

I don't think this is related to #24, even though the error is superficially similar. I see .gcda and .gcno files in the hierarchy, but no .gcov files. This is OS X 10.10.1, with self-built R 3.1.2.

Apologies if I've missed something obvious.

zero_coverage should return the start and end columns as well.

Maybe use `testthat:::reassign_function` rather than regular R assignment.

It would be a more complete solution because it actually changes the function pointer and would hopefully fix some edge cases.

Repeated rows in zero_coverage output

Hi,
I was wondering what's the meaning of repeating rows

The package is RevolutionAnalytics/quickcheck/pkg

pc = package_coverage("pkg/")
zero_coverage(pc)
1  R/quickcheck.R         39        39     0
2  R/quickcheck.R         40        41     0
3  R/quickcheck.R         41        41     0
4  R/quickcheck.R         41        41     0
5  R/quickcheck.R         41        41     0
6  R/quickcheck.R         49        50     0
7  R/quickcheck.R         53        54     0
8  R/quickcheck.R         83        83     0
9  R/quickcheck.R         83        83     0

thanks

Ideas from other coverage tools

• https://metacpan.org/pod/Devel::Cover
• https://github.com/z4r/python-coveralls
• https://github.com/eddyxu/cpp-coveralls
  • might be able to use this for the Rcpp/C code in R packages
• https://gcc.gnu.org/onlinedocs/gcc/Gcov.html
• https://github.com/integrum/SublimeRubyCoverage

Errors in tests lead to under-reporting of coverage

At present, if any of the tests produce an error, covr seems to simply silently ignore it, and report coverage without the test in question, or any subsequent ones. It seems like it would be preferable to at least indicate that an error occurred, rather than leaving the user confused as to why the coverage is so low.

set_makevars()/reset_makevars() are not parallel safe

If multiple covr sessions are used in parallel (which may not be unlikely on server farms), there is a risk that '~/.R/Makevars' is being rewritten while another R session is reading it leading to a potentially corrupt read.

One way to lower the risk for this is to have set_makevars() only modify '~/.R/Makevars' iff the content actually changes, i.e. only call writeLines(..., lines) if lines have changed from the originally read one.

This solution does not fully cover the case when a pre-existing '~/.R/Makevars' is changed, because then reset_makevars() in one session will undo what another session might have written. However, by only rewriting/undoing the file if something changes the risk for incorrect content is lowered.

The long-term solution is probably for R to provide a command-line option/system environment variable where one can specify a session-specific 'Makevars' file path.

Error in seq.default

I've just faced the below strange error:

$ git clone https://github.com/Rapporter/pander.git
$ R 
> library(covr)
> package_coverage('pander')

Error in seq.default(df[i, "first_line"], df[i, "last_line"]) : 
  'from' cannot be NA, NaN or infinite
> traceback()
9: stop("'from' cannot be NA, NaN or infinite")
8: seq.default(df[i, "first_line"], df[i, "last_line"])
7: seq(df[i, "first_line"], df[i, "last_line"])
6: match(x, table, nomatch = 0L)
5: seq(df[i, "first_line"], df[i, "last_line"]) %in% excl[[file]]
4: FUN(1:1553[[1552L]], ...)
3: vapply(seq_len(NROW(df)), function(i) {
       file <- df[i, "filename"]
       file %in% names(excl) && all(seq(df[i, "first_line"], df[i, 
           "last_line"]) %in% excl[[file]])
   }, logical(1))
2: exclude(coverage, exclusions = exclusions, exclude_pattern = exclude_pattern, 
       exclude_start = exclude_start, exclude_end = exclude_end)
1: package_coverage()

I will investigate further later, just wanted to make a note of it for the time being.

---

Related session info:

> sessionInfo()
R version 3.1.2 (2014-10-31)
Platform: x86_64-unknown-linux-gnu (64-bit)

locale:
 [1] LC_CTYPE=hu_HU.utf8       LC_NUMERIC=C             
 [3] LC_TIME=en_US.UTF-8       LC_COLLATE=en_US.UTF-8   
 [5] LC_MONETARY=en_US.UTF-8   LC_MESSAGES=hu_HU.utf8   
 [7] LC_PAPER=hu_HU.utf8       LC_NAME=C                
 [9] LC_ADDRESS=C              LC_TELEPHONE=C           
[11] LC_MEASUREMENT=hu_HU.utf8 LC_IDENTIFICATION=C      

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] covr_0.2.0.9000 pander_0.5.2   

loaded via a namespace (and not attached):
[1] devtools_1.7.0  digest_0.6.8    lazyeval_0.1.10 magrittr_1.5   
[5] Rcpp_0.11.4     rex_0.2.0       tools_3.1.2  

$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/lto-wrapper
Target: x86_64-unknown-linux-gnu
Configured with: /build/gcc-multilib/src/gcc-4.9-20141224/configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=https://bugs.archlinux.org/ --enable-languages=c,c++,ada,fortran,go,lto,objc,obj-c++ --enable-shared --enable-threads=posix --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-clocale=gnu --disable-libstdcxx-pch --disable-libssp --enable-gnu-unique-object --enable-linker-build-id --enable-cloog-backend=isl --enable-lto --enable-plugin --enable-install-libiberty --with-linker-hash-style=gnu --enable-multilib --disable-werror --enable-checking=release
Thread model: posix
gcc version 4.9.2 20141224 (prerelease) (GCC) 

handling tests skipped by testthat::skip ?

I think covr should have the option of not reporting test coverage if any test has been skipped by testthat::skip.

I tend to use the skip function only when a test depends on some external web resource that might not be available. testthat:skip allows intermittent failures to reach that external web resource without the red warning signs of a failed test. If the tests are being skipped on a particular build because the resource was unavailable, it does not seem reasonable for the coverage to drop instead of just remaining at what it's last value was.

Not actually sure how to go about this. Would have to dig into @hadley's testthat code to see if skip is giving any other programmatic indication that a test has been skipped, other than the visual printing S instead of . where the test has would have been run in the testthat summary.

vignette explaining how covr works

inst support?

Seems it doesn't pick up on stuff in inst/tests. All of my packages have tests there per an old comment from Hadley, but this convention might have been deprecated. Should I move all my tests for all of my packages to tests off the package root?

Allow specifying what files to run to test.

To handle weird things like data.table's testing situation.

Coverage on .onAttach(), .onLoad(), .onUnload()?

Is it possible to get coverage on .onLoad() and similar functions? I've tried, but get zero coverage for all those.

EDIT: I eventually got .onUnload() covered (https://coveralls.io/files/438500374), but I do get zero coverage on .onAttach().

README.md should show example of local usage

Rcpp/fortran support?

Is there any chance you might be able to combine covr with an existing C++ code coverage tool in order to also give C++ code coverage stats?

change R version in Depends to R 3.1.0

Does the package actually depend on having R 3.1.2? I was able to clone, change the Depends field in DESCRIPTION to R 3.1.0, and install on my system with R 3.1.1, and run it without difficulty.

lines containing only a symbol without a call are ignored

function(x) {
  x
}

These lines do not have a srcref apart from their enclosing function. Distinguishing them from symbols in a regular call would be somewhat tricky as well, and what increase the number of trace calls with little gain (IMHO).

However this situation should be documented (probably in the README)

Coverage of vignettes

Along with #51, it would be handy to report of coverage of vignettes separately as well.

support BiocGenerics:::testPackage

It differs from testthat::test_check in that the tests are run from the installed directory always.

I need to ensure that the tests are calling the new functions, or modify the installed namespace functions at runtime.

Functions calculated as closures are not covered

From my package, it seems that package functions constructed by an internal meta-function is not traced. For example,

function_builder <- function(x) {
  function(a) {
    x + a
  }
}}

#' @export
useful_function1 <- function_builder(1)
#' @export
useful_function2 <- function_builder(2)

Even if useful_function1 and useful_function2 are both called in tests, the closure function(a) { x + a } is always reported not covered.

Example is here:
The code in red https://coveralls.io/builds/1833742/source?filename=.%2FR%2Fpipe.R#L183 is actually called test https://github.com/renkun-ken/pipeR/blob/master/tests/testthat/test2-Pipe.R#L64

Race condition when pushing before coverage has been reported

See the comments by Coveralls in https://github.com/grake/grake/pull/4. The increase in coverage for c59c498 has been reported twice, while that for 3480d2c, the previous commit, hasn't been reported at all. I haven't investigated this further.

Time for releases?

Hi,
I am about to release quickcheck 3.1.0. It's a testing package and has covr as an optional dep. For lack of releases, I am pointing my users to commit b181831 because I tested on it, but I am sure you are not looking forward to bugs being reported referencing a random commit in the past. So if you planned to start doing releases, here is a vote to get that going sooner rather than later. Thanks

Exclude lines from coverage

It is probably beneficial for a package developer to exclude lines from their coverage. I could see two interfaces for this

1. as a parameter to the given coverage function
   • named vector with filenames c(src/test.R = c(2, 3, 5, 6, 10:20))
2. as an annotation within the source file
   • # no_covr
   • # no_covr_start
   • # no_covr_end

They both have positive and negatives in my mind, perhaps the best solution is to use both...

coverage depends on redundant brackets

You may have mentioned this before but it is quite disturbing that when writing ifs without optional brackets coverage is overestimated. Is this a bug? Thanks

> ff
function(x = T) {
    if(x) {
      1}
    else{
      0}
  }
> cov = function_coverage("ff", ff())
> names(cov) <- paste0("<file>", names(cov))
> cov
Package Coverage: 66.67%
<file>: 66.67%
> ff = 
+   function(x = T) {
+     if(x) 
+       1
+     else
+       0
+   }
> cov = function_coverage("ff", ff())
> names(cov) <- paste0("<file>", names(cov))
> cov
Package Coverage: 100.00%
<file>: 100.00%

Coverage fails on compiled code coverage.

I'm working on a package with compiled code. It builds fine, but fails when I run:

> Rscript -e "library(covr);package_coverage()"
Loading required package: methods
rfsrc tests : 

Unexpected version: *904.
Invalid .gcno File!
.
.
.
Unexpected version: *904.
Invalid .gcno File!
Error in names(res) <- file.path(src_path, names(res)) : 
  attempt to set an attribute on NULL
Calls: package_coverage -> run_gcov
Execution halted

This happens locally on OS X.

It fails on the TravisCI Linux server as well:

Error in seq.default(df[i, "first_line"], df[i, "last_line"]) : 
  'from' cannot be NA, NaN or infinite
Calls: coveralls -> to_coveralls -> per_line -> seq -> seq.default
Execution halted

More robust if set_makevars()/reset_makevars() use file.rename()

If '/.R/Makevars' exists, then set_makevars() rewrites the file and reset_makevars() undoes this rewrite. Although package_coverage() uses on.exit() to lower the risk for failing to undo the rewrite, there is still a risk that the R session is aborted corrupting the preexisting '/.R/Makevars' file. Also, it will mess with the file time stamp in either case.

SUGGESTION:
I would like to propose to have set_makevars() to rename any existing file to, say, '~/.R/Makevars.original.covr' and then have reset_makevars() to rename it back. This way the original file is still there in case of a core dump or similar, and its time stamp should be preserved.

Conditional statements (if) reported as covered even if not run

Why is do_something() shown as covered in the following example:

if (FALSE)
  do_something();

Real-world example: For the linked file there is only one test for which I'm pretty sure that is.matrix(X) is TRUE and is.complex(X) is FALSE. Also, the function doesn't end prematurely -- however, lines 33, 36 and 38 are indicated as covered.