rillig / gobco Goto Github PK

Measure branch coverage of golang tests

Go 99.68% Shell 0.32%

coverage-testing coverage golang go

gobco's Introduction

GOBCO - Golang Branch Coverage

Gobco measures condition coverage of Go code.

Gobco is intended to be used in addition to go test -cover. For example, gobco does not detect functions or methods that are completely unused, it only notices them if they contain any conditions or branches. Gobco also doesn't cover select statements.

Installation

With go1.17 or later:

$ go install github.com/rillig/gobco@latest

With go1.16:

$ go get github.com/rillig/gobco

Older go releases are not supported.

Usage

To run gobco on a single package, run it in the package directory:

$ gobco

The output typically looks like the following example, taken from package github.com/rillig/pkglint:

ok  	github.com/rillig/pkglint/v23	29.648s

Condition coverage: 8720/8840
...
changes.go:171:61: condition "n == 6" was 12 times true but never false
distinfo.go:268:8: condition "alg == \"SHA1\"" was 16 times false but never true
distinfo.go:322:13: condition "remainingHashes[0].algorithm == alg" was 8 times true but never false
...
mkcondsimplifier.go:141:31: condition "p[2] == p[1]-'A'+'a'" was 26 times true but never false
...
vartypecheck.go:1027:11: condition "len(invalid) > 1" was once false but never true
vartypecheck.go:1628:42: condition "cv.MkLines.pkg != nil" was 8 times true but never false
vartypecheck.go:1630:6: condition "distname.IsConstant()" was 8 times true but never false

Adding custom test conditions

If you want to ensure that the tests cover a certain condition in your code, you can insert the desired condition into the code and assign it to the underscore:

func square(x int) int {
    _ = x > 50
    _ = x == 0
    _ = x < 0

    return x * x
}

The compiler will see that these conditions are side-effect-free and will thus optimize them away, so there is no runtime overhead.

Gobco only inserts its coverage code around expressions that are syntactically recognizable as boolean expressions, such as comparisons, '&&', '||', '!'. When a boolean expression is merely passed around, there is no branch involved, thus nothing to do for branch coverage.

gobco's People

Contributors

Stargazers

Watchers

Forkers

myzhan wveitch anmolbabu adiping1 edv1n jopinzhangji tourer-zc zzctmac mkeeler visu-suganya 1st-vil mf-ravioli moneyforward

gobco's Issues

Allow for user defined TestMain()

If a user defines their own TestMain then gobco fails with:

multiple definitions of TestMain

I notice that writeGobcoTestGo() creates a gobco_test.go whose sole purpose is to add the running of gobcoPrintCoverage(@listAll@)

I think this is okay but some allowance is needed for the case where a user defines their own test main. Perhaps we require a go:generate to generate the call to gobcoPrintCoverage() in there test main and/or have a build flag (e.g. / +build gobco) to distinguish running under gobco from normal go test as both should work.

For example if I have:

package main

import (
	"flag"
	"os"
	"testing"
)

func TestMain(m *testing.M) {
	CmdLineVerbose = flag.Bool("verbose", false, "verbosity of tests")
	flag.Parse()
	os.Exit(m.Run())
}

As a workaround I could have my TestMain in a file by itself (main_test.go) as in:


// +build !gobco

package main

import (
	"flag"
	"os"
	"testing"
)

func TestMain(m *testing.M) {
	flag.Parse()
	os.Exit(m.Run())
}

With a separate file (e.g. flags_test.go) for things which would otherwise be in TestMain() for example:

package main

import (
	"flag"
)

var CmdLineVerbose *bool

func init() {
	CmdLineVerbose = flag.Bool("verbose", false, "verbosity of tests")
}

and then run gobco -tags gobco

I don't find this solution particularly elegant.
This might break for a more complicated TestMain()

An alternative might be for gobco to have an option to not generate its own TestMain but instead to provide a hook via //go:generate or something similar.

Problems with assigments inside switch expressions

I tried with the switch example in A Tour of Go. It seems that the assigment inside the switch is cousing problems (I tried moving it outside and it works).

	switch os := runtime.GOOS; os {
	case "darwin":
		fmt.Println("OS X.")

The output is:

PS C:\hecho\go\switch> gobco
# example/switch [example/switch.test]
.\switch.go:10:37: undefined: os      
FAIL    example/switch [build failed]
FAIL
exit status 2open C:\Users\abcd\AppData\Local\Temp\gobco-6ee1adac66cac296\gobco-counts.json:

Complete code:

package main

import (
	"fmt"
	"runtime"
)

func main() {
	fmt.Print("Go runs on ")
	switch os := runtime.GOOS; os {
	case "darwin":
		fmt.Println("OS X.")
	case "linux":
		fmt.Println("Linux.")
	default:
		// freebsd, openbsd,
		// plan9, windows...
		fmt.Printf("%s.\n", os)
	}
}

fmt.Sprint(func) does not compile

./packrat.go:361:43: fmt.Sprint arg List is a func value, not called
./printer.go:66:23: fmt.Sprint arg v is a func value, not called
./printer.go:66:40: fmt.Sprint arg List is a func value, not called
./printer.go:142:119: fmt.Sprint arg List is a func value, not called
./printer.go:165:25: fmt.Sprint arg v is a func value, not called

The thing is, the code is correct and should compile (and does compile on go 1.22.0). The result of fmt.Sprint(f) where f is a func() is a hex value identifying the closure like in:

                // tell if v is the "List" function
                if fmt.Sprint(v) == fmt.Sprint(List) { //fmt.Sprint
                        return "list"
                }

Feature Request: Measure coverage of higher level tests

I have tests at multiple levels.
unit tests in _test.go manner
program tests which run a copy of the program itself

I place these in suite in program_test.go and they are run via go test as normal
though this is not the only way.

In the unit tests a lot of important functionality is mocked leading to low apparent test coverage.
The added coverage from program tests is not included in the output of go test -c or gobco.

This could be supported if an instrumented version of the program could be generated so that program level integration tests generate statistics that can be included in the final coverage report.
To do this we would need to:

generate an instrumented version of the program (similar to -ftest-coverage for gcov for C/c++)
record the results somewhere (e.g. to a file or via a socket)
gobco adds the results as for any other kind of test

The instrumented files created by gobco probably solve the hardest part of this already.
Helpers are needed that:

write statistics,
read and accumulate statistics
report statement and branch coverage (as gobco does now) based on stored statistics

How to use the "gobco" to show the branch coverage of any test.go file

Dear developer,

  I have a requirement. I want to use the following runtime_test.go file to measure the branch coverage of this test unit. The runtime_test.go is put in the

go/src/github.com/ethereum/go-ethereum/core/v.m/runtime path, this is the go-ethereum project. I just can use the "go tool cover" to measure the statements coverage. I don't know how to use the "gobco" tool to measure the branch coverage.

`package runtime

import (
"fmt"
"github.com/ethereum/go-ethereum/common/hexutil"
"testing"

"github.com/ethereum/go-ethereum/common"

)

func TestExecute3(t *testing.T) {
var code, codeError = hexutil.Decode("0x6080604052600436106100d0576000357c0100000000000000000000000000000000000000000000000000000000900463ffffffff16806319449cb2146100d557806328090abb146101185780633c18d318146101a55780635e949fa0146102285780635ed7ca5b1461026b5780636fcb15001461028257806377bb09eb146102d95780638da5cb5b1461033c57806397dc97cb14610393578063a6f9dae1146103ea578063a9b1d5071461042d578063b269681d14610484578063b9b8af0b146104db578063be9a65551461050a575b600080fd5b3480156100e157600080fd5b50610116600480360381019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190505050610521565b005b34801561012457600080fd5b506101a3600480360381019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190803573ffffffffffffffffffffffffffffffffffffffff169060200190929190803573ffffffffffffffffffffffffffffffffffffffff169060200190929190803590602001909291905050506105c0565b005b3480156101b157600080fd5b506101e6600480360381019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190505050610642565b604051808273ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200191505060405180910390f35b34801561023457600080fd5b50610269600480360381019080803573ffffffffffffffffffffffffffffffffffffffff1690602001909291905050506106f5565b005b34801561027757600080fd5b50610280610794565b005b34801561028e57600080fd5b50610297610866565b604051808273ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200191505060405180910390f35b3480156108e757600080fd5b5061033a600480360381019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190803573ffffffffffffffffffffffffffffffffffffffff16906020019092919050505061088c565b005b34801561034857600080fd5b50610351610969565b604051808273ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200191505060405180910390f35b34801561039f57600080fd5b506103a861098e565b604051808273ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200191505060405180910390f35b3480156103f657600080fd5b5061042b600480360381019080803573ffffffffffffffffffffffffffffffffffffffff1690602001909291905050506109b4565b005b34801561043957600080fd5b50610442610a52565b604051808273ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200191505060405180910390f35b34801561049057600080fd5b50610499610bcd565b604051808273ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200191505060405180910390f35b3480156104e757600080fd5b506104f0610bf3565b604051808215151515815260200191505060405180910390f35b34801561051657600080fd5b5061051f610c06565b005b6000809054906101000a900473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff1614151561057c57600080fd5b80600160006101000a81548173ffffffffffffffffffffffffffffffffffffffff021916908373ffffffffffffffffffffffffffffffffffffffff16021790555050565b8173ffffffffffffffffffffffffffffffffffffffff168373ffffffffffffffffffffffffffffffffffffffff168573ffffffffffffffffffffffffffffffffffffffff167fa64da754fccf55aa65a1f0128a648633fade3884b236e879ee9f64c78df5d5d7846040518082815260200191505060405180910390a450505050565b600080600460008473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200190815260200160002060009054906101000a900473ffffffffffffffffffffffffffffffffffffffff16905060008173ffffffffffffffffffffffffffffffffffffffff1614156106ec57600360009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1690505b80915050919050565b6000809054906101000a900473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff1614151561075057600080fd5b80600260006101000a81548173ffffffffffffffffffffffffffffffffffffffff021916908373ffffffffffffffffffffffffffffffffffffffff16021790555050565b600160009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff161415801561083f57506000809054906101000a900473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff1614155b1561084957600080fd5b6001600260146101000a81548160ff021916908315150217905550565b600360009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1681565b6000809054906101000a900473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff161415156108e757600080fd5b80600460008473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200190815260200160002060006101000a81548173ffffffffffffffffffffffffffffffffffffffff021916908373ffffffffffffffffffffffffffffffffffffffff1602179055505050565b6000809054906101000a900473ffffffffffffffffffffffffffffffffffffffff1681565b600160009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1681565b6000809054906101000a900473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff16141515610a0f57600080fd5b806000806101000a81548173ffffffffffffffffffffffffffffffffffffffff021916908373ffffffffffffffffffffffffffffffffffffffff16021790555050565b6000600160009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff1614158015610aff57506000809054906101000a900473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff1614155b15610b0957600080fd5b30610b12610c7e565b808273ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff168152602001915050604051809103906000f080158015610b64573d6000803e3d6000fd5b5090507fef4c8685c12779a52dae7549eb7defa8523f67a054ad425b877a6b2da469a33181604051808273ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200191505060405180910390a190565b600260009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1681565b600260149054906101000a900460ff1681565b6000809054906101000a900473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff16141515610c6157600080fd5b6000600260146101000a81548160ff021916908315150217905550565b60405161033580610c8f833901905600608060405234801561001057600080fd5b5060405160208061033583398101806040528101908080519060200190929190505050806000806101000a81548173ffffffffffffffffffffffffffffffffffffffff021916908373ffffffffffffffffffffffffffffffffffffffff160217905550506102b2806100836000396000f30060806040526004361061004c576000357c0100000000000000000000000000000000000000000000000000000000900463ffffffff1680636ea056a91461004e578063c0ee0b8a146100b3575b005b34801561005a57600080fd5b50610099600480360381019080803573ffffffffffffffffffffffffffffffffffffffff16906020019092919080359060200190929190505050610146565b604051808215151515815260200191505060405180910390f35b3480156100bf57600080fd5b50610144600480360381019080803573ffffffffffffffffffffffffffffffffffffffff16906020019092919080359060200190929190803590602001908201803590602001908080601f0160208091040260200160405190810160405280939291908181526020018383808284378201915050505050509192919290505050610281565b005b600080809054906101000a900473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16633c18d318846040518263ffffffff167c0100000000000000000000000000000000000000000000000000000000028152600401808273ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff168152602001915050602060405180830381600087803b15801561020357600080fd5b505af1158015610217573d6000803e3d6000fd5b505050506040513d602081101561022d57600080fd5b810190808051906020019092919050505073ffffffffffffffffffffffffffffffffffffffff1660003660405180838380828437820191505092505050600060405180830381855af4915050905092915050565b5050505600a165627a7a7230582096d30d43ac84da6ca406f9bdb705dc4d45ae2ad120e8e63ba323e2b2edc50fb80029a165627a7a723058208a622ab443052d29e2b70988be79000000000000639ac1278c7eca1fdd4c28620d7b2be396050029")
if codeError != nil {
fmt.Println(codeError)
}
var sigName = "77bb09eb00000000000000000000000023da6277a6d2318d4de84bccd1b9a77b3a939310000000000000000000000000b433f064e905287187a21f7ed62f97dcae25005e"
var input = common.Hex2Bytes(sigName)
ret, _, error := Execute(code, input, nil)
fmt.Println(ret)
if error != nil {
fmt.Println(error)
}
}
`
BTW, I just can run "gobco sample/foo.go" in the Golang it can generate the result. I use the similar way to run "gobco runtime/runtime.go" it generate the following error information:
/home/xuhang/go/src/github.com/ethereum/go-ethereum/trie/database.go:27:2: cannot find package "github.com/VictoriaMetrics/fastcache" in any of:
/usr/local/go/src/github.com/VictoriaMetrics/fastcache (from $GOROOT)
/tmp/gobco-c142a8d7-7221-42da-b54c-6fc43070c024/src/github.com/VictoriaMetrics/fastcache (from $GOPATH)
/home/xuhang/go/src/github.com/VictoriaMetrics/fastcache
/home/xuhang/go/src/github.com/ethereum/go-ethereum/common/mclock/mclock.go:23:2: cannot find package "github.com/aristanetworks/goarista/monotime" in any of:
/usr/local/go/src/github.com/aristanetworks/goarista/monotime (from $GOROOT)
/tmp/gobco-c142a8d7-7221-42da-b54c-6fc43070c024/src/github.com/aristanetworks/goarista/monotime (from $GOPATH)
/home/xuhang/go/src/github.com/aristanetworks/goarista/monotime

Thanks for your reply in advance!:)

Fails with “argument must be inside $GOPATH/src”

Trying to run gobco 0.9.4 on a module I’m developing (outside of $GOPATH), I get:

$ gobco
argument "." must be inside "/home/vasiliy/.cache/gopath/src"

If I set up a $GOPATH specially for gobco, it still doesn’t work:

$ go env GOPATH
/home/vasiliy/tmp/gunison/gopath

$ head -1 /home/vasiliy/tmp/gunison/gopath/src/github.com/vfaronov/gunison/go.mod 
module github.com/vfaronov/gunison

$ gobco github.com/vfaronov/gunison
argument "github.com/vfaronov/gunison" must be inside "/home/vasiliy/tmp/gunison/gopath/src"

Test Logging does not seem to work when tests are run via gobco

I am able to debug tests by using Logf() statements E.g.

func (suite *FoobarTestSuite) TestFoobar1() {
	t := suite.T()

	t.Logf("foobar value: %s\n", someValue)
        ....
}

Giving output like:

=== RUN   TestFoobar/TestFoobar1
    foobar_test.go:31: foobar value: SNAFU

I am currently trying to use this to debug why my tests are failing under gobco (suspected environment issue not a gobco as they look for files in a location specified on either the test command line or via an environment var).

I note that when run under gobco the test output does not display (this is even adding -test -v to the command line).

Does gobco redirect log statements somehow?
Is there an alternative logging mechanism that would work with gobco
(I can write directly to a file of course)

Improvements to command line interface

Re: #23 (comment)
"It looks a bit strange to have these options interleaved with -test, but I didn't see any better way to avoid all the quoting and escaping issues when passing command line arguments."

May I suggest using -- with the standard unix convention to mean end of options.
So that:

gobco -test -test.v -verbose -test -test.count -test 5 -test -check.vv

Would become

gobco -verbose -- -test.v -test.count 5 -check.vv

That is you are required to give gobco options before options to go test

The -keep option is less useful without a way to set the location of the temporary directory.
I would like to suggest a -workspace option or similar.

I note a gradual build up of gobco directories in /tmp (as per #9 ). I am not using the -keep option but using a new -workspace option I could guarantee my build doesn't cause this.
I'm not sure yet if these are produced when the tests fail or when I press ctrl-C to abort them or both or something else.
You possibly need to add a signal handler to clean things up before the program exits.

How can I get branch coverage using gobco simultaneously while running go test coverage?

I am already using go test cover command, which can compute the coverage of actually running go server

rm -frv ./test-able-exe 
go test -coverprofile=coverage.out -c main.go main_test.go -o test-able-exe -coverpkg=./...
./test-able-exe -test.coverprofile=coverage.out -test.v -test.run=TestMain

and looking for something like following such gobco and go test will simultaneously compute coverage. Right now, I need to up the server 2 times.

rm -frv ./test-able-exe 
go test -coverprofile=coverage.out -c main.go main_test.go -o test-able-exe -coverpkg=./...
./test-able-exe -test.coverprofile=coverage.out -test.v -test.run=TestMain  -toolexec 'gobco'

Add to installation instruction example for Go 1.17

STEPS TO REPRODUCE

Use Go 1.17

go version
go version go1.17 windows/amd64

go get github.com/rillig/gobco

ACTUAL RESULT

go get github.com/rillig/gobco
go: downloading github.com/rillig/gobco v0.10.2
go get: installing executables with 'go get' in module mode is deprecated.
        Use 'go install pkg@version' instead.
        For more information, see https://golang.org/doc/go-get-install-deprecation
        or run 'go help get' or 'go help install'.

EXPECTED RESULT
Documentation is aware about Go 1.17 and suggest to use go install github.com/rillig/gobco@latest

Finding original package path in directory with `_test` package may fails

Overview

In a directory with multiple package (Maybe they are something and something_test), it tries to find the package's path of something. It is performed by looking for its direct import import "github.com/org/repo/something" in the directory. here: https://github.com/rillig/gobco/blob/master/instrumenter.go#L623-L643

Maybe it expects there is such import in a file with something_test package. But, in some cases like it uses another package as helper, the import may not be in the directory. So there are need to provide secure way to determine the package's path.

Example

it can't find the path

something/something.go

package something

type SomeInterface interface {
	Add(a, b int) int
}

type SomeStruct struct {
}

func (s *SomeStruct) Add(a, b int) int {
	return a + b
}

something/something_test.go

package something_test

import (
	"awesomeProject/helper"
	"testing"
)

func Test_Add(t *testing.T) {
	h := helper.TestHelper{}
	s := h.SomeHelperMethod()

	if s.Add(1, 1) != 2 {
		t.Error("error")
	}
}

helper/helper.go

package helper

import "awesomeProject/something"

type TestHelper struct {
}

func (h *TestHelper) SomeHelperMethod() something.SomeInterface {
	return &something.SomeStruct{}
}

For above files, error like below occurred.

% gobco
/private/var/folders/p9/_rh78g8d2clbbn1gk1qd6s6r0000gq/T/gobco-30bfe6a8ee2bdbbc/module-ade35ebcc7c72727/something/gobco_bridge_test.go:3:9: invalid import path: 
go test .: exit status 1

It find incorrect path with same package name

interface/something/something.go

package something

type SomeInterface interface {
	Add(a, b int) int
}

something/factory.go

package something

import "awesomeProject/interface/something"

func NewSomething() something.SomeInterface {
	return &SomeStruct{}
}

something/something.go

package something

type SomeStruct struct {
}

func (s *SomeStruct) Add(a, b int) int {
	return a + b
}

something/something_test.go

package something_test

import (
	"awesomeProject/helper"
	"testing"
)

func Test_Add(t *testing.T) {
	h := helper.TestHelper{}
	s := h.SomeHelperMethod()

	if s.Add(1, 1) != 2 {
		t.Error("error")
	}
}

helper/helper.go

package helper

import (
	somethingInterface "awesomeProject/interface/something"
	"awesomeProject/something"
)

type TestHelper struct {
}

func (h *TestHelper) SomeHelperMethod() somethingInterface.SomeInterface {
	return &something.SomeStruct{}
}

For above files, error like below occurred.

% gobco
# awesomeProject/something_test [awesomeProject/something.test]
./gobco_bridge_test.go:6:19: undefined: something.GobcoCover
FAIL    awesomeProject/something [build failed]
FAIL
go test .: exit status 1

panic: no such file or directory

Hello @rillig I'm trying to use gobco but I'm unable due to the following error

when running gobco in my terminal I get:

panic: open gopath/src/<path_to_my_project>: no such file or directory

I'm able to run gobco successfully in the sample folder of the gobco project

Am I missing some step to use it in my project?

The $gopath variable is correctly set in my mac computer

I'm using go 1.15.2

Thanks in advance

Incorrectly flags last switch case

Given the following toy.go:

package toy

import "math/rand"

func Foo() string {
	switch rand.Intn(2) {
	case 0:
		return "zero"
	case 1:
		return "one"
	}
	panic("impossible")
}

and the following toy_test.go:

package toy

import (
	"math/rand"
	"testing"
)

func TestFoo_zero(t *testing.T) {
	rand.Seed(0)
	if s := Foo(); s != "zero" {
		t.Fatalf(`%q != "zero"`, s)
	}
}

func TestFoo_one(t *testing.T) {
	rand.Seed(1)
	if s := Foo(); s != "one" {
		t.Fatalf(`%q != "one"`, s)
	}
}

gobco reports:

Branch coverage: 3/4
toy.go:9:7: condition "rand.Intn(2) == 1" was once true but never false

although both cases are actually covered.

Tests referring mock non-go files fails with errors

ababu17$ gobco pkg/models/job_test.go 
--- FAIL: TestGetFlow (0.00s)
    defs_test.go:25: failed to parse definitions from /var/folders/dc/ffvyv57j5dzf20lp3tlqfpqw0000gn/T/gobco-9b9d3277-4c11-4dde-9cf0-66bbd231fc28/src/......../pkg/models/mocks/abc.yaml.Error failed to read contents of file /var/folders/dc/ffvyv57j5dzf20lp3tlqfpqw0000gn/T/gobco-9b9d3277-4c11-4dde-9cf0-66bbd231fc28/src/....../pkg/models/mocks/abc.yaml: file /var/folders/dc/ffvyv57j5dzf20lp3tlqfpqw0000gn/T/gobco-9b9d3277-4c11-4dde-9cf0-66bbd231fc28/src/......../pkg/models/mocks/abc.yaml does not exist
    --- FAIL: TestGetFlow/Case_1:_Successful_get_flow (0.00s)
        defs_test.go:301: Expected error: false but got error true
--- FAIL: TestSetSHA256Sum (0.00s)
    job_test.go:24: failed to fetch defs. Error: &os.PathError{Op:"open", Path:"mocks/abc.yaml", Err:0x2}
    --- FAIL: TestSetSHA256Sum/Case_2:_type_only_changed (0.00s)
        job_test.go:72: expecetd: false, found: true
    --- FAIL: TestSetSHA256Sum/Case_3:_name_slightly_changing (0.00s)
        job_test.go:72: expected: false, found: true
FAIL
FAIL	...../pkg/models	0.543s
exit status 1

Does the coverage of a Package calculated only by the tests in same package

Hi @rillig

First of all Thanks so much for Branch(C1) and Condition(C2) coverage implementation. It helps us a lot.

I have to find C1 for whole project. So I planned to pass packages one by one to get whole coverage.
In this case:
Is the package coverage rate (C1) that can be measured by gobco limited to the portion of the test code in the same package that passes?
For example, if there are packages A and B, and a function in package A is only tested within package B, will the test coverage of package A be 0%?

use gobco cannot work

go-version: echo $GOROOT /go/go1.16

origin func:

func BranchCovTest(node int) {
	_ = node > 50
	_ = node == 0
	_ = node < 0

	if node == 1 {
		logs.Errorf("xypDEBUG|1")
	}
	if node == 3 {
		logs.Errorf("xypDEBUG|3")
	} else if node == 1 {
		logs.Errorf("xypDEBUG|1")
	} else {
		logs.Errorf("xypDEBUG|else")
	}
}

test func:

func TestBranchCovTest(t *testing.T) {
	BranchCovTest(1)
	BranchCovTest(10)
}

output:

ok      code.byted.org/ad/demeter-case-box/common       0.032s

Branch coverage: 0/0

How does this tool calculate the branch coverage of the entire project?

How does this tool calculate the branch coverage of the entire project? When I execute it in the root directory of the project, I encounter the following error:

panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
main.(*instrumenter).writeGobcoFiles(0x1?, {0xc00008c540?, 0x1?}, {0x0?, 0x0?, 0x0?})
        /Users/bytedance/go/pkg/mod/github.com/rillig/[email protected]/instrumenter.go:574 +0x3cf
main.(*instrumenter).instrument(0xc000388000, {0x117bcff, 0x1}, {0x0, 0x0}, {0xc00008c540, 0x5f})
        /Users/bytedance/go/pkg/mod/github.com/rillig/[email protected]/instrumenter.go:79 +0x1b2
main.(*gobco).instrument(0xc0000d2000)
        /Users/bytedance/go/pkg/mod/github.com/rillig/[email protected]/main.go:266 +0x275
main.gobcoMain({0x11b4478?, 0xc0000ac008?}, {0x11b4478?, 0xc0000ac010?}, {0xc00009a180, 0x3, 0x3})
        /Users/bytedance/go/pkg/mod/github.com/rillig/[email protected]/main.go:28 +0x72
main.main()
        /Users/bytedance/go/pkg/mod/github.com/rillig/[email protected]/main.go:21 +0x46

If I add some parameters, for example:

gobco test ./app/... -stats ./b.out

I will encounter the following error:

panic: gobco: checking multiple packages doesn't work yet

goroutine 1 [running]:
main.(*gobco).parseArgs(0xc000114000?, {0xc0000140b0?, 0x5?, 0x5?})
        /Users/bytedance/go/pkg/mod/github.com/rillig/[email protected]/main.go:121 +0x205
main.(*gobco).parseCommandLine(0x11b4478?, {0xc0000140a0?, 0x11b4478?, 0xc000012020?})
        /Users/bytedance/go/pkg/mod/github.com/rillig/[email protected]/main.go:61 +0x38
main.gobcoMain({0x11b4478?, 0xc000012018?}, {0x11b4478?, 0xc000012020?}, {0xc0000140a0, 0x5, 0x5})
        /Users/bytedance/go/pkg/mod/github.com/rillig/[email protected]/main.go:26 +0x5e
main.main()
        /Users/bytedance/go/pkg/mod/github.com/rillig/[email protected]/main.go:21 +0x46

-help and -version options

Two minor niggles:

gobco test -help

dumps the output of "go test -help" starting with:

usage: go test [build/test flags] [packages] [build/test flags & test binary flags]

and ending with:

2019/07/18 15:37:37 exit status 2

go test -help inexplicable returns an error code for -help (gobco itself returns 0 as it should for this option)

Also there is no -version option. This would simply report the program version and exit. This wil become useful once there are some versions (as per issue #3 )

On each gobco run, gobco creates folders that it doesn't cleanup

ababu17$ ls /var/folders/dc/ffvyv57j5dzf20lp3tlqfpqw0000gn/T/gobco-
gobco-09af28b7-af8d-408d-9032-4bdc1ed38432/ gobco-47d0be85-f3a1-4c34-a5fa-1b869ed0a45b/ gobco-8544638a-b5fd-47a4-b4ae-24fdd65a4147/ gobco-c32b52d6-b1a0-4a18-8c21-95fbcf2386b3/ gobco-ef7fe756-a826-4f89-a982-bf5f306c3533/
gobco-2033001c-eceb-428c-a7be-fb516c8edf17/ gobco-48d6af0e-6ba8-4b74-8dbe-081259199655/ gobco-879c5e44-0c0f-46cd-9436-a249fcbedb34/ gobco-c6e1336f-20a6-447e-86a6-1a035ef60f8a/ gobco-f04d467e-30c4-42f9-8772-561bcb4dd239/
gobco-23b03285-d1ea-4d71-b000-a4966f35433c/ gobco-4e3d2212-158e-4192-af1e-4eaf347d6e1d/ gobco-93d14d79-7b1a-4ae5-9325-56031bc75261/ gobco-d36b5b71-616b-481b-89b8-0148087b687f/ 
gobco-255e2c5d-0559-4413-9524-3ac5ed9f233f/ gobco-6243a4aa-8bbf-4973-bfee-98d6937fcb84/ gobco-9e31fd2c-69bb-4bb7-bf4b-c027fd6223a3/ gobco-dd6c3847-3cfc-49c4-8f43-dbc57b60088b/ 
gobco-367efb29-538a-415f-b8db-af545b5771be/ gobco-70ca82db-1137-40e3-b295-375e7a42a852/ gobco-9ff8c0f3-9234-4938-9615-013f7d4f2be7/ gobco-e6652f9c-0687-4cf7-b45a-a7bb0aab2b41/ ```

How to use the "gobco" to measure the branch coverage of any test.go file

Dear developer,

I have a requirement. I want to use the following runtime_test.go file to measure the branch coverage of this test unit. The runtime_test.go is put in the
go/src/github.com/ethereum/go-ethereum/core/v.m/runtime path, this is the go-ethereum project. I just can use the "go tool cover" to measure the statements coverage. I don't know how to use the "gobco" tool to measure the branch coverage.

BTW, I just can run "gobco sample/foo.go" in the Golang it can generate the result. I use the similar way to run "gobco runtime/runtime.go" it generate the following error information:
/home/xuhang/go/src/github.com/ethereum/go-ethereum/trie/database.go:27:2: cannot find package "github.com/VictoriaMetrics/fastcache" in any of:
/usr/local/go/src/github.com/VictoriaMetrics/fastcache (from $GOROOT)
/tmp/gobco-c142a8d7-7221-42da-b54c-6fc43070c024/src/github.com/VictoriaMetrics/fastcache (from $GOPATH)
/home/xuhang/go/src/github.com/VictoriaMetrics/fastcache
/home/xuhang/go/src/github.com/ethereum/go-ethereum/common/mclock/mclock.go:23:2: cannot find package "github.com/aristanetworks/goarista/monotime" in any of:
/usr/local/go/src/github.com/aristanetworks/goarista/monotime (from $GOROOT)
/tmp/gobco-c142a8d7-7221-42da-b54c-6fc43070c024/src/github.com/aristanetworks/goarista/monotime (from $GOPATH)
/home/xuhang/go/src/github.com/aristanetworks/goarista/monotime

Thanks for your reply in advance!:)

undefined: gobco8

Hi rillig, thank you for developing the tool: gobco. I have encountered some issues while using the tool. Please refer to the two images above for the problem details. It would be appreciated if you could help me resolve this issue.

provide a way to get net branch coverage at project level

Currently, the awesome gobco tool provides an awesome view of package level branch coverage.
However, since the output of gobco is a string, any attempt to aggregate the individual package level branch coverage into a coverage at the project level, would potentially involve a lot of string magic(undesirable) so is it possible to:

Provide an aggregated metric across all packages
or
Provide outputs as json or any-other parsable format for external integrations like CI systems

cannot use xx (variable of type xx) as type bool in argument to GobcoCover

@rillig thank you for developing the tool: gobco. I have encountered some issues while using the tool. Please refer to the image above for the problem details. It would be appreciated if you could help me resolve this issue.

master branch broken and no tagged release versions

Hi,
The current head commit doesn't work (for me anyway) as follows:


go get github.com/rillig/gobco
go: downloading github.com/rillig/gobco v0.0.0-20190712181434-65bb23983152
go: downloading github.com/google/uuid v1.1.1
# github.com/rillig/gobco
/main.go:82:16: undefined: os.UserHomeDir
../../../go/pkg/mod/github.com/rillig/[email protected]/main.go:204:24: exitErr.ExitCode undefined (type *exec.ExitError has no field or method ExitCode)

Apologies if this is teaching your grandmother to suck eggs but I would recommend having separate feature branches for experimental changes and keeping only tagged release versions on the master branch. See for example, https://datasift.github.io/gitflow/IntroducingGitFlow.html

Regards,

Bruce.

gobco doesn't work well with nested packages

when individual package paths are passed, gobco works like a charm
However, when gobco is run at a higher levels, composed of multiple nested packages, it errors out saying no Go files....

ababu17$ gobco pkg/
can't load package: package ........./pkg: no Go files in /var/folders/dc/ffvyv57j5dzf20lp3tlqfpqw0000gn/T/gobco-f9cfa878-befb-4140-861e-56900a45e25b/src/.........../pkg
exit status 1
open /var/folders/dc/ffvyv57j5dzf20lp3tlqfpqw0000gn/T/gobco-f9cfa878-befb-4140-861e-56900a45e25b/gobco-counts.json: no such file or directory


ababu17$ ls
Makefile	README.md	coverage.out	go.mod		go.sum		pkg		vendor


ababu17$ ls pkg/
cover.sh	env		logging		messaging	models		test.sh		utils


ababu17$ ls pkg/models/
constants.go	defs.go		defs_test.go	interface.go	job.go		job_test.go	mocks


ababu17$ ls pkg/utils/
file.go			file_test.go		object.go		object_test.go		schedule.go		schedule_test.go	str.go			str_test.go

Add support for go modules

I (@rillig) don't need support for go modules myself, therefore I didn't implement it. It would be nice to have though.

See #17 for a workaround.

In Readme.md - Running gobco on package netbsd.org/pkglint: Not getting this

I am not able to understand the meaning of the line

Upgrading from old version

I have been using an old version with some success:

github.com/rillig/gobco v0.0.0-20190630104235-ff234e5b7147

Passing arguments through to go test as follows:

gobco -args -short -tags gobco

I now looking to migrate to a more recent version.
I find that gobco no longer accepts these options.
Presumably it should now be something like:

gobco -test -short -test -tags -test gobco

However, I cannot get far enough to try this.
I am using go 1.11 as that is the latest version backported to Debian 9 (Stretch).

If I run gobco by itself from my source directory I get the somewhat cryptic:

argument "." must be inside "/home/brucea/go/src"

I do not use GOPATH. It is unset in my environment. However go env shows it as

GOPATH="/home/brucea/go"

Which I think is not a coincidence. Sure enough if I run:

GOPATH=`pwd` gobco

I get instead:

argument "." must be inside "<pwd>/src"

My code is not in a directory called src however.
If I rename the path to src just to explore this issue I can then run:

 GOPATH=`pwd`/.. gobco

It gets further but then fails to find packages I depend on as these are not in sub-directories of src (i.e. relative to GOPATH) either:
E.g.

foobar.go:4:2: cannot find package "github.com/cenkalti/backoff" in any of:
/usr/lib/go-1.11/src/github.com/cenkalti/backoff (from $GOROOT)
/tmp/gobco-1f233fb4-c10d-4a4d-b8d3-1f5ea2fd571d/src/github.com/cenkalti/backoff
    /pwd/one/dir/up/src/github.com/cenkalti/backoff

Do you have any suggestions as to how to resolve this?

Request for more information

I have for a long time used the default go test -cover and have seen it lacking in several practical use cases as expected(since it is known to be measuring the statement coverage not the useful branch coverage). I instantly fell in love with this tool most importantly as it not only provides branch coverage as a metric but also provides a way for the user to know the untested code paths... Is there a way also to suppress some of the warnings of un-tested code branches and raise the coverage?
I see the issues section of the repo literally empty(compared to most other open source projects) which is more than awesome and most importantly also, I have almost got an instant help from @rillig on every github issue I have raised on this repo till date which is extremely awesome... But, just out of curiosity, is there a set of known issues grouped by their category that the is not documented for now?
Can the information about the community that built this tool, users using this tool be shared in the docs?
Can the information on some of the alternatives to this tool and why this tool was built outside of go test --cover instead of getting branch coverage as a way to measure code-coverage or at-least as an add-on in the default go test --cover not done?

apologies in advance if some of my questions don't make any sense, but I am just curious before using this tool for my project

Record statistics on which test covers which condition

Assuming that gobco would output which test covers which condition, it would be possible to answer questions like the following:

For each testee, is there a test that covers all branches of that testee? (minimal unit tests)
Which tests are necessary to get 100% branch coverage for a given testee?

Implementation ideas:

Have an additional list of test names for each condition, possibly indirectly by mapping the test name to an int for performance.
Extend the gobco JSON output with this list of tests.

gobco installation fails on mac

$ GO111MODULE=on go get github.com/rillig/gobco
# github.com/rillig/gobco
github.com/rillig/[email protected]/instrumenter.go:306:66: undefined: gobco_fixed_go
github.com/rillig/[email protected]/instrumenter.go:309:71: undefined: gobco_fixed_test_go
github.com/rillig/[email protected]/instrumenter.go:311:75: undefined: gobco_variable_test_go

$ GO111MODULE=on go get -d github.com/rillig/gobco
$ GO111MODULE=on go generate github.com/rillig/gobco
go: not generating in packages in dependency modules
$ GO111MODULE=on go install github.com/rillig/gobco
# github.com/rillig/gobco
github.com/rillig/[email protected]/instrumenter.go:306:66: undefined: gobco_fixed_go
github.com/rillig/[email protected]/instrumenter.go:309:71: undefined: gobco_fixed_test_go
github.com/rillig/[email protected]/instrumenter.go:311:75: undefined: gobco_variable_test_go

C02Y80MSJG5J:hia-api-commons ababu17$ go version
go version go1.12.9 darwin/amd64

TestMain injection with `_test` package suffix fails

As a minimum way to recreate, have these two files in a directory:

// add.go
package add

func Add(x, y int) int {
    return x + y
}

// add_test.go
package add_test

import (
    "os"
    "testing"

    "my-module/pkg/add"
)

func TestMain(t *testing.M) {
    os.Exit(t.Run())
}

func TestAdd(t *testing.T) {
    result := add.Add(1, 2)
    if result != 3 {
        t.Errorf("Got %d but wanted 3", result)
    }
}

The error is:

# Cutting out extra bits
$ gobco
./add_test.go:11:10: undefined: gobcoCounts

If I remove TestMain, it works. Looking at gobco -keep, it looks like gobcoCounts is defined in its own file that would use package add, not package add_test. We use the _test pattern to help enforce testing external behavior, and we can't move TestMain out because it also initializes some connections that are referenced in the actual tests.