maxatome / go-testdeep

Extremely flexible golang deep comparison, extends the go testing package, tests HTTP APIs and provides tests suite

go-testdeep

Extremely flexible golang deep comparison, extends the go testing package.

Currently supports go 1.16 → 1.22.

• Latest news
• Synopsis
• Description
• Installation
• Functions
• Available operators
• Helpers
  • tdhttp or HTTP API testing helper
  • tdsuite or testing suite helper
  • tdutil aka the helper of helpers
• See also
• License
• FAQ

Latest news

• 2023/12/27: v1.14.0 release;
• 2023/03/18: v1.13.0 release;
• 2022/08/07: v1.12.0 release;
• see commits history for other/older changes.

Synopsis

Make golang tests easy, from simplest usage:

import (
  "testing"

  "github.com/maxatome/go-testdeep/td"
)

func TestMyFunc(t *testing.T) {
  td.Cmp(t, MyFunc(), &Info{Name: "Alice", Age: 42})
}

To a bit more complex one, allowing flexible comparisons using TestDeep operators:

import (
  "testing"

  "github.com/maxatome/go-testdeep/td"
)

func TestMyFunc(t *testing.T) {
  td.Cmp(t, MyFunc(), td.Struct(
    &Info{Name: "Alice"},
    td.StructFields{
      "Age": td.Between(40, 45),
    },
  ))
}

Or anchoring operators directly in literals, as in:

import (
  "testing"

  "github.com/maxatome/go-testdeep/td"
)

func TestMyFunc(tt *testing.T) {
  t := td.NewT(tt)

  t.Cmp(MyFunc(), &Info{
    Name: "Alice",
    Age:  t.Anchor(td.Between(40, 45)).(int),
  })
}

To most complex one, allowing to easily test HTTP API routes, using flexible operators:

import (
  "testing"
  "time"

  "github.com/maxatome/go-testdeep/helpers/tdhttp"
  "github.com/maxatome/go-testdeep/td"
)

type Person struct {
  ID        uint64    `json:"id"`
  Name      string    `json:"name"`
  Age       int       `json:"age"`
  CreatedAt time.Time `json:"created_at"`
}

func TestMyApi(t *testing.T) {
  var id uint64
  var createdAt time.Time

  testAPI := tdhttp.NewTestAPI(t, myAPI) // ← ①

  testAPI.PostJSON("/person", Person{Name: "Bob", Age: 42}). // ← ②
    Name("Create a new Person").
    CmpStatus(http.StatusCreated). // ← ③
    CmpJSONBody(td.JSON(`
// Note that comments are allowed
{
  "id":         $id,             // set by the API/DB
  "name":       "Alice",
  "age":        Between(40, 45), // ← ④
  "created_at": "$createdAt",    // set by the API/DB
}`,
      td.Tag("id", td.Catch(&id, td.NotZero())),        // ← ⑤
      td.Tag("createdAt", td.All(                       // ← ⑥
        td.HasSuffix("Z"),                              // ← ⑦
        td.Smuggle(func(s string) (time.Time, error) {  // ← ⑧
          return time.Parse(time.RFC3339Nano, s)
        }, td.Catch(&createdAt, td.Between(testAPI.SentAt(), time.Now()))), // ← ⑨
      )),
    ))
  if !testAPI.Failed() {
    t.Logf("The new Person ID is %d and was created at %s", id, createdAt)
  }
}

1. the API handler ready to be tested;
2. the POST request with automatic JSON marshalling;
3. the expected response HTTP status should be http.StatusCreated and the line just below, the body should match the JSON operator;
4. some operators can be embedded, like Between here;
5. for the $id placeholder, Catch its value: put it in id variable and check it is NotZero;
6. for the $createdAt placeholder, use the All operator. It combines several operators like a AND;
7. check that $createdAt date ends with "Z" using HasSuffix. As we expect a RFC3339 date, we require it in UTC time zone;
8. convert $createdAt date into a time.Time using a custom function thanks to the Smuggle operator;
9. then Catch the resulting value: put it in createdAt variable and check it is greater or equal than testAPI.SentAt() (the time just before the request is handled) and lesser or equal than time.Now().

See tdhttp helper or the FAQ for details about HTTP API testing.

Example of produced error in case of mismatch:

Description

go-testdeep is historically a go rewrite and adaptation of wonderful Test::Deep perl.

In golang, comparing data structure is usually done using reflect.DeepEqual or using a package that uses this function behind the scene.

This function works very well, but it is not flexible. Both compared structures must match exactly and when a difference is returned, it is up to the caller to display it. Not easy when comparing big data structures.

The purpose of go-testdeep, via the td package and its helpers, is to do its best to introduce this missing flexibility using "operators", when the expected value (or one of its component) cannot be matched exactly, mixed with some useful comparison functions.

See go-testdeep.zetta.rocks for details.

Installation

$ go get github.com/maxatome/go-testdeep

Helpers

The goal of helpers is to make use of go-testdeep even more powerful by providing common features using TestDeep operators behind the scene.

tdhttp or HTTP API testing helper

The package github.com/maxatome/go-testdeep/helpers/tdhttp provides some functions to easily test HTTP handlers.

See tdhttp documentation for details or FAQ for an example of use.

tdsuite or testing suite helper

The package github.com/maxatome/go-testdeep/helpers/tdsuite adds tests suite feature to go-testdeep in a non-intrusive way, but easily and powerfully.

A tests suite is a set of tests run sequentially that share some data.

Some hooks can be set to be automatically called before the suite is run, before, after and/or between each test, and at the end of the suite.

See tdsuite documentation for details.

tdutil aka the helper of helpers

The package github.com/maxatome/go-testdeep/helpers/tdutil allows to write unit tests for go-testdeep helpers and so provides some helpful functions.

See tdutil for details.

See also

• testify: a toolkit with common assertions and mocks that plays nicely with the standard library
• go-cmp: package for comparing Go values in tests

License

go-testdeep is released under the BSD-style license found in the LICENSE file in the root directory of this source tree.

Internal function deepValueEqual is based on deepValueEqual from reflect golang package licensed under the BSD-style license found in the LICENSE file in the golang repository.

Uses two files (bypass.go & bypasssafe.go) from Go-spew which is licensed under the copyfree ISC License.

Public Domain Gopher provided by Egon Elbre. The Go gopher was designed by Renee French.

FAQ

See FAQ.

go-testdeep's Issues

Difficult to call t.Parallel with td.T

testdeep encourages wrapping the *testing.T into a *td.T. This
unfortunately makes it difficult to call t.Parallel() on the
*testing.T once hidden behind the *td.T.

func TestFoo(tt *testing.T) {
    t := td.NewT(tt)
    
    t.Run("foo", func(t *td.T) {
        // t.Parallel() <<< errors because Parallel() does not exist
        //                  on the testing.TB interface.
        
        t.Cmp(...)
    })
    
    t.Run("bar", func(t *td.T) {
        // t.Parallel()
        
        t.Cmp(...)
    })
}

It's possible to call t.Parallel() on the *testing.T with some
upcasting on the wrapped testing.TB.

func Parallel(t *td.T) {
	tp, ok := t.TB.(interface{ Parallel() })
	if ok {
		tp.Parallel()
	}
}

If you're open to addressing this within testdeep, there are two
straightforward options:

• Add a top-level func Parallel(*td.T) that does the same as above.
  Usage would then be,

  t.Run("foo", func(t *td.T) {
      td.Parallel(t)
      
      t.Cmp(...)
  })

• Add a Parallel() method on td.T. Usage:

  t.Run("foo", func(t *td.T) {
      t.Parallel()
      
      t.Cmp(...)
  })

The latter is closest to what people already do with *testing.T so it
might be preferable.

---

Separately, either choice opens room for a setting in testdeep to
always call t.Parallel() for subtests invoked with a specific
*td.T.Run. For example,

func TestFoo(tt *testing.T) {
    t := td.NewT(tt).AlwaysParallel(true)
    
    t.Run("foo", func(t *td.T) {
        // already parallel
        t.Cmp(...)
    })
    
    t.Run("bar", func(t *td.T) {
        // already parallel
        t.Cmp(...)
    })
}

But that can be a separate discussion.

Add Delay operator

import (
  "github.com/maxatome/go-testdeep/helpers/tdhttp"
  "github.com/maxatome/go-testdeep/td"
)
…
beforeReq := time.Now().Truncate(0)
tdhttp.CmpJSONResponse(
    tdhttp.PostJSON("/create/item", myBody),
    myAPIHandler,
    td.Struct(&MyBody{…}, td.StructFields{
        "CreatedAt": td.Between(beforeReq, time.Now()),
    }),
)

does not work as expected, coz time.Now() is evaluated before the request is sent, so time.Now() is probably before newItem.CreatedAt.
Of course the test can be replaced by:

"CreatedAt": td.Gte(beforeReq),

but time.Now() is no longer involved, or by:

"CreatedAt": td.Code(func (t time.Time) {
    return !t.Before(beforeReq) && !t.After(time.Now())
})

but it is a bit heavy…

So introducing Delay operator would solve this problem doing:

"CreatedAt": td.Delay(func() td.TestDeep { return td.Between(beforeReq, time.Now()) }),

Add JSON operator

import td "github.com/maxatome/go-testdeep"
…
td.Cmp(t, gotValue, td.JSON(
    `{"name":"$name","age":$2}`,
    td.Tag("name", td.Re(`^Bob`)),
    td.Between(40, 45),
))

Placeholders can be named ($name) or numeric ($12) and mixed:

• named: they reference a td.Tag in parameters list (like "name" above)
• numeric: they reference a positional parameter, from $1.

They can be enclosed in double-quotes (compatible with JSON specification) or not:

`{"name":"$name","age":"$age"}` 
`{"name":$name,"age":$age}`
`{"name":"$name","age":$age}`
`{"name":"$name","age":$2}`

Signature:

JSON(json interface{}, operators ...TestDeep) TestDeep

json can be:

• string
  • JSON string like {"name":"$name","age":"$2"}
  • A file name: the file contents is expected to be one JSON value
• []byte containing JSON
• io.Reader stream containing JSON (is ReadAll before unmarshaling)

To be easier to use, JSON operator enables the BeLax feature during its comparison.

Note the introduction of Tag operator allowing to name an operator. At the time of writing, it is only intended to be used in JSON parameters.

Branch json implements this issue, but not fully documented yet.

Add Catch operator

import (
  "fmt"
  td "github.com/maxatome/go-testdeep"
  "github.com/maxatome/go-testdeep/helper/tdhttp"
)
…
var id int64
tdhttp.CmpJSONResponse(
    tdhttp.NewJSONRequest("POST", "/create/item", myBody),
    myAPIHandler,
    td.Struct(&MyBody{…}, td.StructFields{
        "ID": td.All(td.NotZero(), td.Catch(&id)),
    }),
)
fmt.Printf("The just created item ID is %d\n", id)

Catch(PTR) fails only if type is not compatible. Otherwise, it copy the got value into the passed pointer.

Useful in cases as the example above, to keep a copy of a non-guessable field just set by a function (here a HTTP request).

It could even be an optional smuggler operator, avoiding to use td.All as above in most cases:

import (
  "fmt"
  td "github.com/maxatome/go-testdeep"
  "github.com/maxatome/go-testdeep/helper/tdhttp"
)
…
var id int64
tdhttp.CmpJSONResponse(
    tdhttp.NewJSONRequest("POST", "/create/item", myBody),
    myAPIHandler,
    td.Struct(&MyBody{…}, td.StructFields{
        "ID": td.Catch(&id, td.NotZero()),
    }),
)
fmt.Printf("The just created item ID is %d\n", id)

remove Run method from TF interface

The TestingFT interface includes the Run method, which uses the *testing.T parameter directly, which means that go-testdeep cannot be used with *testing.B or alongside other testing helper packages such as quicktest.

If you embed testing.TB inside T instead of defining your own testingFT interface, this problem goes away, and additionally you can take advantage of new features added to testing.TB (Cleanup for example) automatically and without breaking your API by adding methods to an interface type.

Add NaN and NotNaN operators

Add EnableEqual ContextConfig flag

When this flag is true (defaults to false), go-testdeep tries to use the Equal method, à la go-cmp.

Add Cmp function as a shorter alias of CmpDeeply

Add support for quotation marks in JSON operators

Problem:
We use different operators in JSONs which are then used for comparison. If we add regular operators to json string, e.g. Len, HasPrefix, etc, the JSON becomes invalid and formatters, validators cannot parse the string.

Proposal:
Add support for quotation marks for operators in JSON strings, similarly to currently it is being used for shortcut operators "$^NotEmpty". Ideally it could look similar "$^Len(10)"

Rework tdJSON.String() method

• use (custom?) json.MarshalIndent() + call String() method for each operator encountered
• do not truncate anymore

deepValueEqual infinite recursion with recursive map types

I've found an infinite recursion case with go-testdeep. It's a bit contrived I know; I was actually trying to break things. I'm currently developing an encoding library and trying to break it with test cases galore. Interestingly enough encoding/gob, reflect.DeepEqual (fixed in go1.15), and every 3rd party library I've found suffers from this same issue.

If given a map containing a struct that contains the same map, deepValueEqual recuses infinitely.

This test case reproduces the issue

func TestEqualRecursMap(t *testing.T) {
	gen := func() interface{} {
		type S struct {
			Map map[int]S
		}

		m := make(map[int]S)
		m[1] = S{
			Map: m,
		}

		return m
	}

	checkOK(t, gen(), gen())
}

Thanks so much for your work. Let me know if I can help.

Add tdhttp.Path() function

to build paths with query params:

tdhttp.Path("/1.0/path", "key", "value", url.Values{"x": []string{"a", "ba"}}, "key2", true)

Allow to embed operators directly in go data structures

Due to the go static typing system, the use of TestDeep operators is sometimes boring:

For the following type:

type Person struct {
  Name string
  Nick string
  Age  int
}

A common check for got Person is done as:

t.Cmp(got,
  td.Struct(
    Person{Name: "Bob"},
    td.StructFields{
      Nick: td.HasPrefix("Bobby"),
      Age:  td.Between(40, 45),
    })

What could be done with the operator-anchoring feature:

t.Cmp(got, Person{
  Name: "Bob",
  Nick: t.Anchor(td.HasPrefix("Bobby", "")).(string),
  Age:  t.Anchor(td.Between(40, 45)).(int),
})

(*testdeep.T).Anchor(operator TestDeep, model ...interface{}) interface{} would be a new method of the *testdeep.T type. It anchors the TestDeep operator into t as a value whose type is the same as model one. When the type can be deducted from the operator (as it is the case with Between above), model can be omitted.

The branch anchor implements this mechanism.

It works by injecting specific values t can correlate with corresponding operator during the comparison. After each Cmp* call, the correlation table is cleared.

Used types should provide enough room to inject these specific values and to avoid collisions with right values. So it won't work for bool, int8 and uint8.

The POC allows structs by registering a specific function. time.Time is already implemented this way.

Replace bytes.Buffer by strings.Builder

Now that go-testdeep requires go 1.16, the remaining bytes.Buffer instances only used to produce strings can be replaced by strings.Builder ones.

Recursive SubMap/SuperMap?

Is there a way to compare submaps/supermaps recursively (make sure that partial compare is propagated to nested maps)? Based on what I see, the default behavior is that only the parent map is compared partially, while all nested maps are expected to have exact match.

Comparison of reflect.Value types

Here to annoy you again :P

I'm aiming to support the encoding of reflect.Type and reflect.Value in my encoding library.
go-testdeep has been amazing for tests, but test cases where I compare a reflect.Value values, go-testdeep compares them like a typical struct, and doesn't compare the underlying value inside reflect.Value.

This means a reflect.Value of a value is only equal if the underlying value is at the same pointer address, and the reflect.Value was created in the same way due to how reflect uses the flag field in reflect.Value.

In any case, I don't think the semantics of how reflect.Value works is important. For my use case, I'm looking to compare the value inside reflect.Value, and I don't care so much about the reflect.Value itself, e.g if both are addressable or if only one is, although an argument could be made for this. I believe something like

if got.Type() == reflect.TypeOf(reflect.Value{}) {
    return deepValueEqual(ctx, got.Interface().(reflect.Value), expected.Interface().(reflect.Value))
}

in deepValueEqual could work, but I'm unsure how this might work with the rest of the codebase. I also know that the ptr field of reflect.Value could introduce an infinite recursion, an extreme case being a reflect.Value that is the value of itself, but less contrived examples do exist.

I'd be happy to experiment implementing this and submit a PR. Is this something you'd want to support?

Allow to set default values in requests issued by a tdhttp.TestAPI

Hi,

On one of my projects there are a lot of tests using always a testAPI with the same set of headers such as:

var defaultHeaders http.Header
testAPI.Post("/my/route", json.RawMessage(`{}`),  defaultsHeaders).
  CmpStatus(http.StatusCreated).
  CmpJSONBody(...)

< Same last 3 lines over and over > 

I wonder if it'd be possible to directly set this kind of default values inside the testAPI itself, such as:

testAPI := tdhttp.NewTestAPI(t, testRouter).DefaultRequestParams(<INSERT DEFAULT HEADERS HERE>)
testAPI.Post("/my/route", json.RawMessage(`{}`)).
  CmpStatus(http.StatusCreated).
  CmpJSONBody(...)

< Same last 3 lines over and over > 

Add comparison to github.com/go-test/deep

It would be helpful to have a comparison table or something in the readme to help people understand the difference between this package and https://github.com/go-test/deep. That has more stars and shows up better in search, while this library seems much more active.

Test failure: TestRun/Without_ptr:_only_non-ptr_methods

I'm working on packaging this library for Debian, and have encountered a test failure when building.

Versions:

• go 1.21.5 linux/amd64
• github.com/maxatome/go-testdeep v1.13.0
• github.com/davecgh/go-spew v1.1.1

Here is what I hope is the relevant part of the test output:

=== RUN   TestRun/Without_ptr:_only_non-ptr_methods
    suite_test.go:181: Run(): several methods are not accessible as suite is not a pointer but tdsuite_test.FullNoPtr: PostTest, Test2
=== RUN   TestRun/Without_ptr:_only_non-ptr_methods/Test1
=== RUN   TestRun/Without_ptr:_only_non-ptr_methods/Test3
=== NAME  TestRun/Without_ptr:_only_non-ptr_methods
    suite_test.go:210: ^[[1;33mFailed test^[[0m
        ^[[1;36mDATA.output: does not contain^[[0m
        ^[[1;31m             got: ^[[0;31m([]uint8) (len=140 cap=160) {^[[0m
                          ^[[0;31m 00000000  20 20 20 20 2d 2d 2d 20  50 41 53 53 3a 20 54 65  |    --- PASS: Te|^[[0m
                          ^[[0;31m 00000010  73 74 52 75 6e 2f 57 69  74 68 6f 75 74 5f 70 74  |stRun/Without_pt|^[[0m
                          ^[[0;31m 00000020  72 3a 5f 6f 6e 6c 79 5f  6e 6f 6e 2d 70 74 72 5f  |r:_only_non-ptr_|^[[0m
                          ^[[0;31m 00000030  6d 65 74 68 6f 64 73 2f  54 65 73 74 31 20 28 30  |methods/Test1 (0|^[[0m
                          ^[[0;31m 00000040  2e 30 30 73 29 0a 20 20  20 20 2d 2d 2d 20 50 41  |.00s).    --- PA|^[[0m
                          ^[[0;31m 00000050  53 53 3a 20 54 65 73 74  52 75 6e 2f 57 69 74 68  |SS: TestRun/With|^[[0m
                          ^[[0;31m 00000060  6f 75 74 5f 70 74 72 3a  5f 6f 6e 6c 79 5f 6e 6f  |out_ptr:_only_no|^[[0m
                          ^[[0;31m 00000070  6e 2d 70 74 72 5f 6d 65  74 68 6f 64 73 2f 54 65  |n-ptr_methods/Te|^[[0m
                          ^[[0;31m 00000080  73 74 33 20 28 30 2e 30  30 73 29 0a              |st3 (0.00s).|^[[0m
                          ^[[0;31m}^[[0m
        ^[[1;32m        expected: ^[[0;32mContains("Run(): several methods are not accessible as suite is not a pointer but tdsuite_test.FullNoPtr: PostTest, Test2")^[[0m
        [under operator Contains at suite_test.go:211]
        This is how we got here:
                TestRun.func3() helpers/tdsuite/suite_test.go:210

[snip]

    --- FAIL: TestRun/Without_ptr:_only_non-ptr_methods (0.00s)
        --- PASS: TestRun/Without_ptr:_only_non-ptr_methods/Test1 (0.00s)
        --- PASS: TestRun/Without_ptr:_only_non-ptr_methods/Test3 (0.00s)

Missing Test Name in error message When Using tdhttp.TestAPI inside a test of tdsuite

Hi, I'm new to API test of go-testdeep. When I'm running the following test suite, I got confused info in error message.

package api_test

import (
	"fmt"
	"net/http"
	"net/url"
	"strings"
	"testing"

	"github.com/maxatome/go-testdeep/helpers/tdhttp"
	"github.com/maxatome/go-testdeep/helpers/tdsuite"
	"github.com/maxatome/go-testdeep/td"
)

func Hello(w http.ResponseWriter, r *http.Request) {

	w.Header().Set("Content-Type", "application/json; charset=utf-8")

	if r.Method != http.MethodPost {
		w.WriteHeader(http.StatusMethodNotAllowed)
		w.Write([]byte(`{"errno":1, "msg":"Method not allowed", "data":{}}`))
		return
	}

	r.ParseForm()
	name := r.Form.Get("name")
	w.Write([]byte(fmt.Sprintf(`{"errno":0, "msg":"Hello %s", "data":{}}`, name)))
}

func TestAPISuite(t *testing.T) {
	tdsuite.Run(t, &APISuite{})
}

type APISuite struct {
	ta  *tdhttp.TestAPI
}

func (s *APISuite) Setup(t *td.T) error {
	mux := http.NewServeMux()
	mux.HandleFunc("/hello", Hello)
	s.ta = tdhttp.NewTestAPI(t, mux)
	return nil
}

func (s *APISuite) TestHello(t *td.T) {
	
	s.ta.Run("Wrong Method", func(ta *tdhttp.TestAPI) {
		ta.Get("/hello").
			CmpStatus(http.StatusMethodNotAllowed).
			CmpHeader(http.Header{"Content-Type": []string{"application/json; charset=utf-8"}}).
			CmpJSONBody(td.JSON(`{"errno":1, "msg":"Method not allowed1", "data":{}}`))
	})
	s.ta.Get("/hello").
		Name("Get /hello subtest").
		CmpStatus(http.StatusMethodNotAllowed).
		CmpHeader(http.Header{"Content-Type": []string{"application/json; charset=utf-8"}}).
		CmpJSONBody(td.JSON(`{"errno":1, "msg":"Method not allowed1", "data":{}}`))

	s.ta.PostForm("/hello", url.Values{
		"name": []string{"Longyue"},
	}).
	CmpStatus(http.StatusOK).
	CmpHeader(http.Header{"Content-Type": []string{"application/json; charset=utf-8"}}).
	CmpJSONBody(td.JSON(`{"errno":0, "msg":"Hello Longyue", "data":{}}`))
}

func (s *APISuite) TestFakeHello(t *td.T) {
	t.Cmp(false, true)
}

Run the test suite, you will get:

--- FAIL: TestAPISuite (0.00s)
    --- FAIL: TestAPISuite/TestFakeHello (0.00s)
      ~/Workspaces/vscode/learn-by-test/api/api/bcd_test.go:67: Failed test
            DATA: values differ
            	     got: false
            	expected: true
    --- FAIL: TestAPISuite/Wrong_Method (0.00s)
     ~/Workspaces/vscode/learn-by-test/api/api/bcd_test.go:50: Failed test 'body contents is OK'
            Response.Body["msg"]: values differ
            	     got: "Method not allowed"
            	expected: "Method not allowed1"
            [under operator JSON at bcd_test.go:50]
   ~/Workspaces/vscode/learn-by-test/api/api/bcd_test.go:56: Failed test 'Get /hello subtest: body contents is OK'
        Response.Body["msg"]: values differ
        	     got: "Method not allowed"
        	expected: "Method not allowed1"
        [under operator JSON at bcd_test.go:56]
  ~/Workspaces/vscode/learn-by-test/api/api/bcd_test.go:63: Failed test 'Get /hello subtest: body contents is OK'
        Response.Body["msg"]: values differ
        	     got: "Hello Longyue"
        	expected: "Hello longyue"
        [under operator JSON at bcd_test.go:63]
FAIL

Explain:

1. the error message of

func (s *APISuite) TestFakeHello(t *td.T) {
	t.Cmp(false, true)
}

is expected and got:

    --- FAIL: TestAPISuite/TestFakeHello (0.00s)
      ~/Workspaces/vscode/learn-by-test/api/api/bcd_test.go:67: Failed test
            DATA: values differ
            	     got: false
            	expected: true

everything is ok!

2. the error message of

func (s *APISuite) TestHello(t *td.T) {
	
	s.ta.Run("Wrong Method", func(ta *tdhttp.TestAPI) {
		ta.Get("/hello").
			CmpStatus(http.StatusMethodNotAllowed).
			CmpHeader(http.Header{"Content-Type": []string{"application/json; charset=utf-8"}}).
			CmpJSONBody(td.JSON(`{"errno":1, "msg":"Method not allowed1", "data":{}}`))
	})
       // ......
}

is expected:

    --- FAIL: TestAPISuite/TestHello/Wrong_Method (0.00s) // <--
     ~/Workspaces/vscode/learn-by-test/api/api/bcd_test.go:50: Failed test 'body contents is OK'
            Response.Body["msg"]: values differ
            	     got: "Method not allowed"
            	expected: "Method not allowed1"
            [under operator JSON at bcd_test.go:50]

but got:

    --- FAIL: TestAPISuite/Wrong_Method (0.00s) //<--
     ~/Workspaces/vscode/learn-by-test/api/api/bcd_test.go:50: Failed test 'body contents is OK'
            Response.Body["msg"]: values differ
            	     got: "Method not allowed"
            	expected: "Method not allowed1"
            [under operator JSON at bcd_test.go:50]

comparing //<-- part of error message, missing the test name TestHello

3. the error message of

func (s *APISuite) TestHello(t *td.T) {
	
	// ......

	s.ta.Get("/hello").
		Name("Get /hello subtest").
		CmpStatus(http.StatusMethodNotAllowed).
		CmpHeader(http.Header{"Content-Type": []string{"application/json; charset=utf-8"}}).
		CmpJSONBody(td.JSON(`{"errno":1, "msg":"Method not allowed1", "data":{}}`))

	// ......
}

is expected:

 --- FAIL: TestAPISuite/TestHello  //<--
   ~/Workspaces/vscode/learn-by-test/api/api/bcd_test.go:56: Failed test 'Get /hello subtest: body contents is OK'
        Response.Body["msg"]: values differ
        	     got: "Method not allowed"
        	expected: "Method not allowed1"
        [under operator JSON at bcd_test.go:56]

or

 --- FAIL: TestAPISuite/TestHello/Get_/hello_subtest   // <--
   ~/Workspaces/vscode/learn-by-test/api/api/bcd_test.go:56: Failed test 'body contents is OK'.  // <--
        Response.Body["msg"]: values differ
        	     got: "Method not allowed"
        	expected: "Method not allowed1"
        [under operator JSON at bcd_test.go:56]

but got:

   ~/Workspaces/vscode/learn-by-test/api/api/bcd_test.go:56: Failed test 'Get /hello subtest: body contents is OK' //<--
        Response.Body["msg"]: values differ
        	     got: "Method not allowed"
        	expected: "Method not allowed1"
        [under operator JSON at bcd_test.go:56]

comparing //<-- part of error message, missing the test name TestHello. and the test name 'Get /hello subtest' may be out of place.

4. the error message of

func (s *APISuite) TestHello(t *td.T) {
	
	// ......
        s.ta.PostForm("/hello", url.Values{
		"name": []string{"Longyue"},
	}).
	CmpStatus(http.StatusOK).
	CmpHeader(http.Header{"Content-Type": []string{"application/json; charset=utf-8"}}).
	CmpJSONBody(td.JSON(`{"errno":0, "msg":"Hello Longyue", "data":{}}`))
}

is expected:

 --- FAIL: TestAPISuite/TestHello  // <--
  ~/Workspaces/vscode/learn-by-test/api/api/bcd_test.go:63: Failed test 'body contents is OK' // <--
        Response.Body["msg"]: values differ
        	     got: "Hello Longyue"
        	expected: "Hello longyue"
        [under operator JSON at bcd_test.go:63]

but got:

  ~/Workspaces/vscode/learn-by-test/api/api/bcd_test.go:63: Failed test 'Get /hello subtest: body contents is OK' // <--
        Response.Body["msg"]: values differ
        	     got: "Hello Longyue"
        	expected: "Hello longyue"
        [under operator JSON at bcd_test.go:63]

• Get /hello subtest is the previous s.ta.Get("/hello") test's name, this is a s.ta.PostForm
• comparing //<-- part of error message, missing test name of TestHello

Sorry for my poor English. Thx

Expose some internal types, so custom operators can be built

The request in a nutshell: can enough internal details be exposed to allow building custom operators externally? Ideally in a stable way.

I very much realize this is not a simple request, but someone has to make it :) Maybe it can serve as a placeholder for planning / discussion. Some context / broader reasoning is below.

---

TestDeep looks super interesting! From a quick glance it looks like there are a lot of good, flexible, well-thought-out features, especially around good error messages (that's so rare, thank you!).

While there are quite a few built-in operators, there are always some edge cases that benefit from custom ones. The JSON operators serve as a nice example of this: it's just a string, but there's a lot of structure that can imply much-better error reporting and a simpler API, so it has a custom operator to improve things.
Unfortunately, though td/types.go exposes the TestDeep interface, the base type is not exposed, and location.GetLocationer deals with the (afaict) internal-only Location type, so custom operators cannot be built outside this library. There may be other causes as well, this one was just one I saw while reading.

This is... probably a very reasonable choice for API stability, but somewhat crippling for less-common needs. I assume you don't want to collect every ill-advised and poorly-implemented operator that every user/company/etc dreams up, so being able to build things from the outside would be great. It could also enable "enriching" libraries that enhance TestDeep.

Is there a way this can be done?

Add (*T).Run() method

anchors not working with table driven tests / multiple t.Cmp calls

I'm building some Stripe tests with large, deeply nested structs, where using anchors is very helpful. They work great when I have a single test, but if I'm calling t.Cmp multiple times or have table driven tests, anchors are being compared against the wrong values. Since the anchors have to be set outside the inner test run loop, I'm not sure how td knows when to iterate between them. Am I doing something incorrectly?

Table driven test

func Test_payInvoiceWithStripe(stdT *testing.T) {
	tdt := td.NewT(stdT)

	tests := []struct {
		name                  string
		stripeCustomerID      string
		stripePaymentMethodID string
		wantPaymentIntent     *stripe.PaymentIntent
		wantErr               error
	}{
		// lots of tdt.A
	}

	stripeClient := client.New(apiKey, nil)
	invoice := &billingpb.Invoice{
		InvoiceId:            111,
		AccountId:            22222,
	}

	for i, tt := range tests {
		tdt.Run(tt.name, func(t *td.T) {
			gotPaymentIntent, err := chargeCustomer(context.Background(), stripeClient, tt.stripeCustomerID, tt.stripePaymentMethodID, invoice, int64(i))
			if tt.wantErr == nil {
				t.CmpNoError(err)
			} else {
				t.Cmp(e.Cause(err).(*stripe.Error), e.Cause(tt.wantErr).(*stripe.Error))
			}
			t.Cmp(gotPaymentIntent, tt.wantPaymentIntent)
		})
	}

Output

        DATA.ChargeID: values differ
        	     got: "ch_3KH0DO4HQ2O7mN7V1o3cxNfi"
        	expected: "<testdeep@anchor#12>"

Thanks for a great library!

Use runtime.CallersFrames to avoid ignoring testdeep inlined functions

See https://tip.golang.org/doc/go1.12#compiler
Currently, some pragmas (//go:noinline) are used to avoid some testdeep functions to be inlined and to avoid the reported caller file/function be wrong.
Using runtime.Frames via runtime.CallersFrames instead of runtime.Caller should allow us to remove these pragmas.

Add Keys & Values smuggler operators

any map → keys & values list

Add IgnoreUnreachable ContextConfig flag

When the unsafe package is not available, go-testdeep tries hard to access private values by copying them before accessing them. But it does not work in all cases and in failing cases, it panics.

Using this flag (false by default), it would avoid panicking and skip the test.

Add new operators SubJSONOf and SuperJSONOf

Based on SubMapOf and SuperMapOf but should work for arrays too (or not).

Ignore unexported struct fields?

I want to compare all fields in two large structs, except the unexported fields. Is there any way to achieve that without manually specifying each field in a SStruct? I might be missing an operator/helper function, but the best solution that comes to my mind is to compare the JSON representation of the two structs.

Example for clarification:

type X struct {
  Field1      int
  Field2      int
  Field3      int
  Field4      int
  Field5      int
  fieldIgnore int // <--- want to ignore this when doing a t.Cmp(x1, x2)
}