test-more / test2-suite Goto Github PK

View Code? Open in Web Editor NEW

26.0 26.0 46.0 3.2 MB

Tools, bundles, and plugins for Test2

License: Other

Perl 99.37% Raku 0.63%

test2-suite's Introduction

NAME

Test2 - Framework for writing test tools that all work together.

DESCRIPTION

Test2 is a new testing framework produced by forking Test::Builder, completely refactoring it, adding many new features and capabilities.

WHAT IS NEW?

Easier to test new testing tools.

From the beginning Test2 was built with introspection capabilities. With Test::Builder it was difficult at best to capture test tool output for verification. Test2 Makes it easy with Test2::API::intercept().
Better diagnostics capabilities.

Test2 uses an Test2::API::Context object to track filename, line number, and tool details. This object greatly simplifies tracking for where errors should be reported.
Event driven.

Test2 based tools produce events which get passed through a processing system before being output by a formatter. This event system allows for rich plugin and extension support.
More complete API.

Test::Builder only provided a handful of methods for generating lines of TAP. Test2 took inventory of everything people were doing with Test::Builder that required hacking it up. Test2 made public API functions for nearly all the desired functionality people didn't previously have.
Support for output other than TAP.

Test::Builder assumed everything would end up as TAP. Test2 makes no such assumption. Test2 provides ways for you to specify alternative and custom formatters.
Subtest implementation is more sane.

The Test::Builder implementation of subtests was certifiably insane. Test2 uses a stacked event hub system that greatly improves how subtests are implemented.
Support for threading/forking.

Test2 support for forking and threading can be turned on using Test2::IPC. Once turned on threading and forking operate sanely and work as one would expect.

GETTING STARTED

If you are interested in writing tests using new tools then you should look at Test2::Suite. Test2::Suite is a separate cpan distribution that contains many tools implemented on Test2.

If you are interested in writing new tools you should take a look at Test2::API first.

NAMESPACE LAYOUT

This describes the namespace layout for the Test2 ecosystem. Not all the namespaces listed here are part of the Test2 distribution, some are implemented in Test2::Suite.

Test2::Tools::

This namespace is for sets of tools. Modules in this namespace should export tools like ok() and is(). Most things written for Test2 should go here. Modules in this namespace MUST NOT export subs from other tools. See the "Test2::Bundle::" namespace if you want to do that.

Test2::Plugin::

This namespace is for plugins. Plugins are modules that change or enhance the behavior of Test2. An example of a plugin is a module that sets the encoding to utf8 globally. Another example is a module that causes a bail-out event after the first test failure.

Test2::Bundle::

This namespace is for bundles of tools and plugins. Loading one of these may load multiple tools and plugins. Modules in this namespace should not implement tools directly. In general modules in this namespace should load tools and plugins, then re-export things into the consumers namespace.

Test2::Require::

This namespace is for modules that cause a test to be skipped when conditions do not allow it to run. Examples would be modules that skip the test on older perls, or when non-essential modules have not been installed.

Test2::Formatter::

Formatters live under this namespace. Test2::Formatter::TAP is the only formatter currently. It is acceptable for third party distributions to create new formatters under this namespace.

Test2::Event::

Events live under this namespace. It is considered acceptable for third party distributions to add new event types in this namespace.

Test2::Hub::

Hub subclasses (and some hub utility objects) live under this namespace. It is perfectly reasonable for third party distributions to add new hub subclasses in this namespace.

Test2::IPC::

The IPC subsystem lives in this namespace. There are not many good reasons to add anything to this namespace, with exception of IPC drivers.

Test2::IPC::Driver::

IPC drivers live in this namespace. It is fine to create new IPC drivers and to put them in this namespace.

Test2::Util::

This namespace is for general utilities used by testing tools. Please be considerate when adding new modules to this namespace.

Test2::API::

This is for Test2 API and related packages.

Test2::

The Test2:: namespace is intended for extensions and frameworks. Tools, Plugins, etc should not go directly into this namespace. However extensions that are used to build tools and plugins may go here.

In short: If the module exports anything that should be run directly by a test script it should probably NOT go directly into Test2::XXX.

CONTACTING US

Many Test2 developers and users lurk on irc://irc.perl.org/#perl-qa and irc://irc.perl.org/#toolchain. We also have a slack team that can be joined by anyone with an @cpan.org email address https://perl-test2.slack.com/ If you do not have an @cpan.org email you can ask for a slack invite by emailing Chad Granum [email protected].

SOURCE

The source code repository for Test2 can be found at https://github.com/Test-More/test-more/.

MAINTAINERS

Chad Granum [email protected]

AUTHORS

Chad Granum [email protected]

COPYRIGHT

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

See https://dev.perl.org/licenses/

test2-suite's People

Contributors

Stargazers

Watchers

Forkers

jkeroes cpanxaoc desertfox karenetheridge 2shortplanks grinnz haarg yanick mikeraynham lharey akiym nawglan cv-library spazm tokubass manwar pkg-perl-tools lancew atoomic j1n3l0 robkinyon colinnewell m-dango nephila-nacrea dkechag mjdominus clayne toddr choroba jonjensen nanto fgasper larryl dboehmer wolfsage tobyink leonerd mauke michal-josef-spacek mikkoi jkeenan trwyant sysfce2 garu yoshikazusawa haarg-contrib

test2-suite's Issues

Allow multiple checks to be combined.

For example...

# It's not an empty string, nor is it a reference.
is { foo => 23 }, hash { field foo => !empty_string && !reference };

# It's either an empty string, or it's a reference.
is { foo => [] }, hash { field foo => empty_string || reference };

Unfortunately && and || cannot be overloaded. However, bool can (ie. being used in boolean context) and so can & and |. I don't know enough about how checks work to know what's the best way to handle this.

This idea came out of #73 in this comment.

A check for strings which are only whitespace or empty.

From #73, this would check for something that stringifies to only whitespace, or the empty string.

These match:

"\t"
" "
"\n"
"\x{02006}"
""

These do not:

0
"foo"
[]

The question is what the name should be? I thought only_whitespace but an empty string contains no whitespace.

What to do about isnt()

isnt() is fairly useless currently. It simply passes if $left does not match $right in any way.

discuss....

Should we deprecate using tools/bundles without pins?

I am working on updating all Test2::Suite packages that export symbols. All of them will be updated to use an upcoming feature in Importer called 'pins'. A pin is a way to version exports from a module, the main example is Compre:

use Test2::Tools::Compare ':v1';

use Test2::Tools::Compare ':v2';

The first will import is() that does not automatically add bounds-checking to checks built with the dsl. The second one will import the updated is() that does bound checking. providing no pin will default to ':v1' for back-compat, don't want to break things.

Question is, should we add a warning whenever someone loads a bundle/tool without specifying a pin? I think we should, that way everyone will need to add :v1 or :v2, and will be forced to know about pins. This way they can also always check if there is a newer pin that provides updated features.

I do not think we should warn when people are using older pins, that almost defeats their purpose, though I think an env var to turn on such warnings would be useful.

I am leaning towards warning on no-pin with bundles, I am not so sure on tools:: packages..

Please make plan() combatible with old-style plan()

All my subtests have plans.

subtest foo => sub {
    plan tests => 14;
}

In order to use Test2::Tools::Basic, I would have to change all of those to:

subtest foo => sub {
    plan 14;
}

If you'd allow either

plan tests => 14;
plan 14;

then it would save me from changing 1500 plans in 400 different files.

It would also be nice if we could use

use Test2::Bundle::Extended plan => 14;

just like we currently do

use Test::More plan => 14;

Add a comparison function to Test2::Tools::Compare for two boolean values

I've got a function that I've been calling bool_eq that compares truthiness. Consider this example, probably in the middle of a big loop that goes over a bunch of different test cases.

my $thingy = find_a_thingy();  # Either returns a Thingy object or undef.
if ( $expect_to_find_a_thingy ) {  # Are we expecting to find one?
    ok( $thingy, 'Found a thingy' );
}
else {
    ok( !$thingy, 'Should not find a thingy' );
}

bool_eq lets that be rewritten as:

bool_eq( find_a_thingy(), $expect_to_find_a_thingy );

Or something like:

bool_eq( scalar @errors, $should_have_errors );

The diagnostics on failure would tell what values came in and how they evaluated:

#   Failed test 'Should have no errors'
#   at foo.t line 8.
#          got: true (5)
#     expected: false (0)

#   Failed test 'Should have found at least one user'
#   at foo.t line 10.
#          got: false (0)
#     expected: true (1)

#   Failed test 'Should not find a Mech object'
#   at foo.t line 9.
#          got: true (WWW::Mechanize=HASH(0x3cca328))
#     expected: false (0)

Create tools for defining composite checks

Core is going to focus on composable checks. We need to create tools that make composing them easier:

Define a sub that returns the specified composite check with the right file+line when called. Essentially it closed the check defined by the ... and updates the file+line of the clone.
compose_check name => ...;

Add `items` for `array` and `bag`. `fields` for `hash`.

Like item, items tags a list of items. This makes specifying arrays so much easier.

array { item 1; item 2; item "oh god"; item "please stop"; item "my fingers hurt"; };

array { items 1, 2, 3, 4, 5, 6 };

By allowing it in bag, it lets you easily create a bag from an existing array (see #57).

is $array, bag { items @other_array };

Similarly, fields lets you add multiple fields in hash and object.

hash { field foo => 23; field bar => 42; field too => "verbose" };

hash { fields foo => 23, bar => 42, baz => 99 };

Like with bag, it makes it easy to create a hash spec from other hashes.

like \%thing, hash { fields %expect };

@exodist suggested this.

Create a document of differences and improvements to help those interested in migrating

This ticket is a dumping ground for ideas of what to put in to such a document.

Speedups
Parallelization
Preloading in yath: (yath can preload modules before forking and running the tests. In a typical large business test suite that loads a ton of modules it avoids having to do all that loading for every test.
Flexibility of is and like, especially for testing structures.

Test2::Compare may need to load Test2::Compare::{Hash,Array,etc...}

run this script

use Test2::Tools::ClassicCompare;# or use Test2::Bundle::More;
is_deeply({},{})

then I get module load error.

Can't locate object method "new" via package "Test2::Compare::Hash" (perhaps you forgot to load "Test2::Compare::Hash"?) at lib/perl5/Test2/Compare.pm line 88.
A context appears to have been destroyed without first calling release().
Based on $@ it does not look like an exception was thrown (this is not always
a reliable test)

This is a problem because the global error variables ($!, $@, and $?) will
not be restored. In addition some release callbacks will not work properly from
inside a DESTROY method.

Here are the context creation details, just in case a tool forgot to call
release():
  File: ./basic_sample.pl
  Line: 2
  Tool: Test2::Tools::ClassicCompare::is_deeply

However, Test2-Suite/t/modules/Tools/ClassicCompare.t is a success.
Because this test loads Test2::Bundle::Extended.
(
Test2::Bundle::Extended loads Test2::Tools::Compare. Test2::Tools::Compare loads Test2::Compare::Hash, Test::Compare::Array, and so on
)

Add an all_items(CHECK) for arrays

Most of my code lives in checks. This works nicely, especially when validating deep hashes.

However, I've found that it's a bit awkward when it comes to arrays - currently, array checks can only be called manually. This doesn't work when you don't know how many items in an array before executing.

What I'd like, is something along the lines of:

my $is_below_ten = validator(sub {
  my (%params) = @_;

  return undef if $params{got} < 10;
});

is(
  [3, 4, 5],
  array {
    all_items $is_below_ten;
  }
);

Make the difference between `is` and `like` easier to understand.

#56 and #62 bring up interesting questions about the behavior of is and like. The docs say is is the "strict checker" and like is the "relaxed checker". And I've been thinking of them this way. is is an equality check and like says we expect it to contain the selected things.

But upon reading the docs there's a lot more than that and some of them are quite surprising.

use Test2::Bundle::Extended;

use v5.10;

# This is "foo" =~ qr/foo/
like "foo", qr/foo/, "like regex";               # pass

# This is "foo" eq qr/foo/ ... or something?
is "foo",   qr/foo/,   "is regex";               # fails

# This uses the code ref to check 'foo'
like "foo", sub { $_ eq 'foo' }, 'like coderef'; # pass

# This checks if "foo" is the code ref?
is   "foo", sub { $_ eq 'foo' }, 'is coderef';   # fails

done_testing;

These is features are for extremely rare cases. They complicate understanding is vs like. The like ones are pretty useful, being able to check strings with regexes and evaluating complicated nested conditions with code refs. It sucks that you can only get them with like. You can't choose "strict checking but also let me use regexes and code".

I propose...

Make is and like behave as similar as possible
- Eliminate the code ref and regex and any other special cases
- Replace them with coderef and regex DSLs (see below).
is checks that the two sides are the same
like checks that the LHS contains the RHS

The special case behavior of is is replaced by adding to the DSL. Then you can use it with like if you want.

is $code, codref $some_code_ref;  # check that $code is $some_code_ref
is $regex, regex qr{foo};  # check $regex is the same as qr{foo}

Yes, this does call into question what is really the difference between is and like.

Yes, I'm increasingly thinking of Test2 as a DSL rather than a collection of comparison functions.

array, bag, and hash should add an implicit `end` when used with `is`.

Test2::Tools::Compare says right in the SYNOPSIS that is is "strict checking, everything must match".

    # Strict checking, everything must match
    is(
        $some_hash,
        {a => 1, b => 2, c => 3},
        "The hash we got matches our expectations"
    );

But with array, bag, hash that expectation doesn't hold true.

use Test2::Bundle::Extended;

my $array = [1,2,3,4];

is $array, array { item 1; item 2; item 3; end; }, "array w/end";          # fail
is $array, [1,2,3],                                "array ref";            # fail
is $array, array { item 1; item 2; item 3; },      "array wo/end";         # pass

is $array, bag { item 2; item 3; item 1; end; },   "bag w/end";            # fail
is $array, bag { item 2; item 3; item 1; },        "bag wo/end";           # pass


my $hash = { foo => 23, bar => 42, baz => 99 };

is $hash, { foo => 23, bar => 42 },                "hash ref";             # fail
is $hash, hash { field foo => 23; field bar => 42; end; }, "hash w/end";   # fail
is $hash, hash { field foo => 23; field bar => 42 },       "hash wo/end";  # pass

done_testing;

This is a trap for the user. It's unexpected for the writer and the reader. is is for equality. like is for matches. Having an is that does a match breaks that expectation.
The mistake is invisible. When the user inevitably forgets end the test will silently pass. They won't be aware of your mistake.
It breaks the expectation that [1,2,3] and array { item 1; item 2; item 3; } are equivalent.

It's caught me plenty of times, probably all my code that uses hash or array forgot to use end.

To avoid this, I would suggest that is add an implicit end to any comparison which can take it. This avoids a very common mistake, and it removes a very common exception to the behavior of is vs like.

As for backwards compatibility, I think we'll find that most folks are screwing this up too. It's early enough in Test2's lifespan that we can fix this before it spreads.

Documentation in Test2::Require::Perl has bad synopsis

https://metacpan.org/source/EXODIST/Test2-Suite-0.000016/lib/Test2/Require/Perl.pm#L43

=head1 SYNOPSIS

    # Skip the test unless perl 5.10 or greater is installed.
    use Test2 Perl 'v5.10';

    # Enable 5.10 features.
    use v5.10;

Feature request: U()

The D() quick check is useful, but I find myself wanting the opposite - U(). As it doesn't exist yet, the following works but is a bit tedious:

not_in_set(D())

array, bag, and hash are vulnerable to a very easy mistake

use Test2::Bundle::Extended;

is {foo => 23}, hash  { bar => 42 };  # passes?!
is [1,2,3],     array { 5, 6, 7 };    # passes?!
is [1,2,3],     bag   { 5, 6, 7 };    # passes?!

done_testing;

Those all pass. I understand technically why, they're all equivalent to an empty block, but it's very confusing to the user.

I'm not sure what to do about it, but #56 will help.

string() confesses when not defined

The docs say:

Note: This will fail if the recieved value is undefined, it must be defined.

So when using string() as a quick check, it confesses when not defined. Shouldn't it simply return false if it is not a string?

Test2::Require::Module should act like Test::Needs

Some notable differences ...

Works around some brain damage in earlier Perl versions
Exports a test_needs sub you can use in subtests, which is really nice

I think that's it, but @haarg might know other differences.

Checking An Empty ArrayRef Against A bag Always Passes

It appears to be a problem with Test2::Compare::Bag::deltas but I couldn't devote further time to fixing it.

use Test2::Bundle::Extended;
use Test2::Tools::Compare qw( is bag item );

my $check = bag {
    item 'a';
    item 'b';
    end(); # Ensure no other elements exist.
};

is( [ ], $check, 'All of the elements from bag found!'); # passes but shouldn't

done_testing;

Explicitly specify the license, not "same terms as Perl itself"

I don't think there's any benefit in the "same terms as Perl itself" wording, and can only cause confusion. If it's being licensed as Artistic License 2.0, then I think it makes sense to go ahead and specify it.

http://use.perl.org/use.perl.org/_Shlomi%2BFish/journal/36050.html has some discussion of this.

Add contains()

is() says "what we got is exactly what we expect".

contains() says "what we got contains what we expect".

is [1,2,3,4], [2,3];  # fail
contains [1,2,3,4], [2,3];  # pass

is "foofer", "oof";  # fail
contains "foofer", "oof";  # pass

The idea is to provide a DSL like is and like where the difference in behavior between the two are clearly understood. is and contains should differ in just one specific way: one is an equality check, the other is a match.

There are plenty of open questions about how this should work and how far it should go.

See #63 for background.

Please add prototypes to the comparison functions

Function prototypes are a huge benefit when writing tests, because it's so easy to get the arguments goofed up. You're thinking cmp_ok when writing is, and you do this:

is( 'dog', '==', 'cat', 'Are dogs the same as cats?' );

Under Test2, the test will fail but the code will execute.

Using Test::More I would get:

Too many arguments for Test::More::is at foo.pl line 7, near "'Are dogs the same as cats?' )"
Execution of foo.pl aborted due to compilation errors.

Anything we can do to catch errors at compile time is a big win.

[PATCH] fix spelling

In Debian we are currently applying the following patch to
Test2-Suite.
We thought you might be interested in it too.

Description: fix spelling
Origin: vendor
Author: gregor herrmann <[email protected]>
Last-Update: 2016-07-30

The patch is tracked in our Git repository at
https://anonscm.debian.org/cgit/pkg-perl/packages/libtest2-suite-perl.git/plain/debian/patches/spelling.patch

Thanks for considering,
gregor herrmann,
Debian Perl Group

Test2::Compare does not load Test2::Compare::Array or ::Hash

#!/var/perl/bin/perl

use warnings;
use strict;

use Test2::Bundle::More;

is_deeply( { foo => 1 }, { bar => 2 } );
done_testing();

generates this

$ prove Dev/test2.t
[11:03:15] Dev/test2.t .. Can't locate object method "new" via package "Test2::Compare::Hash" (perhaps you forgot to load "Test2::Compare::Hash"?) at /var/perl5.20.3/lib/site_perl/5.20.3/Test2/Compare.pm line 87.
A context appears to have been destroyed without first calling release().
Based on $@ it does not look like an exception was thrown (this is not always
a reliable test)

This is a problem because the global error variables ($!, $@, and $?) will
not be restored. In addition some release callbacks will not work properly from
inside a DESTROY method.

Here are the context creation details, just in case a tool forgot to call
release():
  File: Dev/test2.t
  Line: 8
  Tool: Test2::Tools::ClassicCompare::is_deeply

Cleaning up the CONTEXT stack...
# No tests run!
# Looks like your test exited with 255 after test #0.
[11:03:15] Dev/test2.t .. Dubious, test returned 255 (wstat 65280, 0xff00)
No subtests run 
[11:03:15]

Test Summary Report
-------------------
Dev/test2.t (Wstat: 65280 Tests: 0 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
Files=1, Tests=0,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.05 cusr  0.00 csys =  0.07 CPU)
Result: FAIL

Similar results when doing array comparisons.

$ cat Dev/test2.t
#!/var/perl/bin/perl

use warnings;
use strict;

use Test2::Bundle::More;

is_deeply( [1], [2] );
done_testing();

$ prove Dev/test2.t
[11:04:16] Dev/test2.t .. Can't locate object method "new" via package "Test2::Compare::Array" (perhaps you forgot to load "Test2::Compare::Array"?) at /var/perl5.20.3/lib/site_perl/5.20.3/Test2/Compare.pm line 84.
A context appears to have been destroyed without first calling release().
Based on $@ it does not look like an exception was thrown (this is not always
a reliable test)

This is a problem because the global error variables ($!, $@, and $?) will
not be restored. In addition some release callbacks will not work properly from
inside a DESTROY method.

Here are the context creation details, just in case a tool forgot to call
release():
  File: Dev/test2.t
  Line: 8
  Tool: Test2::Tools::ClassicCompare::is_deeply

Cleaning up the CONTEXT stack...
# No tests run!
# Looks like your test exited with 255 after test #0.
[11:04:16] Dev/test2.t .. Dubious, test returned 255 (wstat 65280, 0xff00)
No subtests run 
[11:04:16]

Test Summary Report
-------------------
Dev/test2.t (Wstat: 65280 Tests: 0 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
Files=1, Tests=0,  0 wallclock secs ( 0.03 usr  0.00 sys +  0.04 cusr  0.01 csys =  0.08 CPU)
Result: FAIL

Create a "how to work on Test2::Suite" document

Start with explaining that you have to use Dist::Zilla, not EU::MM, and how to use Dist::Zilla to do common tasks. Not everyone uses it, and without a Makefile.PL, someone unfamiliar with Dist::Zilla will not understand what to do.

The Exception tools should provide something like lives_ok

If I expect something to live, I really want to get diagnostic of the exception when it doesn't. The lives or note idiom is really suboptimal.

Even if you don't want to add this, I'd at least update the docs to suggest something like is( exception { ... }, undef, ... )

The Class tools do not allow test description

None pof isa_ok, can_ok, or DOES_ok allow a test description. This makes for a weird and easily forgettable API, where they are the outliers among all the test functions.

This wasn't a good idea for Test::More, and I'd hate to see Test2::Suite repeat that mistake. I think it'd be preferable to accept a single class or arrayref of classes (or methods, etc) and allow the third argument to be the test description.

Clearly distinguish between strings and metavalues.

Examples of the problem.

$ perl -wle 'use Test2::Bundle::Extended; is "\tfoo", "\\tfoo"; done_testing'
# Seeded srand with seed '20160814' from local date.
not ok 1
# Failed test at -e line 1.
# +-------+----+-------+
# | GOT   | OP | CHECK |
# +-------+----+-------+
# | \tfoo | eq | \tfoo |
# +-------+----+-------+
1..1

$ perl -wle 'use Test2::Bundle::Extended; is undef, "<UNDEF>"; done_testing'
# Seeded srand with seed '20160814' from local date.
not ok 1
# Failed test at -e line 1.
# +---------+---------+
# | GOT     | CHECK   |
# +---------+---------+
# | <UNDEF> | <UNDEF> |
# +---------+---------+
1..1

Test::More did this by using quotes to indicate what was a string and what was a meta value.

$ perl -wle 'use Test::More; is undef, "<UNDEF>"; done_testing'
not ok 1
#   Failed test at -e line 1.
#          got: undef
#     expected: '<UNDEF>'
1..1
# Looks like you failed 1 test of 1.

$ perl -wle 'use Test::More; is "\tfoo", "\\tfoo"; done_testing'
not ok 1
#   Failed test at -e line 1.
#          got: '   foo'
#     expected: '\tfoo'
1..1
# Looks like you failed 1 test of 1.

I think what will help is to combine Test::More's approach with Test2's.

Escape all invisible and ambiguous characters.
Display strings as if they were double quoted (including escapes).
All metavalues, like undef, are unquoted.

For is "\tfoo", "\\tfoo"; that would be...

# +---------+----+----------+
# | GOT     | OP | CHECK    |
# +---------+----+----------+
# | "\tfoo" | eq | "\\tfoo" |
# +---------+----+----------+

For is undef, "<UNDEF>" that would be...

# +---------+-----------+
# | GOT     | CHECK     |
# +---------+-----------+
# | <UNDEF> | "<UNDEF>" |
# +---------+-----------+

For is "\$foo", "foo"; that would be...

# +---------+----+-------+
# | GOT     | OP | CHECK |
# +---------+----+-------+
# | "\$foo" | eq | "foo" |
# +---------+----+-------+

See Test-More/test-more#6 for some more background.

'st', 'stc', 'stcs', 'sta', and 'stp' dsl subs

Instead of adding a ton of subs, with possible inconsistencies, we should define these 5 base dsl subs. They should be used consistently, and Compare::XXX types should know what to do with them (consistent!!!)

is(
    $structure,
    st Hash => sub {
        my $st = shift;
        # or
        my $st = st_this;

        stc key1 => 'val1';
        stc key2 => 'val2';

        stc nested => st Array => { sta empty => 1 };

        stc non_empty_string => st String => { sta empty => 0 };

        stc price => st Number => 12; 

        sta 'end'; # Or to be more explicit:
        sta end => 1; # or stp etc => 1,
    },  
    "name"
);

st - define a structure

st Type Val
st Type &Builder
to define a Test2::Compare::Type check

stc - structure component;

stc key => 'val' # like 'field'
stc idx => 'val' # for arrays (like item with an index)
stc 'val' # for arrays (like item with no idx)
stc $val # String/number/etc.
to define components inside the structure

stcs - structure values

stcs ...; # Specify several components at once

sta - structure attributes

sta 'attr' - toggle attribute on
sta attr => $val - set attribute to value
This is essentially calling a method on the check when used in a builder to toggle attributes

stp - structure property

stp what => val
This would be like a 'prop' to check properties of the thing you got.

Create a check "nonblank"

In the old system, I have many tests that are of the form

is_nonblank( $foo->{name} );

which is effectively

ok( !ref($foo->{name}) && length($foo->{name}) > 0 );

And we should be able to do

is( $foo, hash { field name => nonblank } );

DNE() with in_set() fails

Maybe I'm doing something wrong, but I would have expected that all of these tests should pass:

#!/usr/bin/perl

use strict;
use warnings;

use Test2::Bundle::Extended;

my $check = hash {
  field first   => 42;
  field second  => undef;
  field third   => DNE();
  field fourth  => in_set(42, undef);
  field fifth   => in_set(42, undef);
  field sixth   => in_set(42, DNE());
  field seventh => in_set(42, DNE());
};

is(
  {
    first   => 42,
    second  => undef,
    # third DNE
    fourth  => 42,
    fifth   => undef,
    sixth   => 42,
    # seventh DNE <- fails :(
  },
  $check
);

Use v1/v2 tagging for is+implicit end change

The current state of master applies the backcompat breaking change to all code. I think we need to use the planned v# export tag system instead, that is what it is there for!

Update Test2::Tools::Compare to use Importer
Current defaults, including old-style is() without implicit declare as ':v1' tag (and default when no tag specified)
':v2' tag will have a new is() with implicit end, for now it will have extra diag when implicit end causes a failure to help people migrating, the diag will eventually go away.

encoding.t flaky on older File::Temp

http://www.cpantesters.org/cpan/report/6e2476ea-3a17-11e6-9fc5-14bfbd15da07

This is more common in older perls, and only seems to happen on runs after the first. Old File::Temp versions only try a couple names before giving up. Since this test uses SRand it always has the same temp files/dirs.

The easy fix is to stop using SRand in this test. However it may be worth it to make a more complete fix (not sure how, maybe require a newer File::Temp?)

Update Negatable 'role'

make sure it is consistent with policy.

bag has issues if an item matches multiple elements

    my $check = bag {
        item match qr/a/;
        item match qr/b/;
        end();    # Ensure no other elements exist.
    };  

    is(['fab', 'bad'], $check);

In this case we get a failure cause both items match the first element, so the second element is considered 'extra' for the end check. A quick-fix would be to never re-check an element, but that is not a valid fix:

    my $check = bag {
        item match qr/a/;
        item match qr/ab/; # <---- change is here
        end();    # Ensure no other elements exist.
    };  

    is(['fab', 'bad'], $check);

This one will still fail because the less greedy bag item got 'fab' and 'ab' will not match against 'bad' but a human reader knows this should pass.

I think bag as it is currently needs a bit of an overhaul. We need to iterate over the array, not the bag items. We need to record all matches. When 'end; is not specified we just need to check that all items have at least 1 match in the array, easy. When 'end' is used we need to make sure both lists have a match in the other, but in cases where there are multiple matches we need to pick one. This can get complicated.

Discourage checking hash fields in `object`.

It's glassbox testing to say that an object must contain certain hash fields and should not be encouraged. Unfortunately the object doc example spends half its time on this.

This is particularly egregious with end. This requires that the object contain exactly certain specific fields and no others. And end only applies to certain parts of the object description complicating how the user expects end to work.

To that end...

Remove field and end from the object docs.
- Leave them working for backwards compat.
Allow hash, array, and bag to work inside object.

This eliminates a bunch of redundancy in object with hash and array, and it allows the object docs to focus on blackbox testing.

V 2.9e-05 is fu**ing ridiculous unless users are warned against using the module

Unparsable version '2.9e-05' for prerequisite Test2::Suite treated as 2.9 at /home/ron/perl5/perlbrew/perls/perl-5.20.2/lib/site_perl/5.20.2/CPAN/Meta/Requirements.pm line 136.
Warning: prerequisite Test2::Suite 2.9 not found. We have 0.000029.

I have written some more comparisons

First of all, thank you for the herculean work you've done. Test2 is much nicer to work with than Test::Builder could ever be.

Now, I found myself in need of the functionality of Test::Deep::bag, and I couldn't find something equivalent for Test2. So I wrote it: Test2::Compare::Bag, Test2::Tools::MoreCompare.

The latter also provides a call_list which behaves like call but evaluates the method in list context. list_call $method => $result is a shortcut for call(sub{[$_->$method]},$result,$method)

Should I release those packages to CPAN, or should I provide them here as a pull request? If the former, does anyone have a better name than Test2::Tools::MoreCompare?

comment srand plugin

So people don't think seeding with the date is wrong.

Compare tool docs are very outdated.

Allow `bag` to take an array ref.

is $set, bag { item 1; item 2; item 3; end; };   # annoying
is $set, bag [1,2,3];  # easy

is [sort @thing], [sort @thing];  # annoying
is \@thing, bag \@thing;  # easy

Audit all checks, they should be within policy

#73 discussed a policy, which is summarized in #77. We need to be sure all existing checks follow policy.

Tools::Compare

#30 https://github.com/Test-More/Test2-Suite/blob/master/t/modules/Tools/Compare.t#L412

Confused to what the behavior here should be, Operator is '' when checking comparison operator of a string. So no OP is returned in the message.

Add `etc`

As discussed in Slack and on #56, add etc. This explicitly states that an array or hash is not the complete specification regardless of being used by is or like.

Examples...

# After #56, both of these will be better written as `like`.
is [1,2,3], array { item 1; item 2; etc };
is { foo => 23, bar => 42 }, hash { field foo => 23; etc };

This opens the opportunity for etc to appear in other parts of an array.

# It contains 2 and 3 in that order
is [1,2,3,4,5], array { etc; item 2; item 3; etc; };

# It starts with 1 and ends with 5
is [1,2,3,4,5], array { item 1; etc; item 5; };

Please add release tests running Test::Pod and Test::Spelling

lintian (Debian's package checker) says about Test2::Suite:

I: libtest2-suite-perl: spelling-error-in-manpage usr/share/man/man3/Test2::Bundle::More.3pm.gz seperate separate
I: libtest2-suite-perl: spelling-error-in-manpage usr/share/man/man3/Test2::Bundle::More.3pm.gz compatability compatibility
I: libtest2-suite-perl: spelling-error-in-manpage usr/share/man/man3/Test2::Bundle::Simple.3pm.gz seperate separate
I: libtest2-suite-perl: spelling-error-in-manpage usr/share/man/man3/Test2::Compare::Base.3pm.gz overriden overridden
I: libtest2-suite-perl: spelling-error-in-manpage usr/share/man/man3/Test2::Compare::Number.3pm.gz recieved received
I: libtest2-suite-perl: spelling-error-in-manpage usr/share/man/man3/Test2::Compare::String.3pm.gz recieved received
I: libtest2-suite-perl: spelling-error-in-manpage usr/share/man/man3/Test2::Mock.3pm.gz overriden overridden
I: libtest2-suite-perl: spelling-error-in-manpage usr/share/man/man3/Test2::Mock.3pm.gz overriden overridden
I: libtest2-suite-perl: spelling-error-in-manpage usr/share/man/man3/Test2::Mock.3pm.gz orignal original
I: libtest2-suite-perl: spelling-error-in-manpage usr/share/man/man3/Test2::Plugin::UTF8.3pm.gz wether whether
I: libtest2-suite-perl: spelling-error-in-manpage usr/share/man/man3/Test2::Require.3pm.gz recieve receive
I: libtest2-suite-perl: spelling-error-in-manpage usr/share/man/man3/Test2::Require::Fork.3pm.gz mimick mimic
I: libtest2-suite-perl: spelling-error-in-manpage usr/share/man/man3/Test2::Suite.3pm.gz compatability compatibility
I: libtest2-suite-perl: spelling-error-in-manpage usr/share/man/man3/Test2::Tools.3pm.gz seperate separate
I: libtest2-suite-perl: spelling-error-in-manpage usr/share/man/man3/Test2::Tools::Basic.3pm.gz seperator separator
I: libtest2-suite-perl: spelling-error-in-manpage usr/share/man/man3/Test2::Tools::Basic.3pm.gz seperator separator
I: libtest2-suite-perl: spelling-error-in-manpage usr/share/man/man3/Test2::Tools::ClassicCompare.3pm.gz isnt isn't
I: libtest2-suite-perl: spelling-error-in-manpage usr/share/man/man3/Test2::Tools::Compare.3pm.gz isnt isn't
I: libtest2-suite-perl: spelling-error-in-manpage usr/share/man/man3/Test2::Tools::Compare.3pm.gz specfied specified
I: libtest2-suite-perl: spelling-error-in-manpage usr/share/man/man3/Test2::Tools::Mock.3pm.gz existance existence
I: libtest2-suite-perl: spelling-error-in-manpage usr/share/man/man3/Test2::Util::Table.3pm.gz anomolies anomalies

W: libtest2-suite-perl: manpage-has-errors-from-pod2man usr/share/man/man3/Test2::Util::Grabber.3pm.gz:218

the latter being (perldoc output)

POD ERRORS
    Hey! The above document had some coding errors, which are explained
    below:

    Around line 125:
        '=item' outside of any '=over'

    Around line 163:
        You forgot a '=back' before '=head1'

Maybe someone (@karenetheridge?) can add author tests like in Test::Simple's xt/author/ here as well?

Thanks,
gregor, Debian Perl Team

cmp_ok() with non-integer numbers gives incorrect diagnostic output

Call cmp_ok with integers and all is good.

$ perl -MTest2::Tools::ClassicCompare -E'cmp_ok( 1, ">", 4 )'
not ok 1
# Failed test at -e line 1.
# +-----+----+-------+
# | GOT | OP | CHECK |
# +-----+----+-------+
# | 1   | >  | 4     |
# +-----+----+-------+

But call it with decimals and you get this:

$ perl -MTest2::Tools::ClassicCompare -E'cmp_ok( 1.23456, ">", 4.5678 )'
not ok 1
# Failed test at -e line 1.
# +------+---------+----+--------+
# | TYPE | GOT     | OP | CHECK  |
# +------+---------+----+--------+
# | num  | 1       | >  | 4      |
# | orig | 1.23456 |    | 4.5678 |
# +------+---------+----+--------+

`meta` with Test2::Bundle::Extended has problems with Moose classes

When your test suite is a Moose class (for example, if you use Test::Class::Moose) then the exported meta function conflicts with both the inherited meta method and the meta that Moose exports, which causes problems:

package Hello;

use Test::Class::Moose bare => 1;
use Test2::Bundle::Extended;

sub test_reverse_dtrt {
    is(scalar(reverse('dlroW olleH')), 'Hello World', 'Hi');
}

__PACKAGE__->meta->make_immutable;
1;

Mark@travis:~/tmp/tcm$ prove -v runner.pl
runner.pl ..
# Seeded srand with seed '20160706' from local date.
Prototype mismatch: sub Hello/meta: none vs (&) at Hello.pm line 4.
Can't use string ("Hello") as a subroutine ref while "strict refs" in use at /opt/markperl/lib/site_perl/5.22.0/Test2/Compare.pm line 57.
 at /opt/markperl/lib/site_perl/5.22.0/Test2/Compare.pm line 59.
    Test2/Compare/build("Test2/Compare/Meta", "Hello") called at /opt/markperl/lib/site_perl/5.22.0/Test2/Tools/Compare.pm line 142
    Test2/Tools/Compare/meta("Hello") called at Hello.pm line 10
    require Hello.pm called at (eval 5) line 1
    Test/Class/Moose/Load/BEGIN() called at Hello.pm line 0
    eval {...} called at Hello.pm line 0
    eval 'use Hello ()' called at /opt/markperl/lib/site_perl/5.22.0/Test/Class/Moose/Load.pm line 54
    Test/Class/Moose/Load/_load("Test/Class/Moose/Load", "./Hello.pm", ".") called at /opt/markperl/lib/site_perl/5.22.0/Test/Class/Moose/Load.pm line 68
    Test/Class/Moose/Load/__ANON__() called at /opt/markperl/lib/5.22.0/File/Find.pm line 447
    File/Find/_find_dir(HASH(0x7fd55c99e718), ".", 4) called at /opt/markperl/lib/5.22.0/File/Find.pm line 235
    File/Find/_find_opt(HASH(0x7fd55c99e718), ".") called at /opt/markperl/lib/5.22.0/File/Find.pm line 759
    File/Find/find(HASH(0x7fd55c99e718), ".") called at /opt/markperl/lib/site_perl/5.22.0/Test/Class/Moose/Load.pm line 73
    Test/Class/Moose/Load/import("Test/Class/Moose/Load", ".") called at runner.pl line 6
    main/BEGIN() called at Hello.pm line 0
    eval {...} called at Hello.pm line 0
Compilation failed in require at (eval 5) line 1.
BEGIN failed--compilation aborted at (eval 5) line 1.
BEGIN failed--compilation aborted at runner.pl line 6.
# No tests run!
# Looks like your test exited with 255 after test #0.
Dubious, test returned 255 (wstat 65280, 0xff00)
No subtests run

Test Summary Report
-------------------
runner.pl (Wstat: 65280 Tests: 0 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
Files=1, Tests=0,  1 wallclock secs ( 0.01 usr  0.01 sys +  0.26 cusr  0.03 csys =  0.31 CPU)
Result: FAIL

Set/document policy for how checks behave

If the sub name contains a type (array, string, number, hash, object) it should only VERIFY if the thing on the left hand side is-a, or can behave as-a TYPE.
undef is a special case, even though perl can treat it as a string ('') or number (0) the core tools should not.
<does not exist> is a special case just like undef, $h->{empty} can be treated as 0 or '', but should not be by our tools.
scalar checks should treat references that do not have overloading as a special case. If you are checking for a string, but get a plain reference (without overloading) it should not match.
Core tools should be small composable units, IE they check one thing.
Minimal precombinations are allowed in core as long as they are demonstrably common (example: FDNE - false or does not exist)
Negation of a check that includes a type name should not negate the type verification, it should only negate the contents of the thing.
Pre-composing the checks in third party modules is fine, and they are free to deviate from this policy, but should probably document that they do so.

This is the policy as I defined it in #73

Explain the when & why of using is/like/hash/field/etc

The is() and like() of Test2::Tools::Compare is very different to how things are done in Test::More.

Include some explanation for people migrating to Test2 about these and other things:

When to use is() and when to use like()

# Check for empty arrayref of errors.
is( $errors, [] );  # Good
like( $errors, [] );    # Bad because it will pass if $errors = ['foo']

Also examples like:

my $employee = get_employee();
my $expected_employee = { name => ‘Bob’, dept => ‘IT’ };

# If you want to check the API, and ensure that get_employee() returns two and only two fields:
is( get_employee(), $expected_employee );

# If you want to check that you got the right record, but don’t care if it comes back with, say, a phone_number field:
like( get_employee(), $expected_employee );

When to use hash/field when constructor comparisons
Examples of using coderefs in like()

Modification of a read-only value attempted at /Test2/Compare/Delta.pm line 294.

http://www.cpantesters.org/cpan/report/ca1ad884-3a50-11e6-91c8-03b0607dd3ab

Also other cpan-testers results occasionally have this.

No obvious cause, cannot reproduce.