leont / app-prove6

A test harness script for perl6

NAME

prove6 - Run tests through a TAP harness.

USAGE

prove6 [options] [files or directories]

Boolean options

-v	--verbose	Print all test lines.
-l	--lib	Add 'lib' to the path for your tests (-Ilib).
	--shuffle	Run the tests in random order.
	--ignore-exit	Ignore exit status from test scripts.
	--reverse	Run the tests in reverse order.
-q	--quiet	Suppress some test output while running tests.
-Q	--QUIET	Only print summary results.
	--timer	Print elapsed time after each test.
	--trap	Trap Ctrl-C and print summary on interrupt.
	--help	Display this help
	--version	Display the version

Options with arguments

-I	--incdir	Library paths to include.
-e	--exec	Interpreter to run the tests ('' for compiled tests.)
	--ext	Set the extensions for tests (default <t rakutest t6>)
	--harness	Define test harness to use. See TAP::Harness.
	--reporter	Result reporter to use.
-j	--jobs	Run N test jobs in parallel (try 9.)
	--cwd	Run in certain directory
	--err=stderr	Direct the test's $*ERR to the harness' $*ERR.
	--err=ignore	Ignore test scripts' $*ERR.

NOTES

Default Test Directory

If no files or directories are supplied, prove6 looks for all files matching the pattern *.{t,t6,rakutest} under the directory t.

Colored Test Output

Colored test output is the default, but if output is not to a terminal, color is disabled.

Color support requires Terminal::ANSIColor on Unix-like platforms. If the necessary module is not installed colored output will not be available.

Exit Code

If the tests fail prove6 will exit with non-zero status.

-e

Normally you can just pass a list of Raku tests and the harness will know how to execute them. However, if your tests are not written in Raku or if you want all tests invoked exactly the same way, use the -e switch:

prove6 -e='/usr/bin/ruby -w' t/
prove6 -e='/usr/bin/perl -Tw -mstrict -Ilib' t/
prove6 -e='/path/to/my/customer/exec'

--err

• --err=stderr

  Direct the test's $*ERR to the harness' $*ERR.

  This is the default behavior.

• --err=ignore

  Ignore the test script' $*ERR

--trap

The --trap option will attempt to trap SIGINT (Ctrl-C) during a test run and display the test summary even if the run is interrupted

$*REPO

prove6 introduces a separation between "options passed to the raku which runs prove6" and "options passed to the raku which runs tests"; this distinction is by design. Thus the raku which is running a test starts with the default $*REPO. Additional library directories can be added via the RAKULIB environment variable or via the -Ilib option to prove6.

app-prove6's Issues

Add support for -I

Would it be possible to provide support for -I option to add own custom lib directories? It was in the old version of prove6.

App::Prove6 0.0.16+ broke compat with old Rakudo builds

(May be related to #16 also, although I debugged from a different angle.)

In particular, 87897cc broke compatibility with mainline Rakudo builds older than the merge of rakudo/rakudo#4917 -- Rakudo built at that merge commit works with all recent App::Prove6, while Rakudo built at the PR's branch point is broken unless reverting App::Prove6 to 9e0e2a6 .

The symptom of the broken behavior is that attempts to run prove6 in a few different variations -- such as prove -l -v t, my usual invocation -- would fail and give a usage error message (which varies slightly depending on which newer App::Prove6 build I tested with).

Unfortunately, this affects module CI against several important older versions of Rakudo (see https://gist.github.com/japhb/88322d76326df16856f31f108c90e567 for an analysis of these key releases).

prove6 needs a -b option for META6.json support

As discussed at libxml-raku/LibXML-raku#72, we need an option that takes into account META6.json instead of plain 'lib/'.

prove6 looks for files ending with '.t'

It seems that the recommended extension for Raku test files is now .rakutest.

Path::Finder not being installed

DBIish CI started failing yesterday due to the new Path::Finder requirement.

https://github.com/raku-community-modules/DBIish/runs/7721786152?check_suite_focus=true

The dependency entry in META6.json may be incorrect?

Run zef install --/test App::Prove6
===> Searching for: App::Prove6
===SORRY!=== Error while compiling /home/runner/work/DBIish/DBIish/site#sources/1209C4541C24B061C68E3544744C945ED9BEF63C (App::Prove6)
Could not find Path::Finder in:
    inst#/opt/hostedtoolcache/rakudo/2021.04-01/x86_64/share/perl6/site
    inst#/opt/hostedtoolcache/rakudo/2021.04-01/x86_64/share/perl6/vendor
    inst#/opt/hostedtoolcache/rakudo/2021.04-01/x86_64/share/perl6/core
    ap#
    nqp#
    perl5#
at /home/runner/work/DBIish/DBIish/site#sources/1209C4541C24B061C68E3544744C945ED9BEF63C (App::Prove6):5

Pod::Usage cannot be installed

Run zef install --/test App::Prove6
  zef install --/test App::Prove6
  shell: /usr/bin/bash -e {0}
===> Searching for: App::Prove6
Failed to resolve some missing dependencies
===> Searching for missing dependencies: Getopt::Long:ver<0.3.0+>, Path::Finder:ver<0.4.4+>, Pod::Usage, TAP:ver<0.3.4+>
===> Failed to find dependencies: Pod::Usage
Error: Process completed with exit code 1.

Readme wrong quoting

Readme source says under the directory <t>. which renders to under the directory .. Probably backticks should be used.

prove6 fails silently when testing JSON::Fast

Since it fails silently, I really don't have a lot of information. To reproduce, download JSON::Fast, and run zef test . on the directory. It fails with:

/test $ zef test --verbose .
===> Testing: JSON::Fast:ver<0.16>
===> Testing [FAIL]: JSON::Fast:ver<0.16>
Aborting due to test failure: JSON::Fast:ver<0.16> (use --force-test to override)

prove6 -I. in the same directory works correctly.

Refs: timo/json_fast#75 librasteve/raku-Physics-Constants#5 JJ/docker-raku-test#12

Test output is not colored when `verbose` option is used

It is colored without verbose.

Release a new version to CPAN

zef install App::Prove6 is still trying to fetch TAP:ver<0.0.8>

C:\Users\naoum\Desktop>zef install TAP
===> Searching for: TAP
===> Testing: TAP:ver<0.0.9>
===> Testing [OK] for TAP:ver<0.0.9>
===> Installing: TAP:ver<0.0.9>

C:\Users\naoum\Desktop>zef install App::Prove6
===> Searching for: App::Prove6
===> Searching for missing dependencies: TAP:ver<0.0.8>
===> Testing: TAP:ver<0.0.8>
===> Testing [OK] for TAP:ver<0.0.8>
===> Testing: App::Prove6:ver<0.0.10>:auth<cpan:LEONT>
===> Testing [OK] for App::Prove6:ver<0.0.10>:auth<cpan:LEONT>
===> Installing: TAP:ver<0.0.8>
===> Installing: App::Prove6:ver<0.0.10>:auth<cpan:LEONT>

Latest changes broke prove6 in GitLab Actions somehow?

I have GitLab actions set up like this: https://github.com/bbkr/UpRooted/blob/master/.github/workflows/test.yml

Previously everything was working perfectly (from Action log):

Run /home/runner/.raku/bin/prove6
t/10-schema.rakutest ............ ok
t/11-schema-mysql.rakutest ...... ok
t/20-tree.rakutest .............. ok
t/30-reader.rakutest ............ ok
t/31-reader-mysql.rakutest ...... ok
t/40-writer.rakutest ............ ok
t/41-writer-mysqlfile.rakutest .. ok
t/42-writer-mysql.rakutest ...... ok
All tests successful.
Files=8, Tests=47,  18 wallclock secs
Result: PASS

Now I get:

Run /home/runner/.raku/bin/prove6
    prove6 [options] [files or directories]


Boolean options
  -v  --verbose      Print all test lines.
  -l  --lib          Add 'lib' to the path for your tests (-Ilib).
      --shuffle      Run the tests in random order.

Prove6 is displaying help when ran in paramless mode. Changing it to /home/runner/.raku/bin/prove6 t/ does not help either. Something is causing https://github.com/Leont/app-prove6/blob/master/lib/App/Prove6.rakumod#L18 this signature to fail? I do not see any obvious errors.

Rename prove6

Given the Perl 6 to Raku rename, the current name of prove6 is a bit awkward. I think I would like to rename it.

Best suggestion I've heard so far is improve, but I'm open for other suggestions.

Doesn't seem to use ENV variables

I have to set export DYLD_LIBRARY_PATH="/opt/homebrew/Cellar/postgresql/14.3/lib/" to use DBIish with Postgres on my Mac. Details: raku-community-modules/DBIish#229

This solution doesn't seem to work when I run prove6 on my tests. It acts as if I do not have the environment variable set. My tests run fine if I do raku -Ilib t/001-basic.t6 or similar. If I just run prove6 I get errors described in the above issue.

Also tried prove6 -e="raku -Ilib" t/ but it didn't help.

Any idea why prove6 would ignore my environment variable?

Thanks.