haskell / test-framework Goto Github PK

Suppose we we something like

main = defaultMain [testA, testB]

where testA is constructed with buildTestBracketed. If we run the test suite using

test-suite -t testB

then the IO actions of testA will still be executed.

The progress bar is messing up the logs in Travis. Maybe could we disable the progress bar when running in a pseudo terminal, either by environment detection or by explicit option.

I'd be happy to help :)

A mailing list cannot be a maintainer (in particular, not an open list like [email protected]):

see https://github.com/haskell/test-framework/blob/master/core/Test/Framework/Tests.hs

Some time ago I tried resurrecting test:test-framework-tests but ran out of time; it compiles but I ran into an XML related failure which needs more investigation; this ticket is to remind me about it.

Current upper bound is < 2.5, current QuickCheck is 2.8.2.

test-framework depends on QuickCheck < 2.7, so there are no issues with the build.
However, QuickCheck>=2.7 uses a new random generator tf-random.
https://github.com/nick8325/quickcheck/blob/master/changelog

Bumping the package requirements thus gives the following error;

Test/Framework/Providers/QuickCheck2.hs:73:41:
    Couldn't match expected type `Test.QuickCheck.Random.QCGen'
                with actual type `System.Random.StdGen'
    In the expression: gen
    In the first argument of `Just', namely `(gen, 0)'
    In the `replay' field of a record

newSeededStdGen (that triggers the error above) from test-framework depends on System.Random.StdGen, so test-framework-quickcheck2 needs to implement its own function that may or may not depend on QCGen depending on the QC version.

regex-posix is just broken (on Windows), that's what I heard from @RyanGlScott

To my best knowledge the "test-framework" cannot be build with the latest ghc compiler and/or Stack environment.

Background:
My plan was to receive sample data output from QuickCheck2. QuickCheck is capable to do so - but not QuickCheck2 - because the needed parameters cannot be passed. In order to achieve that I tried to modify QuickCheck2. Since, I work with The Haskell Stack and the newest environment - I got stuck.

Please, help

Code details:
https://github.com/JoergBrueggmann/Test6

What's the best way to cancel the test runner on the first failure (like bash's -e/-o errexit)? Is it even possible?

I looked in RunnerOptions, but I didn't find it here.

In my case, the Tests are actually HUnit Tests which are IO actions (-> shelltestrunner).

I can't even just use die or fail in the IO action because the test runner continues on error:

I'd be glad if an experienced Haskeller could give me a hint!

I could install test-framework 0.8.2.0 with GHC 8.8.1 by running

$ cabal install test-framework --allow-newer=regex-posix

That is, I could install the library by using regex-posix 0.96.0.0. Could someone make the following revision in Hackage please

regex-posix (>=0.72 && <0.96) -> regex-posix (>=0.72 && <0.97)

Blocking agda/agda#3725.

Changelog: https://github.com/nick8325/quickcheck/blob/2.8/changelog

I recently tried building test-framework with GHC 8 and got this error when building Test/Framework/Improving.hs.

Using the flag -a you can change the number of test cases that should run for each test. The default is 100 tests.

$ dist/build/qc/qc 
Utils:
  refragment id: [OK, passed 100 tests]
...

Now with increased number of test cases;

 dist/build/qc/qc -a 1000
Utils:
  refragment id: [OK, passed 1000 tests]
...

However, if I use more than 1000 test cases, test-framework somehow decides to not run any tests at all.

$ dist/build/qc/qc -a 1001
Utils:
  refragment id: [Failed]
Arguments exhausted after 0 tests
...

As the comment notes, the arguments are exhausted so it's unable to execute any of the tests.

Currently test-framework-0.8.0.3 does not build with GHC HEAD due to a missing Applicative instance (which was added as per 0dac7f3)

Current master builds w/ GHC HEAD though; moreover, all previously uploaded test-framework packages need their cabal file edited to constrain base <4.8 as they're now known to be incompatible w/ base-4.8

Just let you know that the latest version of ansi-terminal is 0.10.

It does not seem that it introduces any breaking change, so it should be fine to bump up.

https://hackage.haskell.org/package/test-framework-quickcheck2 in version 0.3.0.5 reports random (>=1 && <1.2), but random-1.2 is out, and QuickCheck also supprots that already.

The latest version of test-framework available on Hackage is 0.8.0.3, but the one in the repo is 0.8.0.2.

/cc @bos

http://packdeps.haskellers.com/feed?needle=test-framework-hunit

Now that QuickCheck-2.12 has been released, it would be handy to have a version of test-framework-quickcheck2 which supported it. Currently, trying to bump the upper version bounds on QuickCheck will lead to a build failure:

[1 of 1] Compiling Test.Framework.Providers.QuickCheck2 ( Test/Framework/Providers/QuickCheck2.hs, /home/rgscott/Documents/Hacking/Haskell/test-framework/dist-newstyle/build/x86_64-linux/ghc-8.4.3/test-framework-quickcheck2-0.3.0.4/build/Test/Framework/Providers/QuickCheck2.o )

Test/Framework/Providers/QuickCheck2.hs:131:23: error:
    Not in scope: data constructor ‘InsufficientCoverage’
    |
131 |     toPropertyStatus (InsufficientCoverage _ _ _)              = PropertyInsufficientCoverage
    |                       ^^^^^^^^^^^^^^^^^^^^

This can be worked around easily enough:

diff --git a/quickcheck2/Test/Framework/Providers/QuickCheck2.hs b/quickcheck2/Test/Framework/Provi
index b61b808..6109400 100644
--- a/quickcheck2/Test/Framework/Providers/QuickCheck2.hs
+++ b/quickcheck2/Test/Framework/Providers/QuickCheck2.hs
@@ -127,6 +127,6 @@ runProperty topts testable = do
     toPropertyStatus (GaveUp {})                               = PropertyArgumentsExhausted
     toPropertyStatus (Failure { reason = rsn, output = otpt }) = PropertyFalsifiable rsn otpt
     toPropertyStatus (NoExpectedFailure {})                    = PropertyNoExpectedFailure
-#if MIN_VERSION_QuickCheck(2,8,0)
+#if MIN_VERSION_QuickCheck(2,8,0) && !MIN_VERSION_QuickCheck(2,12,0)
     toPropertyStatus (InsufficientCoverage _ _ _)              = PropertyInsufficientCoverage
 #endif

But if we do this, then PropertyInsufficientCoverage becomes unused when test-framework-quickcheck2 is built against QuickCheck-2.12. Should we CPP out PropertyInsufficientCoverage as well with QuickCheck-2.12 (since it's already conditionally defined on MIN_VERSION_QuickCheck(2,8,0) anyway)?

For long-running tests, it is convenient to learn which test is running even before one knows any results of the test. The console runner does not print anything until the full result is known. The following code demonstrates this:

{-# LANGUAGE MultiParamTypeClasses #-}

import Control.Concurrent
import Control.Monad
import Data.Typeable
import System.IO.Unsafe
import Test.Framework.Runners.Console
import Test.Framework.Providers.API

instance TestResultlike () Bool where testSucceeded = id
instance Testlike       () Bool (IO a) where
  runTest _ act = do
    v <- unsafeInterleaveIO act
    return (Improving () (v `seq` Finished True), return ())
  testTypeName _ = "IO action"

main = defaultMain [Test "infinity" (forever (threadDelay 1000000) >> return ())]

When I run this program, I would like to see at least:

infinity:

but instead see no output.

This package is on life support for at least five years and requires regular bumps from Hackage trustees to be compatible with the latest ecosystem. It's only a matter of time when revisions would no longer be enough. I'd like to mark it deprecated in favor of tasty (since migration from test-framework to tasty is mostly a matter of changing imports) to give a clear signal to clients that it's time to move on.

Support for QuickCheck-2.11.3 would be extremely nice to have with regard to the upcoming GHC 8.4.x release.

Could you please relax restriction for quickcheck 2.9.
Test framework fails to build with the latest stackage build(which has quickcheck 2.9).
Subsequently it impacts bson driver: commercialhaskell/stackage#1691

GHC 8 is coming with time-1.6