nick8325 / quickcheck Goto Github PK
View Code? Open in Web Editor NEWAutomatic testing of Haskell programs.
License: Other
Automatic testing of Haskell programs.
License: Other
This is QuickCheck 2, a library for random testing of program properties. Add `QuickCheck` to your package dependencies to use it in tests or REPL. The quickcheck-instances [1] companion package provides instances for types in Haskell Platform packages at the cost of additional dependencies. The make-hugs script makes a Hugs-compatible version of QuickCheck. It may also be useful for other non-GHC implementations. [1]: http://hackage.haskell.org/package/quickcheck-instances
GHC 7.10 will ship with base-4.8.0.0
which will have the Natural
data type. It'd be great if QuickCheck came with a Arbitrary
instance.
Also see kolmodin/binary#63.
Commit e0607bf removed the Integral
and Real
instances for Positive
, NonZero
and NonNegative
.
Two questions:
Real
instances removed? They only provide toRational :: a -> Rational
, which cannot break any invariants.Integral
instances for Positive
and NonZero
were removed makes sense since with mod n n
we could create create a forbidden 0
. But for NonNegative
, 0 is allowed. Could its Integral
instance be reintroduced?Running quickCheckWith stdArgs { maxSuccess = a_big_number } (\() -> True)
consumes more and more memory. The culprit seems to be the terminal module - strings which are output with putTemp
are appended to the captured output even after they've been erased from the screen.
Hi all -
I'm having trouble with QuickCheck reporting correct examples as failing. I've put together a minimal example demonstrating this behavior in a Gist. I've trimmed it down and tested it on multiple machines, and I believe by this point that it's a library bug.
The following is copied for your convenience from the README in the Gist:
This repo documents a strange behavior I encountered when trying to test some code using the Data.Graph.Inductive library with QuickCheck.
QuickCheck will report failing examples for the test mergeWithFixTrue
(see Test.hs
). However, when these failing examples are copied into unit tests and run using HUnit, they invariably succeed.
This begs the question - why QuickCheck? Might HUnit not as easily be at fault? There are two reasons why I believe QuickCheck might be at fault here:
mergeWith
(see Merge.hs
) behaves differently when called with fixBody
from QuickCheck vs. HUnit or cabal repl
. When it is called via QuickCheck, mergeWith fixBody
returns a graph with an extra edge in it, and it is this unwanted edge that causes the test to fail.To reproduce the behavior described above, clone this repository and initialize a cabal sandbox with cabal sandbox init
. Then run the following:
You may have to run cabal test
a few times to get a failing example from QuickCheck. I like to run cabal test
until I get a nice, bite-sized failing example (no more than 3 or 4 edges in the example graph).
Once you've done this, insert the failing example into the HUnit test in Test.hs
on the line starting with: "testCase" (the failing example should go in the let-binding: let g = <failing example here>
).
Run cabal test
again, and notice that the example which QuickCheck reports as a failure runs perfectly well in HUnit.
The following are particularly notably missing: Word
, Word16
, Word32
, Word64
, Int8
, Int16
, Int32
, Int64
, Natural
, Complex
, Data.Sequence.Seq
, Data.IntMap.IntMap
, Data.Set.Set
, Data.IntSet.IntSet
, Data.Tree.Tree
, Data.Array.Array
, Data.Array.Unboxed.UArray
, Text.PrettyPrint.Doc
, Text.PrettyPrint.Annotated.Doc
.
Just a question of understanding. We currently have:
instance (Arbitrary a, CoArbitrary b) => CoArbitrary (a -> b) where
coarbitrary f gen =
do xs <- arbitrary
coarbitrary (map f xs) gen
Why do we do map f xs
here? Why not simply:
instance (Arbitrary a, CoArbitrary b) => CoArbitrary (a -> b) where
coarbitrary f gen =
do x <- arbitrary
coarbitrary (f x) gen
Thanks!
I tried to make a matrix with GHC 6.12.3, and it starts to be quite troublesome to compile. Maybe one could get further by disabling tests, but then tested-with
would lie.
Particularly, seems that I broke binary-0.8
for GHC <7.0 (because it isn't tested)
EDIT this would highly simplify cabal file too.
This is just a little feature request so I don't forget it. If I end up implementing it I'll report back or do a PR if this would be useful.
The idea is if we're testing functions that make use of some bit twiddling, operate on bytes of a word, etc. we might like an Arbitrary
instance for instances of Data.Bits.Bits
that implements a shrink
by perhaps zeroing individual bytes of a word, and then maybe zeroing individual bits (or something; I haven't thought much about this yet).
If I run verboseCheck prop
with prop
defined below, it slows my computer down and doesn't terminate. If I cancel the running command with Ctrl+c Ctrl+c
, the command stops and prompt returns, but it seems process is still running and computer is slowed. Until I exit ghci prompt. This problem does not occur for quickCheck prop
.
prop :: Fun String Integer -> Bool
prop (Fun _ f) = f "monkey" == f "banana" || f "banana" == f "elephant"
When running with Git HEAD:
QuickCheck-2.9.1: test (suite: test-quickcheck)
=== prop_ToSortedList from examples/Heap.hs:123 ===
*** Failed! Falsifiable (after 7 tests):
Node (-4) (Node 4 (Node 7 Empty Empty) (Node 8 Empty Empty)) (Node (-2) (Node 5 Empty Empty) (Node (-1) Empty Empty))
QuickCheck-2.9.1: test (suite: test-quickcheck-gcoarbitrary)
*** Failed! Falsifiable (after 4 tests and 4 shrinks):
{_->2}
In the instance for Arbitrary (Positive a)
the definition for arbitrary is this:
arbitrary =
((Positive . abs) `fmap` (arbitrary `suchThat` (/= 0))) `suchThat` gt0
where gt0 (Positive x) = x > 0
Is the check ``suchThat gt0
just in case the `Num` instance for `a` has a misbehaving definition for `abs`?
I'm pretty sure:
instance Testable Property where
property = property . unProperty
Should have a definition of exhaustive
. Currently:
Test.QuickCheck> quickCheck $ 1 == 1
+++ OK, passed 1 tests.
Test.QuickCheck> quickCheck $ property $ 1 == 1
+++ OK, passed 100 tests.
I think adding the line:
exhaustive = exhaustive . unProperty
Would do.
This package depends on array
transitively, via containers
, so adding it as a direct dependency can't do any harm.
I'm wondering if it would be possible to add an Alternative
instance for Gen
The starting point for the change would be to move from:
newtype Gen a = MkGen { unGen :: QCGen -> Int -> a }
to either:
newtype Gen a = MkGen { unGen :: QCGen -> Int -> Maybe a }
or:
newtype Gen a = MkGen { unGen :: QCGen -> Int -> [a] }
I think the version with the list for the results would probably work better for <|>
, with something like oneof
being used to get back to a single value - like the current behaviour - when the list is non-empty.
While making a breaking change like that, I'd also be tempted to add another Int
parameter in there, to be used to make <|>
behave more like frequency
, with corresponding helpers like sized
/ resize
to help out.
I'm keen to give this a go myself and see how far I can push it, but I thought I'd put something here and so whether folks are strongly for or strongly against the idea, or if there are any landmines here I should be aware of, before I put too much time into it.
We have a few tests that are hitting the 4MB log limit on Travis because all the intermediate progress is logged out by QuickCheck. We have a temporary fix which processes \b
characters before outputting a line, but it would be useful if we could just pass an option to QuickCheck to provide slightly quieter output.
Happy to do the work if you have some idea how you'd like it done.
monomorphic
always uses VarE
internally, which fails if it's given the Name
of a constructor. For example, to make monomorphic lists:
{-# LANGUAGE TemplateHaskell #-}
import Test.QuickCheck.All
nil = $(Test.QuickCheck.All.monomorphic '[])
cons = $(Test.QuickCheck.All.monomorphic '(:))
Loading this in GHCi gives:
Illegal variable name: ‘[]’
When splicing a TH expression:
GHC.Types.[] :: [GHC.Integer.Type.Integer]
In the splice: $(Test.QuickCheck.All.monomorphic '[])
Failed, modules loaded: none.
Using ConE
would allow constructors (after branching on whether the Name
could be a variable or not).
If we have a property that uses (===)
and fails with an exception, e.g.
prop = (=== (undefined :: Int))
then quickCheckWithResult
will leak that exception as part of the output
when run with chatty = False
.
Here are two GHCi sessions that demonstrate the issue. Note that for the first case (when chatty = True
) the output
field is a total string. In contrast, for the second case (when chatty = False
) the output
field contains an error thunk.
ghci> output <$> quickCheckWithResult stdArgs {chatty = True} prop
"*** Failed! Exception: 'Prelude.undefined' (after 1 test): \n0\nException thrown by generator: 'Prelude.undefined'\n"
ghci> output <$> quickCheckWithResult stdArgs {chatty = False} prop
"*** Failed! Exception: 'Prelude.undefined' (after 1 test): \n0\n0 /= *** Exception: Prelude.undefined
Negative Gen sizes cause all sorts of trouble. Probably resize should just throw an error on negative sizes to highlight incorrect generators early during development/testing.
I was trying to figure out why generate (vector 100 :: Gen [Large Int])
was only giving me small domain values, and noticed that arbitrarySizedBoundedIntegral
is incorrect:
> fmap maximum $ replicateM 1000000 $ generate (arbitrarySizedBoundedIntegral :: Gen Int)
4096
That code's a bit opaque to me, sorry I'm not of much help.
To make Arbitrary
instances for my types I have to include quickcheck in my main library depends, rather than just testing, or deal with orphaned instance warnings by defining it in the test code only (and then I can't put properties in doctest, as well, I think).
We have quite a lot of quickcheck properties with replay arguments. The replay arguments are specified with the old StdGen. However, the most recent quickcheck version forces you to use tf-random.
Would it be possible to have a cabal flag specifying that you want to use StdGen? For older versions of ghc and base, you do this anyway.
Congratulations on the release
This commit adds a Boolean but it is unclear what is being tested when I see not b
, why not use an ad-hoc type instead? I need help coming up with a name though :)
data HaveIBeenShrunk = UnShrunken | Shrunken
data Honey_I = Didn'tShrinkTheKids | ShrunkTheKids
I think this is probably sensible:
instance Arbitrary QCGen where
arbitrary = MkGen (\g _ -> g)
Zemyla (jokingly?) suggested also making it an instance of CoArbitrary
so that Gen
can also be made an instance of Arbitrary
. Why not?
instance CoArbitrary QCGen where
coarbitrary s = variant (fst (next s))
With this in place, a property can ask for a generator and do with it what it likes.
It would be nice to be able to use existential quantification in QuickCheck.
Since ∃x(φ(x))
is equivalent to ¬∀x(¬φ(x))
, the easiest thing to do would be add an invert :: Testable prop => prop -> Property
"Property Combinator".
Citing from http://hydra.cryp.to/build/1054975/nixlog/1/raw:
Test/QuickCheck/All.hs:76:11: error:
The constructor ‘ClassOpI’ should have 3 arguments, but has been given 4
In the pattern: ClassOpI _ ty _ _
In an equation for ‘infoType’: infoType (ClassOpI _ ty _ _) = ty
How would we feel about adding the text
package as a dependency of QuickCheck? I know it's good to keep QuickCheck's dependencies light, but I think depending on Haskell's recommended string type is a reasonable measure.
Right now there's probably an orphan Arbitrary Text
instance in pretty much every sizable application that uses QuickCheck (and library authors have it even worse because they don't get that option).
I want to test a property with a limited range of values, say the characters from 'a' to 'z'. I create a custom Gen
for this purpose and run quickCheck
on it. Unfortunately, if a test fails the failing arguments are not reported:
QC> QC.quickCheck $ fmap (<'f') $ QC.choose ('a','z')
*** Failed! Falsifiable (after 1 test):
Is it possible to add this report? If not, how about an alternative form of quickCheck
that better supports Gen
parameters? E.g. an alternative form might be called like this:
QC> QC.quickCheckCustom (QC.choose ('a','z')) $ \c -> c<'f'
*** Failed! Falsifiable (after 1 test):
't'
Hi,
first many thanks for opening the issue tracker. I think this will benefit QuickCheck a lot.
Now to my issue. Note that it is written with more information than necessary for the QuickCheck maintainer (most of it is probably immediately clear to him) because I wanted to explain the situation to those who haven't spent the last 10 hours on the code.
Currently (as of commit 205f9f9), genericShrink (0::Int) == [0]
. I don't think this is intended, since the documentation of shrink
mentions that shrinking a value to itself will result in an infinite loop.
The problem is this:
instance Typeable a => Subterms (K1 i a) where
gsubterms (K1 x) =
case cast x of
Nothing -> []
Just y -> [y] -- Bug: this will shrink 0 to [0]
I first believed that it can be fixed by Just y -> shrink y
, but that is not so because then genericShrink [0::Int] == []
, which is also wrong, it should be [[]]
.
I noticed that 2 years ago, @bitonic created a gShrink
/gArbitrary
implementation (see here that doesn't use the Typeable
constraint, but instead relies on OverlappingInstances
. In my opinion this is better because it removes a constraint from genericShrink
, requiring only Generic
, and the OverlappingInstances
is only needed in Arbitrary.hs
to cleanly select the current type and has no effect outside of this file. And of course, it does not come with a run-time penalty while Typeable
does.
I have created a branch that compares the 5 different ways that I've seen so far - have a look here:
genericShrinkOrig
- the original genericShrink
implementation. Wrong because it shrinks 0 to 0.genericShrinkOrigFixed
- my try to fix it by using shrink
in the base case. Wrong because it shrinks [0] to [].genericShrinkBitonic
- same output as subtermsOrig
(so recursively shrinking subterms is missing) and having the same problem as 1.genericShrinkBitonicFixed
- my try to fix it by using shrink
in the base case. Same output as subtermsOrigFixed
(so recursively shrinking subterms is missing) and having the same problem as 3.genericShrink
, my improved version, is the apparent solution (please check thorougly if what I say is correct). Explanation:The core problem is to correctly capture the semantics of shrink
in the generic instance, that being the points:
Let's go through them, on the example of data Tree a = Nil | Branch a (Tree a) (Tree a)
:
subterms
. What are the immediate subterms of Tree a
that are of the same type Tree a
(since we can only shrink to values of the same type)? The left and the right branch. Not the value a
, because it is not a tree. This is what the Typeable
is for: GHC.Generics will give us all 3 subterms, [a, Tree a (left), Tree a (right)]
separated by *:*
, and we have to cast
in order to check if each of them really is a Tree. In @bitonic's implementation, this check is done at run-time instead of at compile time by using class GShrinkBitonic f b
instead of class SubtermsOrig f
- note how there's one extra type parameter. This allows him to distinguish the two cases instance Arbitrary a => GShrinkBitonic (K1 i a) a
(subterm is of same type as the term) and instance GShrinkBitonic (K1 i a) b
(subterm is of different type) using OvelappingInstances
. Back to the problem: In the Subterms (K1 i a)
case, the case x of Just y -> [y]
returns [0]
when run on 0::Int
because the generic instance of Int
is basically a K1
around itself: from (1::Int) == M1 {unM1 = M1 {unM1 = M1 {unM1 = K1 {unK1 = 1}}}}
.recursivelyShrink
. The idea here is to keep the same outer structure, but shrink the inner structures. So basically Branch x l r
becomes Branch x (shrink l) (shrink r)
, for any combination of shrinking the left and right part. This is done correctly by the current implementation. For sum types (RecursivelyShrink (f :+: g)
) just do it inside, for product types (RecursivelyShrink (f :*: g)
) recursively do it for left or right side inside the existing outer structure, for the base case RecursivelyShrink (K1 i a)
do grecursivelyShrink (K1 x) = map K1 (shrink x)
, calling shrink
on the values inside.Nothing
and Nil
), and that genericShrink
can only be used for doing all the non-specific cases, as is explained with shrink x = shrinkToNil x ++ genericShrink x
in the QuickCheck docs. For types where the constructors are equally simple, a shrink = genericShrink
is enough.So the problem is in 1, namely that subterms
creates things that are not subterms (0
isn't a subterm of 0
).
Looking at 1, my *Fixed
versions are wrong, since I call shrink
inside of subterms
, and "Shrink a term to any of its immediate subterms" really means "just take the inner terms of your structure, as they are".
So what's the right subterms
?
A value can only have true _sub_terms if it has a product type (:*:)
somewhere.
[]
.(:*:)
.That means we need two type classes for this: An outer GSubterms
and an inner GSubtermsIncl
(usive).
gSubterms (K1 _) = []
, and on products calls the inner one: gSubterms (l :*: r) = gSubtermsIncl l ++ gSubtermsIncl r
.gSubtermsIncl (K1 x) = [x]
if the type matches (this can be done with GSubtermsIncl (K1 i a) a
or Typeable
).As a result, subterms
returns the first container in a value, unless the value consists only and entirely of that container.
This gives the right result for all subterms
invocations I've tried so far. Please check if you agree with my reasoning. Again, the working example is genericShrink
in here.
I will soon submit a pull request with the cleaned-up version of this.
The new release candidate for GHC 8.x cannot compile QuickCheck because of the following restrictions on template-haskell
:
template-haskell >=2.4 && <2.11
This issue prevents of from building (or rather: testing) pretty much any significant package with that compiler. Would it be possible to get an update for QuickCheck that support the 8.x series of GHC?
Generate edge cases (minBound
, maxBound
, minBound + 1
, ...) with some low frequency.
abs
always returns a positive number right?
>>> quickCheck (\(n :: Int) -> abs n >= 0)
+++ OK, passed 100 tests.
Alright let's ship it — well — the test fails if we run it on Int8
s often enough
>>> quickCheck (\(n :: Int8) -> abs n >= 0)
+++ OK, passed 100 tests.
>>> quickCheck (\(n :: Int8) -> abs n >= 0)
+++ OK, passed 100 tests.
>>> quickCheck (\(n :: Int8) -> abs n >= 0)
+++ OK, passed 100 tests.
>>> quickCheck (\(n :: Int8) -> abs n >= 0)
*** Failed! Falsifiable (after 23 tests):
-128
Turns out it doesn't actually hold for Int
s
>>> abs @Int minBound
-9223372036854775808
This could be detected for larger sample spaces like Int
if minBound
were generated. I would like to see this in the Arbitrary Int
instance but maybe a modifier is better?
Hello,
It would be nice to have a built-in generator for randomly ordering a list.
Something along these lines:
import System.Random.Shuffle (shuffle)
-- | Take a list and generate a shuffled version of it.
shuffled ::[a] -> Gen [a]
shuffled xs = do
rs <- randomOrdering (length xs - 1)
return $ shuffle xs rs
where
-- a sequence (r1,...r[n-1]) of numbers such that r[i] is an independent
-- sample from a uniform random distribution [0..n-i]
randomOrdering 0 = return []
randomOrdering n =
do y <- choose (0, n)
ys <- randomOrdering (n - 1)
return (y:ys)
Depends on System.Random.Shuffle
I saw this SO post https://stackoverflow.com/questions/8191131/find-the-value-that-failed-for-quickcheck
is it possible to have a function like this build into quickcheck itself?
My use case is that i have a parser and a pretty printer and the test flow is like this:
quickcheck generate random syntax tree -> print -> source code -> parse -> syntax tree (should be same as input)
If a test case fails i get the syntax tree which is hard to read and i like to try to send it to the printer.
We use cover
to ensure that our tests are meaningful (e.g. not vacuously true due to ==>
or that cases we care about are actually generated).
Currently cover
is only informative, so if a cover is false you get some warning printed out by quickCheck
, but the test still passes. If you use a test runner like test-framework
, tasty
or hspec
, they do not notice.
Could we have a version of cover
that strictly lets the test fail?
This would make it possible to notice when somebody changes implementation code that lets our test cases degenerate.
@jtdaugherty reports that somewhere between QuickCheck 2.8.2 and 2.9.1 a bug (or a silent and undocumented in the Changelog change in semantics) was introduced: UnkindPartition/tasty#147
Could you look into it?
quickCheck $ forAll arbitrary $ \b -> cover b 100 "a" True
+++ OK, passed 100 tests (only 68% a; not 100%).
forAll arbitrary $ \b -> conjoin [ cover b 100 "a" True ]
+++ OK, passed 100 tests.
I believe the conjoin
case should also print a warning.
If you write with cover
: "I want that x% (e.g. 100%) of my generated characters are a
", and no a
s are generated at all, cover
doesn't warn you when it should.
Example 1:
quickCheck $ forAll (return 'b') $ \c -> cover (c == 'a') 100 "a" True
+++ OK, passed 100 tests.
The problem is that quickcheck only considers stamp
s that actually occur.
This also happens if the percentage is very small so that with high probability no suitable sample is found; example 2:
> quickCheck $ forAll arbitrary $ \c -> cover (c == 'a') 100 "a" True
+++ OK, passed 100 tests (only 1% a; not 100%).
> quickCheck $ forAll arbitrary $ \c -> cover (c == 'a') 100 "a" True
+++ OK, passed 100 tests.
The location where cover
is handled is covers
in success :: State -> IO ()
in QuickCheck.Test
.
Similar to dreixel/generic-deriving#41, QuickCheck doesn't build with template-haskell-2.11.0.0 on GHC-7.8.4 (and GHC-7.10.3). Cabal produces install plans which attempt to do so.
I will provide a simple cabal file to reproduce the issue.
The following are used in containers
apply2 :: Fun (a, b) c -> a -> b -> c
apply2 f a b = apply f (a, b)
apply3 :: Fun (a, b, c) d -> a -> b -> c -> d
apply3 f a b c = apply f (a, b, c)
I see two directions we can go. We can supply those functions, along with:
pattern Fn :: (a -> b) -> Fun a b
pattern Fn f <- Fun _ f
pattern Fn2 :: (a -> b -> c) -> Fun (a, b) c
pattern Fn2 f <- Fun _ (curry -> f)
pattern Fn3 :: (a -> b -> c -> d) -> Fun (a, b, c) d
pattern Fn3 f <- Fun _ (curry3 -> f)
curry3 f a b c = f (a, b, c)
now it can be written
pInsertWithKeyStrict :: Fun (Int, Int) Int -> Int -> IntMap Int -> Bool
pInsertWithKeyStrict (Fn2 f) v m = isBottom $ M.insertWith f bottom v m
pInsertLookupWithKeyKeyStrict :: Fun (Int, Int, Int) Int -> Int -> IntMap Int -> Bool
pInsertLookupWithKeyKeyStrict (Fn3 f) v m = isBottom $ M.insertLookupWithKey f bottom v m
Alternatively we could use some type class trickery to provide a single function and pattern that supersede apply{,2,3}
and Fn{,2,3}
. So that the user never has to think about how many arguments their function has.
I have some doubts about the effects that will have on inference.
One last solution: somehow define an Arbitrary (Fun a (b -> c))
instance. I have not thought this through but if this makes sense it might be the most elegant solution.
I found this issue while preparing Agda for supporting GHC 7.10.1. The following example compiles with GHC 7.8.4:
{-# LANGUAGE TemplateHaskell #-}
import Test.QuickCheck
prop_id :: Eq a => a -> Bool
prop_id x = x == x
return []
main :: IO Bool
main = $quickCheckAll
Using GHC 7.10.1 RC 1, I got the following error:
$ ghc Test.hs
Test.hs:12:8:
Exception when trying to run compile-time code:
Higher-kinded type variables in type: forall (a_0 :: *) . GHC.Classes.Eq a_0 => a_0 -> GHC.Types.Bool
Code: quickCheckAll
In the splice: $quickCheckAll
I found myself wanting this today.
class Monad m => MonadGen m where
liftGen :: Gen a -> m a
instance MonadGen Gen where
liftGen = id
instance MonadGen m => MonadGen (StateT s m) where
liftGen = lift . liftGen
-- and so on
Recently I found out QC never finds counterexample to following property even after 1e8 test cases:
p_failing :: Word64 -> Bool
p_failing i = testBit i 63 == False
But it's false for exactly half of Word64
! Then plotted frequency for each bit of Word64 which is passed to property (1e6 test cases). Bits 63 and 62 are always zero in all test cases.
And looking at consecutive inputs reveals strange pattern:
1 0x0000000000000001
2 0x0000000000000000
3 0x0000000000000001
4 0x0000000000000000
5 0x0000000000000002
6 0x0000000000000004
7 0x0000000000000005
8 0x000000000000000b
9 0x000000000000000c
10 0x0000000000000009
....
90 0x004fe9b98bdd482e
91 0x00bf83bfb1a60c70
92 0x00a8e49cc9fee6ca
93 0x00385b6297fbde9d
94 0x007069be7a8569e9
95 0x06f7323a8211c169
96 0x05668182d04237f2
97 0x0ebac5a91be8d419
98 0x134527f8393c2980
99 0x1f2b0200977d9706
100 0x3a1ce8ef8c3eac26
101 0x0000000000000000
102 0x0000000000000000
103 0x0000000000000000
104 0x0000000000000000
105 0x0000000000000004
Every 100 of test cases generator starts to explore search space from small numbers again and never reaches big Word64
Ideally the exception handling logic would be based off of either the exceptions or the monad-control package, but even a simple try :: (Exception e, Monad m) => PropertyM m a -> PropertyM m (Either e a)
would be satisfactory.
I took a stab at each of these myself, but I couldn't get the types to work out. I'm creating this issue in hopes that there's something I missed that will be obvious to someone more familiar with QuickCheck's inner workings.
-- file Main.hs
import Test.QuickCheck
import Test.QuickCheck.State
import Test.QuickCheck.Property
main :: IO ()
main = quickCheckWithResult args (callback progress prop) >>= print
where
prop :: [Int] -> Bool
prop xs = (reverse . reverse) xs == xs
args = stdArgs {maxSuccess = 100000, chatty = False}
progress = PostTest NotCounterexample $ \st _ -> (print $ numSuccessTests st)
$ ghci Main.hs
*Main> :main
While the test is running, press ctrl-c
.
Expected result:
UserInterrupt
exception is propagated, so that the default exception handler kicks in)Actual result:
With high probability
Success
, with output
set to "*** Exception in callback: user interrupt\n+++ OK, passed 100000 tests.\n"
Quick Check 2.8.2 does not work with any template-haskell
. In particular, it does not work with version 2.11. Please take time and specify package version bounds.
QuickCheck depends on containers, could the Arbitrary instances for it be added?
Naive Map
generation might not be what you want, but it's still better then no instance. Set
and Seq
should be trivial.
I can write a PR, if the idea is ok.
I've got a quickcheck property that times out (5 second timeout) when the test case gets large enough. If I run that specific test case through the functions under test within GHCI, it completes instantly. Test failure is reproducible, happening after about 90 test cases every time.
You should be able to reproduce it with this file.
GHC version:
> ghc -v
Glasgow Haskell Compiler, Version 7.10.1, stage 2 booted by GHC version 7.8.4
If you want a cabal file, here's a chunk:
test-suite test
default-language:
Haskell2010
type:
exitcode-stdio-1.0
hs-source-dirs:
Test
main-is:
test.hs
default-extensions:
build-depends:
base >= 4.8
, QuickCheck
, bytestring
An example failure is
*** Failed! Exception: '<<timeout>>' (after 90 tests):
Large {getLarge = 69071890884779960}
When I pass it through GHCI, it completes instantly, whereas QuickCheck says the test times out, taking >= 5 seconds.
> fromSignedByteString . toSignedByteString $ 69071890884779960
69071890884779960
It makes no progress if I remove the timeout and just let it run
> time dist/build/test/test
^C*** Failed! Exception: 'user interrupt' (after 94 tests):
Large {getLarge = 102007353455633744}
182.37user
Memory usage doesn't climb, so it doesn't seem to be a large space leak.
Since I can't reproduce the problem when running my functions outside of quickcheck, I'm assuming that quickcheck is the problem.
import Test.QuickCheck
main = quickCheck $
forAll arbitrary $ \b ->
cover b 100 "b is True" $
b ==> True
I would expect this to tell me something like b is true onl 50% of the time
, but I get OK, passed 100 tests
.
It seems odd to me that the inner ==>
can sneak around the outer cover
.
If I had written
import Test.QuickCheck
main = quickCheck $
forAll arbitrary $ \b ->
b ==>
cover b 100 "b is True" $
True
then that output would be expected, but in my case above cover
comes first, so it surprised me.
Is this intended?
If so, could we make it clear in the documentation, maybe even this exact example?
Like an http://hackage.haskell.org/package/transformers-0.5.1.0/docs/Data-Functor-Classes.html
I can make an PR if this idea is OK
Applies to ver. 2.8.1
{-# LANGUAGE TemplateHaskell #-}
import Test.QuickCheck
import Test.QuickCheck.All
{-
prop_old_freeVars_Term conf x = True
-}
-- Template Haskell hack to make the following $quickCheckAll work under ghc-7.8.
return []
-- | All tests as collected by 'quickCheckAll'.
tests :: IO Bool
tests = do
putStrLn "Agda.TypeChecking.Free.Tests"
$quickCheckAll
gives warning
/home/abel/agda/src/full/Agda/TypeChecking/Free/Tests.hs:129:3: Warning:
Name prop_old_freeVars_Term found in source file but was not in scope
which turns into error with -Werror
Already asked this on Stackoverflow AND Reddit, but no one seems to know.
I've been trying to solve the problem for the past 3 days now, but no matter what I try, the types don't compose elegantly.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.