Comments (13)
Well, I guess it needs both seed and size. And I just found CheckerArguments.replay
which lets me specify the seed/size for the property, but since the test failure didn't print it, I don't know what it was.
from swiftcheck.
Verbose output seems the best place to put that when people ask for it.
from swiftcheck.
My concern is that if it doesn't show me the seed when it fails, if it's a rare failure it's already too late to go back and request verbose output. For example, in the failure I had that prompted this ticket, I still haven't reproduce it. I'm about to bump up maxAllowableSuccessfulTests
to some high number in the hopes of seeing it again.
from swiftcheck.
If you convert it to an existential, SwiftCheck will bombard it for you trying to search for a passing value.
from swiftcheck.
I admit to not really being sure how to do that. I found the exists
function, but presumably just replacing forAll
with exists
does the wrong thing, right? I assume that exists
looks for any passing test, rather than requiring all generated values to pass. So I'd need to negate my test too in addition to adding exists
, right? But I just tried that, and after printing a bunch of lines saying 0 tests passed, it finally reported test failure. Presumably that means it was unable to find an example where my property doesn't hold, right?
from swiftcheck.
Ah, I remember why I didn't let you see this (even though SwiftCheck does keep track of it). The CheckerState
struct was a mess for a while that was undocumented and in a crazy state of flux as I was adding and removing things to and from the framework. It's stabilized now, so it should be possible to implement something like QuickCheck's verboseCheckResult
that will hand you back the last Result
that happened rather than just pass it along SwiftCheck's internals.
I found the exists function, but presumably just replacing forAll with exists does the wrong thing, right?
Yeah. #43 Shows the identities for that stuff (if you'll excuse the notation).
Presumably that means it was unable to find an example where my property doesn't hold, right?
If that were the case, your log should contain "Could not satisfy existential"
from swiftcheck.
If that were the case, your log should contain
"Could not satisfy existential"
Yeah it did:
Could not satisfy existential (after 500 tests):
<# some random arbitrary value description, I'm guessing the last one it tried #>
This was after printing 500 copies of
*** Passed 0 tests
.
from swiftcheck.
Interesting. Seems I wasn't treating them like discards after all.
from swiftcheck.
OK, so this still leaves you with the problem of how to reproduce a test case. I feel it's particularly inelegant to ask you to manually invoke some future verboseCheckResult
. Perhaps a hatchway into the testing state could be used. Maybe some way of extending Property.withCallback
? Do you have an elegant way you envision going about this?
from swiftcheck.
I feel like the most important thing I need is just the info necessary to deterministically re-run the tests (which presumably is just the initial seed and size; I haven't checked but I hope the RNG you're using is deterministic and local to the library, so if there's other threads running they can't interfere).
Once I have the seed and size, I can look into trying to replay it with a checker argument. But without knowing the seed/size, there's no way to replay. And if the failure is rare, then I can't just re-run it and hope to see the problem again.
I'm thinking this just needs to print the seed/size on a failure. No need to print anything different when things pass. After all, there's no benefit in replaying a successful check.
from swiftcheck.
which presumably is just the initial seed and size; I haven't checked but I hope the RNG you're using is deterministic and local to the library, so if there's other threads running they can't interfere.
Lucky for you, it is neither of those things. At the time the little Random module in here was written, I was feeling neither confident in my ability to, nor ready to present, an implementation of a Mersenne twister or somesuch other feat of entropy-generating prowess. Nor was I willing to link with OpenSSL to accomplish the same. All this to say we use rand()
. We should be using rand_r()
. If you'd like to take a crack at it, see Random.swift
.
After that is done, then the seed and RNG can be handed back to you in a useful manner.
from swiftcheck.
The thing about printing them to the console is that you need both seed and StdGen
to replay, and the latter is a runtime entity only. It makes sense to only print the seed, but I'm not sure what you'll gain by having the console spit it out.
from swiftcheck.
@kballard With the merge of #120, you can now see seeds used for failing tests with the .verbose
property, and you can initialize our new deterministic pseudo-RNG StdGen
with those seed values to replay tests.
Thanks for the bug report!
from swiftcheck.
Related Issues (20)
- How to define arbitrary extensions for recursive datatypes? HOT 5
- Preferred idiom for testing exception-throwing code HOT 1
- Generator is not used during shrinking. HOT 2
- no Gen.let or heterogeneous sequence/tuple? HOT 7
- `forAll` doesn't support explicit generator being passed HOT 9
- Gen.frequency seems to fixate on a single generator for a single run HOT 1
- passing any checkerargs results in only a single test case running HOT 8
- allow specifying seed so that `generate` can be repeatable. HOT 14
- How exactly to rerun failed examples? HOT 5
- Doesn't compile with Xcode 10.2 HOT 1
- String generator breaks if test not always true HOT 2
- StdGen.genRange seems to have wrong value HOT 2
- Xcode canvas for SwiftUI always fail HOT 1
- Code from README doesn't compile
- Asynchronous Testing
- Integer partition generator
- Xcode 12 or swift 5.3 incompatibility
- SwiftCheck
- Unable to run Playground HOT 1
- (U)Int `arbitrary` implementation provides limited values with `.proliferate(withSize:)`
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from swiftcheck.