ps-pat / jcheck.jl Goto Github PK

Hi there! I'm the lead maintainer of Hypothesis, the Python property-based testing library, and just saw your talk on JCheck.jl. It's so exciting to see someone working on PBT for Julia!

I'm opening this issue in the hope that I can help you skip past the 2--6 years that it took us to work out a better internal model for PBT, since it looks like you've hit the same port-quickcheck approach that I'd describe as a tempting trap 😬. It's reasonably easy to get something working at first, but seriously hurts both shrinking and replay (and therefore usability) in the longer term. So my proposal is, in short: port the single-file https://github.com/DRMacIver/minithesis to Julia and rebase JCheck on top of that!

• Shrinking is implemented once in the library internals; then every generator is composed out of primitives such that the composed generator also shrinks automatically.
  • This is much easier for users and avoids all kinds of bugs - see https://hypothesis.works/articles/integrated-shrinking/
• Saving and replaying the "choice sequence" is easy (it's a sequence of integers), and suffices to replay inputs from any strategy
• You can generate non-serializable things like "which function to call" and that just works too, even if the options are determined based on earlier parts of the test
• Coverage-based fuzzing integration is also remarkably easy; just get the fuzzer to generate the choice sequence!
• Directed search ("targeted PBT") becomes remarkably easy, with any number of user-supplied objective functions.
• And minithesis is a single file with <900 lines of code, including comments and docs and whitespace, so it's easy to port.

I hope this is helpful! If you'd like to talk in more detail I'd be very happy to have a call sometime too 🙂

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!