7.4.1 compat about doctest

[sol]

@LeventErkok Preliminary support for ghc-7.4 is on my ghc-7.4 branch.

You need haddock-2.10 for that. (not yet on Hackage, ghc/haddock@8b71c99 should work)

There is still one major issue left. It does not test examples in documentation for non-exported identifiers anymore; not sure if it is possible to support this with haddock-2.10.

Using the GHC API directly and getting rid of the Haddock dependency altogether may solve this (and a couple of other issues), I gave it a try, but it's quite some work.

[LeventErkok]

Not being able run tests on non-exported items would be a show-stopper for doctest.. Surely the haddock folks can be convinced to support that?

Looking forward to the new release..

[LeventErkok]

looks like haddock 2-10 is out.. any progress on doctest with 7.4.1?

[sol]

@LeventErkok not yet. I'm still sorting out if we can support this with haddock-2.10.

I know that the current situation is far from optimal; the first thing I want to do when upgrading to a new ghc is running my tests. Sorry for the inconvenience!

In the meantime you can still run doctests for exported identifiers with what's on my ghc-7.4 branch.

[sol]

@LeventErkok I have made some progress on that. So if things go well, there will be something to test in the next days.

[LeventErkok]

Excellent! I'm looking forward to the new release.

[sol]

@LeventErkok can you test what's on my extract-docstrings branch with your project? This probably still has some rough edges, and it definitively needs more testing. So some feedback would be awesome.

This now uses a standalone parser and the GHC API directly to extract documentation. Fixing this in Haddock would have delayed a new release for a few weeks.

The new approach will find more examples than Haddock did, in particular examples in named chunks that are not referred in the export list. This allows you to have examples that are not part of the generated documentation, but are still tested by doctest.

Other advantages are:

• it will be easier to support several versions of GHC
• it will be easier to ensure consistent behavior of doctest across versions
• examples are now executed in the exact same order as they appear in the source code
• source code locations for failing tests will be very easy to implement now (#8)

Downsides:

• I use syb to extract documentation. An obvious advantage of this approach is that it makes it very hard to miss any documentation. However, GHC stores error thunks in the AST, which requires special-casing of that subtrees. I think I took care of all the problematic cases, but it could still happen that you stumble into a GHC "panic". They are pretty easy to fix, as we discover them.
• The parser is somewhat more permissive than Haddock's parser. This again has the advantage, that we will never miss any examples, but we may still need to set this straight as we discover issues.

[LeventErkok]

For the following file:

{-# LANGUAGE ParallelListComp #-}

foo :: [Int]
foo = [x+y | x <- [1,2,3] | y <- [4,5,6]]

I'm getting Prelude.undefined.

I also suspect there're some type-checking issues at "haddock" time, but I don't have an isolated failure case for that yet.

[sol]

@LeventErkok The issue with parallel list comprehensions is fixed in f0d88da7c5.

What do you mean with type-checking issues? Currently I do no type-checking at "haddock" time, not even renaming. Both are easy to add. Would that help?

(My assumption was, that type-checking will be done when the examples are tested, so it is safe to skip when extracting the docstrings.)

[sol]

@LeventErkok by now I'm convinced that we really want to do type checking at "haddock" time. Otherwise the error messages that the user gets are not very nice.

[LeventErkok]

Right. I think the issue is it's spitting out type-errors when it shouldn't. I suspect it might have to do with the options that are passed down to ghc; much like the failure above for ParallelListComp's.

Do let me know when there's a new version to test.

[sol]

@LeventErkok Can you give my extract-docstrings branch an other try?

[LeventErkok]

Great! It works fine on my project now.

I can't recall the old behavior, but didn't we get some feedback while the tests were running before? (Like "passed 5, failed 3.. etc). Right now it looks like it goes and comes back with a success message when all the tests are done; without any feedback per test. It'd be really nice to have that HUnit like behavior.

[sol]

@LeventErkok can't reproduce this. I still get feedback while the tests are running.

Here is what I tried right now.

module Foo where

import Control.Concurrent

-- |
-- >>> threadDelay 1000000
foo :: Int
foo = 23

-- |
-- >>> threadDelay 1000000
bar :: Int
bar = 23

-- |
-- >>> threadDelay 1000000
baz :: Int
baz = 23

[LeventErkok]

my bad; I had some weird Makefile set-up that made the output invisible. It does indeed work fine.

[sol]

@LeventErkok Thx for helping with this. I know of at least one more minor issue that needs to be fixed (which I will do now). After that it needs to be battle tested. If you stumble upon anything, please report it.