"not in scope" errors when running with GHC 9 + stack test; redux about doctest

[sol]

@chshersh hey👋 I'll try to look at it and see if I can help, but it may take a week or so until I get to it. If it gets lost somehow then please feel free to ping me again.

[antoine-fl]

I also get this error but it seems related to the OverloadedStrings extension in my case.

I made a simple repository to reproduce the issue: https://github.com/antoine-fl/doctest-bug

Using GHC 9:

$ stack test
doctest-bug> test (suite: doctest-bug-test)

src/Lib.hs:11: failure in expression `isBar "bar"'
expected: True
 but got: 
          ^
          <interactive>:25:1: error: Variable not in scope: isBar :: t0 -> t

Examples: 2  Tried: 2  Errors: 0  Failures: 1

doctest-bug> Test suite doctest-bug-test failed
Test suite failure for package doctest-bug-0.1.0.0
    doctest-bug-test:  exited with: ExitFailure 1
Logs printed to console

Using GHC 8:

$ stack test --stack-yaml stack-8.yaml
doctest-bug> test (suite: doctest-bug-test)

Examples: 2  Tried: 2  Errors: 0  Failures: 0

doctest-bug> Test suite doctest-bug-test passed

The relevant doctest is in that file: https://github.com/antoine-fl/doctest-bug/blob/main/src/Lib.hs

With GHC 9, using Data.Text.pack instead of OverloadedStrings in the doctest fixes the issue.

Edit: testing with GHC 9.2, the tests pass again.

[andreasabel]

I had some luck with getting rid of "not in scope" errors by replacing :set -XOverloadedStrings by

:seti -XOverloadedStrings

(on GHC 9.4.4).

[vaclavsvejcar]

@sol I'm having the same issue, do you have any ideas what could be the root cause and if there's any workaround? Thanks a lot!

[sol]

@echatav do you think it's possible to reproduce your issue to the point where it does not require a database?

[sol]

@vaclavsvejcar I think your issue is due to :set -XQuasiQuotes triggering https://gitlab.haskell.org/ghc/ghc/-/issues/20670.

[sol]

@vaclavsvejcar as a workaround, I think you can't set -XQuasiQuotes in Haddock examples, but you can set it in doctest/Main.hs instead.

[vaclavsvejcar]

@vaclavsvejcar as a workaround, I think you can't set -XQuasiQuotes in Haddock examples, but you can set it in doctest/Main.hs instead.

@sol Thanks a lot for hints. In my case, that was the issue. It still didn't work with GHC 9.0.1 due to https://gitlab.haskell.org/ghc/ghc/-/issues/19460, but works fine with 9.0.2.

[echatav]

@echatav do you think it's possible to reproduce your issue to the point where it does not require a database?

@sol You don't need the database to see the doctest failures, it'll just add some more failures without a database.

[echatav]

Here is the doctest failure in Squeal's CI for GHC 9.0

[sol]

@vaclavsvejcar you are welcome. Ideally, we would enable -XQuasiQuotes by default (as we already do for -XTemplateHaskell). However, this will render e.g. [i|i <- [0..3]] invalid. So not sure if this is the right thing to do.

[sol]

@echatav I don't have any spare cycles to investigate this myself right now. If you (or somebody else) can provide a somewhat more minimal example that reproduces the issue then I'll try to look at it.

[echatav]

@sol I've made a minimal example branch which demonstrates the test error.

[chshersh]

Relude heavily uses doctest and has this issue as well. Moreover, it uses neither TemplateHaskell nor QuasiQuotes but still has this issue. CI error example:

• https://github.com/kowainik/relude/runs/6039897885?check_suite_focus=true

The current workaround is to use GHC 8.10.7 on builds using stack.

[sol]

@chshersh in general, if you can provide exact steps on how to reproduce, this will save me a lot of time.

Here is what I tried:

$ git rev-parse HEAD
3011495606b8437a3ccd21d03d743f4d31e9c257
$ git diff

diff --git a/relude.cabal b/relude.cabal
index 55e085f..187e237 100644
--- a/relude.cabal
+++ b/relude.cabal
@@ -258,7 +258,7 @@ test-suite relude-doctest
   main-is:             Doctest.hs
 
   build-depends:       relude
-                     , doctest < 0.19
+                     , doctest >= 0.20
                      , Glob
 
   ghc-options:         -threaded
diff --git a/stack.yaml b/stack.yaml
index fd834e1..db2f745 100644
--- a/stack.yaml
+++ b/stack.yaml
@@ -1 +1 @@
-resolver: lts-17.5
+resolver: nightly-2022-04-24

$ stack test relude:test:relude-doctest
...
Examples: 447  Tried: 447  Errors: 0  Failures: 0

relude> Test suite relude-doctest passed

[simonmichael]

I am way out of touch with this/these issues, but just FYI: the failures originally reported in July with hledger tests and GHC 9.0 are still happening today, but they don't happen with GHC 9.2.

[NorfairKing]

The same is happening to me with autodocodec: https://github.com/NorfairKing/autodocodec
I find this really sad because it forces me to turn off doctests :(

[aniketd]

Adding some examples to this issue, hoping they will be helpful for investigating further:

Package random - branch doctests-attempt

https://github.com/haskell/random/tree/doctests-attempt

random on  doctests-attempt [!?] via λ lts-18.28
❯ stack test random:test:doctests
random> build (test)
Preprocessing library for random-1.3.0..
Building library for random-1.3.0..
Preprocessing test suite 'doctests' for random-1.3.0..
Building test suite 'doctests' for random-1.3.0..
random> test (suite: doctests)
                                                                                                                                                                                     
src/System/Random/Internal.hs:651: failure in expression `import GHC.Generics (Generic)'
expected:
 but got:
          ^
          <interactive>:1:32: error:
              Module ‘System.Random.Stateful’ does not export ‘globalStdGen’
                                                                                                                                                                                     
          <interactive>:1:32: error:
              Module ‘System.Random.Stateful’ does not export ‘newAtomicGenM’
                                                                                                                                                                                     
          <interactive>:1:46: error:
              Module ‘System.Random.Stateful’ does not export ‘randomRM’
                                                                                                                                                                                     
          <interactive>:1:47: error:
              Module ‘System.Random.Stateful’ does not export ‘applyAtomicGen’
                                                                                                                                                                                     
src/System/Random/Internal.hs:691: failure in expression `import GHC.Generics (Generic)'
expected:
 but got:
          ^
          <interactive>:1:32: error:
              Module ‘System.Random.Stateful’ does not export ‘globalStdGen’
                                                                                                                                                                                     
          <interactive>:1:32: error:
              Module ‘System.Random.Stateful’ does not export ‘newAtomicGenM’
                                                                                                                                                                                     
          <interactive>:1:46: error:
              Module ‘System.Random.Stateful’ does not export ‘randomRM’
                                                                                                                                                                                     
          <interactive>:1:47: error:
              Module ‘System.Random.Stateful’ does not export ‘applyAtomicGen’
                                                                                                                                                                                     
src/System/Random/GFinite.hs:88: failure in expression `import GHC.Generics (Generic)'
expected:
 but got:
          ^
          <interactive>:1:32: error:
              Module ‘System.Random.Stateful’ does not export ‘globalStdGen’
                                                                                                                                                                                     
          <interactive>:1:32: error:
              Module ‘System.Random.Stateful’ does not export ‘newAtomicGenM’
                                                                                                                                                                                     
          <interactive>:1:46: error:
              Module ‘System.Random.Stateful’ does not export ‘randomRM’
                                                                                                                                                                                     
          <interactive>:1:47: error:
              Module ‘System.Random.Stateful’ does not export ‘applyAtomicGen’
                                                                                                                                                                                     
Examples: 183  Tried: 171  Errors: 0  Failures: 3
                                                                                                                                                                                     
random> Test suite doctests failed
Completed 2 action(s).
Test suite failure for package random-1.3.0
    doctests:  exited with: ExitFailure 1
Logs printed to console
                                                                                                                                                                                     
                                                                                                                                                                                     
random on  doctests-attempt [!?] via λ lts-18.28 took 53s
❯

NOTES: Notice Examples: 183 Tried: 171 Errors: 0 Failures: 3. This is definitely weird.

Minimal example package using stack

doctest-stack.tar.gz

Code/oss/doctest-stack via λ lts-20.4
❯ lsd --tree
 .
├──  app
│   └──  Main.hs
├──  CHANGELOG.md
├──  doctest-example-stack.cabal
├──  LICENSE
├──  package.yaml
├──  README.md
├──  Setup.hs
├──  src
│   └──  Lib.hs
├──  stack.yaml
├──  stack.yaml.lock
└──  test
    ├──  doctests.hs
    └──  Spec.hs

Code/oss/doctest-stack via λ lts-20.4
❯ stack test doctest-example-stack:test:doctest
doctest-example-stack> test (suite: doctest)
                                                                                                                                                                                    
Examples: 5  Tried: 5  Errors: 0  Failures: 0
                                                                                                                                                                                    
doctest-example-stack> Test suite doctest passed

NOTES: This just works!

Minimal example package using cabal-install

doctest-cabal.tar.gz

[aniketd@nixos:~/Code/oss/doctest-example]$ tar --exclude-vcs --exclude-vcs-ignores --exclude-caches-all --exclude=.stack-work --exclude=dist-newstyle -zvcf doctest-cabal.tar.gz doctest-example
doctest-example/
doctest-example/CHANGELOG.md
doctest-example/src/
doctest-example/src/Lib.hs
doctest-example/LICENSE
doctest-example/app/
doctest-example/app/Main.hs
doctest-example/flake.lock
doctest-example/test/
doctest-example/test/doctests.hs
doctest-example/test/Main.hs
doctest-example/doctest-example.cabal
doctest-example/flake.nix

[aniketd@nixos:~/Code/oss/doctest-example]$ cabal test doctest-example:test:doctests
Resolving dependencies...
Build profile: -w ghc-8.10.7 -O1
In order, the following will be built (use -v for more details):
 - doctest-example-0.1.0.0 (lib) (configuration changed)
 - doctest-example-0.1.0.0 (test:doctests) (configuration changed)
Configuring library for doctest-example-0.1.0.0..
Preprocessing library for doctest-example-0.1.0.0..
Building library for doctest-example-0.1.0.0..
[1 of 1] Compiling Lib              ( src/Lib.hs, /home/aniketd/Code/oss/doctest-example/dist-newstyle/build/x86_64-linux/ghc-8.10.7/doctest-example-0.1.0.0/build/Lib.o, /home/aniketd/Code/oss/doctest-example/dist-newstyle/build/x86_64-linux/ghc-8.10.7/doctest-example-0.1.0.0/build/Lib.dyn_o ) [System.Random.Stateful changed]
                                                                                                                                                                                    
src/Lib.hs:18:1: warning: [-Wunused-top-binds]
    Defined but not used: ‘runStateGen_’
   |
18 | runStateGen_ g f = runState (f StateGenM) g
   | ^^^^^^^^^^^^
                                                                                                                                                                                    
src/Lib.hs:25:1: warning: [-Wunused-top-binds]
    Defined but not used: ‘foo’
   |
25 | foo = 42
   | ^^^
                                                                                                                                                                                    
src/Lib.hs:31:1: warning: [-Wunused-top-binds]
    Defined but not used: ‘bar’
   |
31 | bar = "bar"
   | ^^^
Configuring test suite 'doctests' for doctest-example-0.1.0.0..
Preprocessing test suite 'doctests' for doctest-example-0.1.0.0..
Building test suite 'doctests' for doctest-example-0.1.0.0..
Linking /home/aniketd/Code/oss/doctest-example/dist-newstyle/build/x86_64-linux/ghc-8.10.7/doctest-example-0.1.0.0/t/doctests/build/doctests/doctests ...
Running 1 test suites...
Test suite doctests: RUNNING...
                                                                                                                                                                                    
src/Lib.hs:5:1: error:
    Could not find module ‘Control.Monad.State.Strict’
    Perhaps you meant Control.Monad.ST.Strict (from base-4.14.3.0)
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.
                                                                                                                                                                                    
src/Lib.hs:6:1: error:
    Could not find module ‘System.Random.Stateful’
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.
src/Lib.hs:13: failure in expression `import System.Random.Stateful'
expected:
 but got:
          ^
          <no location info>: error:
              Could not find module ‘System.Random.Stateful’
              It is not a module in the current program, or in any known package.
                                                                                                                                                                                    
src/Lib.hs:22: failure in expression `foo + 13'
expected: 55
 but got:
          ^
          <interactive>:33:1: error: Variable not in scope: foo
                                                                                                                                                                                    
src/Lib.hs:28: failure in expression `bar'
expected: "bar"
 but got:
          ^
          <interactive>:39:1: error: Variable not in scope: bar
                                                                                                                                                                                    
Examples: 5  Tried: 3  Errors: 0  Failures: 3
                                                                                                                                                                                    
Test suite doctests: FAIL
Test suite logged to:
/home/aniketd/Code/oss/doctest-example/dist-newstyle/build/x86_64-linux/ghc-8.10.7/doctest-example-0.1.0.0/t/doctests/test/doctest-example-0.1.0.0-doctests.log
0 of 1 test suites (0 of 1 test cases) passed.
Error: cabal: Tests failed for test:doctests from doctest-example-0.1.0.0.

NOTES: This is the most confusing. You can see that GHC warns us of unused-top-binds for foo and bar, but doctest complains of them not being in scope during test execution.

[benjaminweb]

did you try cabal repl --with-ghc=doctest? does it work?

[tomjaguarpaw]

Strangely I get this error when running locally on trial (https://github.com/tomjaguarpaw/trial) but not when running in Github Actions CI.