Comments (25)
alpmestan
commented on June 11, 2024
3
#689 and https://phabricator.haskell.org/D5146 solve some more. Down to 42 unexpected failures.
Unexpected results from:
TEST="T10294 T10294a T10420 T12567a T12962 T13168 T13702 T14931 T2552 T5363 T5559 T680 T9203 T9208 T9293 TH_spliceE5_prof TH_spliceE5_prof_ext bkpcabal01 bkpcabal02 bkpcabal03 ghci057 haddock.Cabal haddock.base haddock.compiler plugin-recomp-change plugin-recomp-change-prof plugin-recomp-flags plugin-recomp-impure plugin-recomp-pure plugins01 plugins07 plugins09 plugins11 plugins12 plugins13 plugins14 plugins15 prof-doc-fib prof-doc-last profinline001 scc001 scc002 scc003 scc005"
SUMMARY for test run started at Thu Sep 13 09:04:24 2018 CEST
0:15:55 spent to go through
6539 total tests, which gave rise to
24054 test cases, of which
17532 were skipped
27 had missing libraries
6357 expected passes
94 expected failures
3 caused framework failures
0 caused framework warnings
1 unexpected passes
42 unexpected failures
1 unexpected stat failures
Unexpected passes:
testsuite/tests/stranal/should_compile/T9208.run T9208 [unexpected] (optasm)
Unexpected failures:
testsuite/tests/backpack/cabal/bkpcabal02/bkpcabal02.run bkpcabal02 [bad exit code] (normal)
testsuite/tests/backpack/cabal/bkpcabal03/bkpcabal03.run bkpcabal03 [bad exit code] (normal)
testsuite/tests/backpack/cabal/bkpcabal01/bkpcabal01.run bkpcabal01 [bad stdout] (normal)
testsuite/tests/dynlibs/T13702.run T13702 [bad exit code] (normal)
testsuite/tests/ghci/scripts/ghci057.run ghci057 [bad stdout] (ghci)
testsuite/tests/ghci/scripts/T9293.run T9293 [bad stdout] (ghci)
testsuite/tests/perf/haddock/haddock.base.run haddock.base [[Errno 2] No such file or directory: 'testsuite/tests/perf/haddock/haddock.base.run/haddock.t'] (normal)
testsuite/tests/perf/haddock/haddock.Cabal.run haddock.Cabal [[Errno 2] No such file or directory: 'testsuite/tests/perf/haddock/haddock.Cabal.run/haddock.t'] (normal)
testsuite/tests/perf/haddock/haddock.compiler.run haddock.compiler [[Errno 2] No such file or directory: 'testsuite/tests/perf/haddock/haddock.compiler.run/haddock.t'] (normal)
testsuite/tests/plugins/plugins01.run plugins01 [bad exit code] (normal)
testsuite/tests/plugins/plugins07.run plugins07 [bad exit code] (normal)
testsuite/tests/plugins/plugins09.run plugins09 [bad exit code] (normal)
testsuite/tests/plugins/plugins11.run plugins11 [bad exit code] (normal)
testsuite/tests/plugins/plugins12.run plugins12 [bad exit code] (normal)
testsuite/tests/plugins/plugins13.run plugins13 [bad exit code] (normal)
testsuite/tests/plugins/plugins14.run plugins14 [bad exit code] (normal)
testsuite/tests/plugins/plugins15.run plugins15 [bad exit code] (normal)
testsuite/tests/plugins/T10420.run T10420 [bad exit code] (normal)
testsuite/tests/plugins/T10294.run T10294 [bad exit code] (normal)
testsuite/tests/plugins/T10294a.run T10294a [bad exit code] (normal)
testsuite/tests/plugins/T12567a.run T12567a [bad exit code] (normal)
testsuite/tests/plugins/plugin-recomp-pure.run plugin-recomp-pure [bad exit code] (normal)
testsuite/tests/plugins/plugin-recomp-impure.run plugin-recomp-impure [bad exit code] (normal)
testsuite/tests/plugins/plugin-recomp-flags.run plugin-recomp-flags [bad exit code] (normal)
testsuite/tests/plugins/plugin-recomp-change.run plugin-recomp-change [bad exit code] (normal)
testsuite/tests/plugins/plugin-recomp-change-prof.run plugin-recomp-change-prof [bad exit code] (normal)
testsuite/tests/profiling/should_compile/T14931.run T14931 [bad exit code] (normal)
testsuite/tests/profiling/should_run/scc001.run scc001 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/scc002.run scc002 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/scc003.run scc003 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/scc005.run scc005 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T680.run T680 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T2552.run T2552 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/prof-doc-fib.run prof-doc-fib [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/prof-doc-last.run prof-doc-last [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T5559.run T5559 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/profinline001.run profinline001 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T5363.run T5363 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T12962.run T12962 [bad exit code] (ghci-ext-prof)
testsuite/tests/th/TH_spliceE5_prof_ext.run TH_spliceE5_prof_ext [bad exit code] (normal)
testsuite/tests/th/TH_spliceE5_prof.run TH_spliceE5_prof [bad exit code] (normal)
testsuite/tests/typecheck/T13168/T13168.run T13168 [bad stderr] (normal)
Unexpected stat failures:
testsuite/tests/perf/should_run/T9203.run T9203 [stat too good] (normal)
Framework failures:
testsuite/tests/perf/haddock/haddock.base.run haddock.base [normal] (extra_file does not exist: ../../../../libraries/base/dist-install/haddock.t)
testsuite/tests/perf/haddock/haddock.Cabal.run haddock.Cabal [normal] (extra_file does not exist: ../../../../libraries/Cabal/Cabal/dist-install/haddock.t)
testsuite/tests/perf/haddock/haddock.compiler.run haddock.compiler [normal] (extra_file does not exist: ../../../../compiler/stage2/haddock.t)
I can already smell the wonderful perfume of a completely green hadrian tesΕ§ running in our future CI jobs in the main GHC repo. π π
from hadrian.
simonmar
commented on June 11, 2024
2
In that case can I request that support for this be added? It's really important when developing to be able to work on a single test, or a single directory of tests, and to be able to customize options, compilers and cleanup behaviour. I think we'll need this before we can drop the old build system.
from hadrian.
alpmestan
commented on June 11, 2024
1
hsc2hs is expected under _build/stage1/bin/hsc2hs but not needed by the test rules. This is fixed by #672.
With this patch, f I only run the failing tests from the first comment, I get:
Test results
Unexpected results from:
TEST="CaseBinderCPR Conversions DeriveNull HyperStrUse InlineArrayAlloc InlineByteArrayAlloc InlineCloneArrayAlloc InternalCounters MethSharing Ppr001 Ppr002 Ppr003 Ppr004 Ppr005 Ppr006 Ppr007 Ppr008 Ppr009 Ppr010 Ppr011 Ppr012 Ppr013 Ppr014 Ppr015 Ppr016 Ppr017 Ppr018 Ppr019 Ppr020 Ppr021 Ppr022 Ppr023 Ppr024 Ppr025 Ppr026 Ppr027 Ppr028 Ppr029 Ppr030 Ppr031 Ppr032 Ppr033 Ppr034 Ppr035 Ppr036 Ppr037 Ppr038 Ppr039 Ppr040 Ppr041 Ppr042 Ppr043 Ppr044 Ppr045 Ppr046 Ppr048 T10255 T10268 T10269 T10276 T10278 T10280 T10294 T10294a T10296b T10307 T10309 T10312 T10354 T10357 T10358 T10359 T10396 T10399 T10414 T10420 T10598 T10694 T10728 T11018 T11321 T11332 T11627b T12010 T12370 T12417 T12567a T12603 T12962 T12990 T12996 T13001 T13050p T13143 T13163 T13168 T13191 T13199 T13340 T13543 T13550 T13623 T13702 T13942 T14289 T14289b T14289c T14306 T14346 T14465 T14702 T14931 T14936 T14955 T15056 T15226 T15226a T15303 T2552 T2740 T2976 T3055 T3231 T3429 T3586 T3717 T3738 T3772 T3924 T3994 T4267 T4334 T4474a T4474b T4474c T4830 T4850 T4908 T4930 T4978 T5113 T5205 T5237 T5327 T5363 T5536 T5549 T5558 T5559 T5949 T680 T7257 T7360 T7619 T7797 T7815 T7865 T8124 T8209 T8242 T8472 T876 T8766 T8832 T9078 T9203 T9208 T9293 TH_spliceE5_prof TH_spliceE5_prof_ext atomicxchg boolFormula break001 break005 break006 break026 break027 bundle-export conc069 conc069a conc070 conc072 debug dynbrk002 exampleTest ffi014 ffishutdown forkprocess01 ghci055 ghci057 haddock.Cabal haddock.base haddock.compiler hist001 hist002 inline-check integerConstantFolding join004 join007 lazy-bs-alloc linker_error1 linker_error2 linker_error3 load-main memo001 numa001 numsparks001 nursery-chunks1 plugin-recomp-change plugin-recomp-change-prof plugin-recomp-flags plugin-recomp-impure plugin-recomp-pure plugins01 plugins07 plugins09 plugins11 plugins12 plugins13 plugins14 plugins15 print001 print002 print003 print004 print005 print006 print008 print010 print012 print013 print014 print016 print017 print023 print024 print028 prof-doc-fib prof-doc-last profinline001 scc001 scc002 scc003 scc005 space_leak_001 spec-inline stm056 testwsdeque"
SUMMARY for test run started at Mon Sep 3 15:31:26 2018 CEST
0:04:11 spent to go through
708 total tests, which gave rise to
1660 test cases, of which
878 were skipped
0 had missing libraries
506 expected passes
2 expected failures
3 caused framework failures
0 caused framework warnings
1 unexpected passes
230 unexpected failures
43 unexpected stat failures
Unexpected passes:
testsuite/tests/stranal/should_compile/T9208.run T9208 [unexpected] (optasm)
Unexpected failures:
testsuite/tests/codeGen/should_run/T10414.run T10414 [exit code non-0] (threaded2)
testsuite/tests/concurrent/should_run/conc069.run conc069 [bad exit code] (ghci)
testsuite/tests/codeGen/should_run/T14346.run T14346 [exit code non-0] (threaded1)
testsuite/tests/concurrent/should_run/conc069a.run conc069a [bad exit code] (ghci)
testsuite/tests/codeGen/should_compile/debug.run debug [bad stdout] (normal)
testsuite/tests/concurrent/should_run/conc070.run conc070 [bad stdout] (ghci)
testsuite/tests/concurrent/should_run/conc072.run conc072 [exit code non-0] (threaded2)
testsuite/tests/concurrent/should_run/numsparks001.run numsparks001 [exit code non-0] (threaded1)
testsuite/tests/concurrent/should_run/T5558.run T5558 [exit code non-0] (threaded2)
testsuite/tests/concurrent/should_run/T3429.run T3429 [exit code non-0] (debug_numa)
testsuite/tests/driver/inline-check.run inline-check [stderr mismatch] (normal)
testsuite/tests/dynlibs/T13702.run T13702 [bad exit code] (normal)
testsuite/tests/ffi/should_run/ffi014.run ffi014 [exit code non-0] (threaded1)
testsuite/tests/ghc-api/annotations/exampleTest.run exampleTest [bad exit code] (normal)
testsuite/tests/ghc-api/annotations/T10255.run T10255 [bad exit code] (normal)
testsuite/tests/ghc-api/annotations/T10268.run T10268 [bad exit code] (normal)
testsuite/tests/ghc-api/annotations/T10269.run T10269 [bad stdout] (normal)
testsuite/tests/ghc-api/annotations/T10280.run T10280 [bad exit code] (normal)
testsuite/tests/ghc-api/annotations/T10312.run T10312 [bad exit code] (normal)
testsuite/tests/ghc-api/annotations/T10307.run T10307 [bad exit code] (normal)
testsuite/tests/ghc-api/annotations/T10309.run T10309 [bad exit code] (normal)
testsuite/tests/ghc-api/annotations/boolFormula.run boolFormula [bad exit code] (normal)
testsuite/tests/ghc-api/annotations/T10357.run T10357 [bad exit code] (normal)
testsuite/tests/ghc-api/annotations/T10358.run T10358 [bad exit code] (normal)
testsuite/tests/ghc-api/annotations/T10278.run T10278 [bad exit code] (normal)
testsuite/tests/ghc-api/annotations/T10354.run T10354 [bad exit code] (normal)
testsuite/tests/ghc-api/annotations/T10396.run T10396 [bad exit code] (normal)
testsuite/tests/ghc-api/annotations/T10399.run T10399 [bad exit code] (normal)
testsuite/tests/ghc-api/annotations/T11018.run T11018 [bad exit code] (normal)
testsuite/tests/ghc-api/annotations/bundle-export.run bundle-export [bad exit code] (normal)
testsuite/tests/ghc-api/annotations/T10276.run T10276 [bad exit code] (normal)
testsuite/tests/ghc-api/annotations/T10598.run T10598 [bad exit code] (normal)
testsuite/tests/ghc-api/annotations/T11321.run T11321 [bad exit code] (normal)
testsuite/tests/ghc-api/annotations/T11332.run T11332 [bad exit code] (normal)
testsuite/tests/ghc-api/annotations/load-main.run load-main [bad exit code] (normal)
testsuite/tests/ghc-api/annotations/T12417.run T12417 [bad exit code] (normal)
testsuite/tests/ghc-api/annotations/T13163.run T13163 [bad exit code] (normal)
testsuite/tests/ghc-api/annotations/T15303.run T15303 [bad exit code] (normal)
testsuite/tests/ghci/scripts/ghci055.run ghci055 [bad stdout] (ghci)
testsuite/tests/ghci/scripts/ghci057.run ghci057 [bad stdout] (ghci)
testsuite/tests/ghci/scripts/T2976.run T2976 [bad stdout] (ghci)
testsuite/tests/ghci/scripts/T9293.run T9293 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print001.run print001 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print001.run print001 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print002.run print002 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print003.run print003 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print002.run print002 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print003.run print003 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print004.run print004 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print005.run print005 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print006.run print006 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print005.run print005 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print004.run print004 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print008.run print008 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print006.run print006 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print010.run print010 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print012.run print012 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print013.run print013 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print008.run print008 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print010.run print010 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print012.run print012 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print013.run print013 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print014.run print014 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print016.run print016 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print017.run print017 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print014.run print014 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print023.run print023 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print017.run print017 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print016.run print016 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print024.run print024 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print023.run print023 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print028.run print028 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/break001.run break001 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print024.run print024 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/break005.run break005 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print028.run print028 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/break001.run break001 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/break006.run break006 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/break026.run break026 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/break005.run break005 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/break027.run break027 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/break006.run break006 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/break026.run break026 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/dynbrk002.run dynbrk002 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/break027.run break027 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/hist001.run hist001 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/hist002.run hist002 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/dynbrk002.run dynbrk002 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/T2740.run T2740 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/hist001.run hist001 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/hist002.run hist002 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/T2740.run T2740 [bad stdout] (ghci-ext)
testsuite/tests/lib/integer/integerConstantFolding.run integerConstantFolding [bad stdout] (normal)
testsuite/tests/numeric/should_compile/T14465.run T14465 [bad stdout] (normal)
testsuite/tests/perf/haddock/haddock.base.run haddock.base [[Errno 2] No such file or directory: 'testsuite/tests/perf/haddock/haddock.base.run/haddock.t'] (normal)
testsuite/tests/perf/haddock/haddock.Cabal.run haddock.Cabal [[Errno 2] No such file or directory: 'testsuite/tests/perf/haddock/haddock.Cabal.run/haddock.t'] (normal)
testsuite/tests/perf/haddock/haddock.compiler.run haddock.compiler [[Errno 2] No such file or directory: 'testsuite/tests/perf/haddock/haddock.compiler.run/haddock.t'] (normal)
testsuite/tests/plugins/plugins01.run plugins01 [bad exit code] (normal)
testsuite/tests/plugins/plugins07.run plugins07 [bad exit code] (normal)
testsuite/tests/plugins/plugins09.run plugins09 [bad exit code] (normal)
testsuite/tests/plugins/plugins11.run plugins11 [bad exit code] (normal)
testsuite/tests/plugins/plugins12.run plugins12 [bad exit code] (normal)
testsuite/tests/plugins/plugins13.run plugins13 [bad exit code] (normal)
testsuite/tests/plugins/plugins14.run plugins14 [bad exit code] (normal)
testsuite/tests/plugins/plugins15.run plugins15 [bad exit code] (normal)
testsuite/tests/plugins/T10420.run T10420 [bad exit code] (normal)
testsuite/tests/plugins/T10294.run T10294 [bad exit code] (normal)
testsuite/tests/plugins/T10294a.run T10294a [bad exit code] (normal)
testsuite/tests/plugins/plugin-recomp-impure.run plugin-recomp-impure [bad exit code] (normal)
testsuite/tests/plugins/plugin-recomp-pure.run plugin-recomp-pure [bad exit code] (normal)
testsuite/tests/plugins/T12567a.run T12567a [bad exit code] (normal)
testsuite/tests/printer/Ppr001.run Ppr001 [bad exit code] (normal)
testsuite/tests/printer/Ppr002.run Ppr002 [bad exit code] (normal)
testsuite/tests/printer/Ppr003.run Ppr003 [bad exit code] (normal)
testsuite/tests/printer/Ppr004.run Ppr004 [bad exit code] (normal)
testsuite/tests/plugins/plugin-recomp-flags.run plugin-recomp-flags [bad exit code] (normal)
testsuite/tests/plugins/plugin-recomp-change.run plugin-recomp-change [bad exit code] (normal)
testsuite/tests/printer/Ppr005.run Ppr005 [bad exit code] (normal)
testsuite/tests/printer/Ppr006.run Ppr006 [bad exit code] (normal)
testsuite/tests/printer/Ppr007.run Ppr007 [bad exit code] (normal)
testsuite/tests/printer/Ppr008.run Ppr008 [bad exit code] (normal)
testsuite/tests/printer/Ppr009.run Ppr009 [bad exit code] (normal)
testsuite/tests/plugins/plugin-recomp-change-prof.run plugin-recomp-change-prof [bad exit code] (normal)
testsuite/tests/printer/Ppr010.run Ppr010 [bad exit code] (normal)
testsuite/tests/printer/Ppr011.run Ppr011 [bad exit code] (normal)
testsuite/tests/printer/Ppr013.run Ppr013 [bad exit code] (normal)
testsuite/tests/printer/Ppr012.run Ppr012 [bad exit code] (normal)
testsuite/tests/printer/Ppr014.run Ppr014 [bad exit code] (normal)
testsuite/tests/printer/Ppr015.run Ppr015 [bad exit code] (normal)
testsuite/tests/printer/Ppr017.run Ppr017 [bad exit code] (normal)
testsuite/tests/printer/Ppr016.run Ppr016 [bad exit code] (normal)
testsuite/tests/printer/Ppr018.run Ppr018 [bad exit code] (normal)
testsuite/tests/printer/Ppr019.run Ppr019 [bad exit code] (normal)
testsuite/tests/printer/Ppr020.run Ppr020 [bad exit code] (normal)
testsuite/tests/printer/Ppr021.run Ppr021 [bad exit code] (normal)
testsuite/tests/printer/Ppr022.run Ppr022 [bad exit code] (normal)
testsuite/tests/printer/Ppr023.run Ppr023 [bad exit code] (normal)
testsuite/tests/printer/Ppr024.run Ppr024 [bad exit code] (normal)
testsuite/tests/printer/Ppr025.run Ppr025 [bad exit code] (normal)
testsuite/tests/printer/Ppr026.run Ppr026 [bad exit code] (normal)
testsuite/tests/printer/Ppr027.run Ppr027 [bad exit code] (normal)
testsuite/tests/printer/Ppr028.run Ppr028 [bad exit code] (normal)
testsuite/tests/printer/Ppr029.run Ppr029 [bad exit code] (normal)
testsuite/tests/printer/Ppr030.run Ppr030 [bad exit code] (normal)
testsuite/tests/printer/Ppr031.run Ppr031 [bad exit code] (normal)
testsuite/tests/printer/Ppr033.run Ppr033 [bad exit code] (normal)
testsuite/tests/printer/Ppr032.run Ppr032 [bad exit code] (normal)
testsuite/tests/printer/Ppr034.run Ppr034 [bad exit code] (normal)
testsuite/tests/printer/Ppr035.run Ppr035 [bad exit code] (normal)
testsuite/tests/printer/Ppr036.run Ppr036 [bad exit code] (normal)
testsuite/tests/printer/Ppr037.run Ppr037 [bad exit code] (normal)
testsuite/tests/printer/Ppr038.run Ppr038 [bad exit code] (normal)
testsuite/tests/printer/Ppr039.run Ppr039 [bad exit code] (normal)
testsuite/tests/printer/Ppr041.run Ppr041 [bad exit code] (normal)
testsuite/tests/printer/Ppr040.run Ppr040 [bad exit code] (normal)
testsuite/tests/printer/Ppr042.run Ppr042 [bad exit code] (normal)
testsuite/tests/printer/Ppr043.run Ppr043 [bad exit code] (normal)
testsuite/tests/printer/Ppr045.run Ppr045 [bad exit code] (normal)
testsuite/tests/printer/Ppr044.run Ppr044 [bad exit code] (normal)
testsuite/tests/printer/Ppr046.run Ppr046 [bad exit code] (normal)
testsuite/tests/printer/Ppr048.run Ppr048 [bad exit code] (normal)
testsuite/tests/printer/T13199.run T13199 [bad exit code] (normal)
testsuite/tests/printer/T13050p.run T13050p [bad exit code] (normal)
testsuite/tests/printer/T13550.run T13550 [bad exit code] (normal)
testsuite/tests/printer/T13942.run T13942 [bad exit code] (normal)
testsuite/tests/printer/T14289.run T14289 [bad exit code] (normal)
testsuite/tests/printer/T14289b.run T14289b [bad exit code] (normal)
testsuite/tests/printer/T14289c.run T14289c [bad exit code] (normal)
testsuite/tests/printer/T14306.run T14306 [bad exit code] (normal)
testsuite/tests/profiling/should_compile/T14931.run T14931 [bad exit code] (normal)
testsuite/tests/profiling/should_run/scc001.run scc001 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/scc005.run scc005 [bad profile] (profasm)
testsuite/tests/profiling/should_run/scc002.run scc002 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/scc003.run scc003 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/scc005.run scc005 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T680.run T680 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/prof-doc-fib.run prof-doc-fib [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T2552.run T2552 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T5559.run T5559 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/prof-doc-last.run prof-doc-last [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/profinline001.run profinline001 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T12962.run T12962 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T11627b.run T11627b [bad exit code] (prof_hc_hb)
testsuite/tests/profiling/should_run/T5363.run T5363 [bad exit code] (ghci-ext-prof)
testsuite/tests/rts/atomicxchg.run atomicxchg [exit code non-0] (threaded1)
testsuite/tests/rts/testwsdeque.run testwsdeque [exit code non-0] (threaded1)
testsuite/tests/rts/T4850.run T4850 [bad exit code] (normal)
testsuite/tests/rts/ffishutdown.run ffishutdown [exit code non-0] (threaded1)
testsuite/tests/rts/T8209.run T8209 [bad stderr] (ghci)
testsuite/tests/rts/T8242.run T8242 [bad stderr] (ghci)
testsuite/tests/rts/T9078.run T9078 [exit code non-0] (threaded1)
testsuite/tests/rts/linker_error1.run linker_error1 [bad exit code] (normal)
testsuite/tests/rts/T8124.run T8124 [exit code non-0] (threaded1)
testsuite/tests/rts/linker_error2.run linker_error2 [bad exit code] (normal)
testsuite/tests/rts/linker_error3.run linker_error3 [bad exit code] (normal)
testsuite/tests/rts/T10728.run T10728 [exit code non-0] (threaded2)
testsuite/tests/rts/T10296b.run T10296b [exit code non-0] (threaded2)
testsuite/tests/rts/numa001.run numa001 [exit code non-0] (debug_numa)
testsuite/tests/rts/T14702.run T14702 [exit code non-0] (threaded1)
testsuite/tests/rts/InternalCounters.run InternalCounters [bad stdout] (normal)
testsuite/tests/rts/nursery-chunks1.run nursery-chunks1 [exit code non-0] (threaded1)
testsuite/tests/simplCore/should_compile/T3717.run T3717 [stderr mismatch] (optasm)
testsuite/tests/simplCore/should_compile/spec-inline.run spec-inline [stderr mismatch] (optasm)
testsuite/tests/simplCore/should_compile/T4908.run T4908 [stderr mismatch] (optasm)
testsuite/tests/simplCore/should_compile/T4930.run T4930 [stderr mismatch] (optasm)
testsuite/tests/simplCore/should_compile/T3055.run T3055 [bad stdout] (normal)
testsuite/tests/simplCore/should_compile/T3772.run T3772 [bad stdout] (normal)
testsuite/tests/simplCore/should_compile/T5327.run T5327 [bad exit code] (normal)
testsuite/tests/simplCore/should_compile/T7360.run T7360 [stderr mismatch] (optasm)
testsuite/tests/simplCore/should_compile/T7865.run T7865 [bad stdout] (normal)
testsuite/tests/simplCore/should_compile/T8832.run T8832 [bad stdout] (normal)
testsuite/tests/simplCore/should_compile/T12603.run T12603 [bad stdout] (normal)
testsuite/tests/simplCore/should_compile/T13143.run T13143 [stderr mismatch] (optasm)
testsuite/tests/simplCore/should_compile/T13340.run T13340 [bad exit code] (normal)
testsuite/tests/simplCore/should_compile/T13543.run T13543 [stderr mismatch] (optasm)
testsuite/tests/simplCore/should_compile/T15056.run T15056 [stderr mismatch] (normal)
testsuite/tests/stranal/should_compile/T10694.run T10694 [stderr mismatch] (optasm)
testsuite/tests/stranal/sigs/HyperStrUse.run HyperStrUse [stderr mismatch] (optasm)
testsuite/tests/stranal/sigs/T12370.run T12370 [stderr mismatch] (optasm)
testsuite/tests/stranal/sigs/CaseBinderCPR.run CaseBinderCPR [stderr mismatch] (optasm)
testsuite/tests/th/TH_spliceE5_prof_ext.run TH_spliceE5_prof_ext [bad exit code] (normal)
testsuite/tests/th/TH_spliceE5_prof.run TH_spliceE5_prof [bad exit code] (normal)
testsuite/tests/typecheck/T13168/T13168.run T13168 [bad stderr] (normal)
libraries/base/tests/memo001.run memo001 [bad exit code] (normal)
libraries/base/tests/IO/T12010/T12010.run T12010 [bad exit code] (threaded1)
libraries/process/tests/T3231.run T3231 [exit code non-0] (threaded1)
libraries/stm/tests/stm056.run stm056 [exit code non-0] (threaded1)
libraries/unix/tests/forkprocess01.run forkprocess01 [exit code non-0] (threaded1_ls)
libraries/process/tests/T3994.run T3994 [exit code non-0] (threaded1)
testsuite/tests/rts/T7815.run T7815 [exit code non-0] (threaded1)
Unexpected stat failures:
testsuite/tests/callarity/perf/T3924.run T3924 [stat not good enough] (normal)
testsuite/tests/perf/join_points/join004.run join004 [stat not good enough] (optasm)
testsuite/tests/perf/should_run/T10359.run T10359 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T14955.run T14955 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T3586.run T3586 [stat not good enough] (normal)
testsuite/tests/perf/join_points/join007.run join007 [stat not good enough] (optasm)
testsuite/tests/perf/should_run/T4830.run T4830 [stat not good enough] (normal)
testsuite/tests/perf/should_run/lazy-bs-alloc.run lazy-bs-alloc [stat not good enough] (normal)
testsuite/tests/perf/should_run/T3738.run T3738 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T876.run T876 [stat not good enough] (normal)
testsuite/tests/perf/should_run/MethSharing.run MethSharing [stat not good enough] (normal)
testsuite/tests/perf/should_run/T5113.run T5113 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T4978.run T4978 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T5205.run T5205 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T4474a.run T4474a [stat not good enough] (normal)
testsuite/tests/perf/should_run/T4474b.run T4474b [stat not good enough] (normal)
testsuite/tests/perf/should_run/T4474c.run T4474c [stat not good enough] (normal)
testsuite/tests/perf/should_run/T5237.run T5237 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T7257.run T7257 [stat not good enough] (normal)
testsuite/tests/perf/should_run/Conversions.run Conversions [stat not good enough] (normal)
testsuite/tests/perf/should_run/T5536.run T5536 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T7797.run T7797 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T5949.run T5949 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T4267.run T4267 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T7619.run T7619 [stat not good enough] (normal)
testsuite/tests/perf/should_run/InlineByteArrayAlloc.run InlineByteArrayAlloc [stat not good enough] (normal)
testsuite/tests/perf/should_run/InlineArrayAlloc.run InlineArrayAlloc [stat not good enough] (normal)
testsuite/tests/perf/should_run/InlineCloneArrayAlloc.run InlineCloneArrayAlloc [stat not good enough] (normal)
testsuite/tests/perf/should_run/T8472.run T8472 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T9203.run T9203 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T12996.run T12996 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T13001.run T13001 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T12990.run T12990 [stat not good enough] (normal)
testsuite/tests/perf/should_run/DeriveNull.run DeriveNull [stat not good enough] (normal)
testsuite/tests/perf/should_run/T14936.run T14936 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T13623.run T13623 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T15226.run T15226 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T15226a.run T15226a [stat not good enough] (normal)
testsuite/tests/perf/space_leaks/space_leak_001.run space_leak_001 [stat not good enough] (normal)
testsuite/tests/perf/space_leaks/T4334.run T4334 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T5549.run T5549 [stat not good enough] (normal)
libraries/base/tests/T8766.run T8766 [stat not good enough] (normal)
libraries/base/tests/T13191.run T13191 [stat not good enough] (normal)
So, 500 unexpected failures are gone with the PR.
Now, as a note for future me, I've noticed that check-api-annotations is expected under inplace/.... This is because of some code in testsuite/mk/boilerplate.mk which just doesn't leave any other option at the moment. I guess I'll have to modify it to allow hadrian to completely bypass this and specify a path of its own with a new dedicated variable.
from hadrian.
alpmestan
commented on June 11, 2024
1
OK; make sure you use --flavour=perf with the test target.
from hadrian.
alpmestan
commented on June 11, 2024
1
#677 makes all the ghc-compact tests pass.
from hadrian.
alpmestan
commented on June 11, 2024
1
@simonmar I opened #685 and #686 to remember about this (which involves two distinct improvements, AFAICT) and track the progress. Feel free to subscribe to those if you want to get a notification when this gets implemented. Or to even leave comments if you have anything to add to those issues.
from hadrian.
alpmestan
commented on June 11, 2024
1
@simonmar As I pointed out in earlier comments, we can already specify a particular WAY, through --test-way. And the test compiler with --test-compiler. We however don't have:
- support for controling whether we clean up or not (easy to implement, just not done yet, and we don't have an issue for that yet either)
- support for running a whole directory of tests (which is what most of #685 is about)
Thanks for the input! It's helpful to have very precise TODOs like this :-)
from hadrian.
alpmestan
commented on June 11, 2024
1
Earlier today, I ran the test rule (with the perf flavour as usual) and... things went pretty well.
SUMMARY for test run started at Tue Oct 23 11:07:46 2018 CEST
0:15:00 spent to go through
6601 total tests, which gave rise to
25634 test cases, of which
19054 were skipped
27 had missing libraries
6431 expected passes
89 expected failures
3 caused framework failures
0 caused framework warnings
1 unexpected passes
31 unexpected failures
1 unexpected stat failures
I looked at all the failures and managed to classify them in a few categories.
Require a GHC built with -dynamic
testsuite/tests/typecheck/should_compile/T11462.run T11462 [exit code non-0] (normal)
testsuite/tests/typecheck/should_compile/T11525.run T11525 [exit code non-0] (normal)
testsuite/tests/th/TH_linker/path_with_commas.run path_with_commas [bad exit code] (normal) -- not entirely sure about this one
testsuite/tests/plugins/plugins05.run plugins05 [exit code non-0] (dyn)
testsuite/tests/plugins/plugins06.run plugins06 [exit code non-0] (dyn)
This will be fixed once the joint work with David lands (Cabal + ghc + hadrian patch). See #691.
Cannot locate libffi or libHSrts
testsuite/tests/driver/dynHelloWorld.run dynHelloWorld [exit code non-0] (dyn)
testsuite/tests/dynlibs/T5373.run T5373 [bad exit code] (normal)
testsuite/tests/dynlibs/T3807.run T3807 [bad exit code] (normal)
testsuite/tests/dynlibs/T13702.run T13702 [bad exit code] (normal)
We don't build ghc-iserv-prof
The make build system builds a -prof version of ghc-iserv which it calls ghc-iserv-prof, and this is required by a few tests. I'm going to look into how we can produce two executables from just one package...
testsuite/tests/profiling/should_run/scc001.run scc001 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/scc002.run scc002 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/scc003.run scc003 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/scc005.run scc005 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T680.run T680 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T2552.run T2552 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/prof-doc-fib.run prof-doc-fib [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/prof-doc-last.run prof-doc-last [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T5559.run T5559 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/profinline001.run profinline001 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T5363.run T5363 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T12962.run T12962 [bad exit code] (ghci-ext-prof)
testsuite/tests/th/TH_spliceE5_prof_ext.run TH_spliceE5_prof_ext [bad exit code] (normal)
Less output than expected
Those in generally look benign, but I have no idea why the output differs.
testsuite/tests/annotations/should_compile/th/annth_compunits.run annth_compunits [bad exit code] (normal)
testsuite/tests/annotations/should_compile/th/annth_make.run annth_make [bad stdout] (normal)
testsuite/tests/ghci/scripts/ghci057.run ghci057 [bad stdout] (ghci)
testsuite/tests/ghci/scripts/T9293.run T9293 [bad stdout] (ghci)
testsuite/tests/typecheck/T13168/T13168.run T13168 [bad stderr] (normal)
Bad handling of -osuf + -prof options ?
testsuite/tests/th/TH_spliceE5_prof.run TH_spliceE5_prof [bad exit code] (normal)
haddock perf failures
We quite likely don't fit with the expected inplace layout, something like that?
testsuite/tests/perf/haddock/haddock.base.run haddock.base [[Errno 2] No such file or directory: 'testsuite/tests/perf/haddock/haddock.base.run/haddock.t'] (normal)
testsuite/tests/perf/haddock/haddock.Cabal.run haddock.Cabal [[Errno 2] No such file or directory: 'testsuite/tests/perf/haddock/haddock.Cabal.run/haddock.t'] (normal)
testsuite/tests/perf/haddock/haddock.compiler.run haddock.compiler [[Errno 2] No such file or directory: 'testsuite/tests/perf/haddock/haddock.compiler.run/haddock.t'] (normal)
from hadrian.
snowleopard
commented on June 11, 2024
Many thanks @alpmestan!
I will check if there are any outdated testsuite related issues and will close them.
There is also this PR #639, which now seems abandoned. Feel free to close it.
from hadrian.
alpmestan
commented on June 11, 2024
https://phabricator.haskell.org/D5127 should fix all the ghc-api-annotations failures. We do set the CHECK_API_ANNOTATIONS environment variable but it is not picked up. testsuite/mk/boilerplate.mk, which is included by many (most? all?) test makefiles, didn't give us a chance to specify anything.
from hadrian.
alpmestan
commented on June 11, 2024
With the aforementionned patch, I'm down to 184 unexpected failures
Test results
SUMMARY for test run started at Tue Sep 4 18:18:34 2018 CEST
0:19:48 spent to go through
6536 total tests, which gave rise to
24042 test cases, of which
17523 were skipped
27 had missing libraries
6168 expected passes
96 expected failures
3 caused framework failures
0 caused framework warnings
1 unexpected passes
184 unexpected failures
43 unexpected stat failures
Unexpected passes:
testsuite/tests/stranal/should_compile/T9208.run T9208 [unexpected] (optasm)
Unexpected failures:
testsuite/tests/codeGen/should_compile/debug.run debug [bad stdout] (normal)
testsuite/tests/codeGen/should_run/T10414.run T10414 [exit code non-0] (threaded2)
testsuite/tests/codeGen/should_run/T14346.run T14346 [exit code non-0] (threaded1)
testsuite/tests/concurrent/should_run/conc069.run conc069 [bad exit code] (ghci)
testsuite/tests/concurrent/should_run/conc069a.run conc069a [bad exit code] (ghci)
testsuite/tests/concurrent/should_run/conc070.run conc070 [bad stdout] (ghci)
testsuite/tests/concurrent/should_run/conc072.run conc072 [exit code non-0] (threaded2)
testsuite/tests/concurrent/should_run/numsparks001.run numsparks001 [exit code non-0] (threaded1)
testsuite/tests/concurrent/should_run/T3429.run T3429 [exit code non-0] (debug_numa)
testsuite/tests/concurrent/should_run/T5558.run T5558 [exit code non-0] (threaded2)
testsuite/tests/driver/inline-check.run inline-check [stderr mismatch] (normal)
testsuite/tests/dynlibs/T13702.run T13702 [bad exit code] (normal)
testsuite/tests/ffi/should_run/ffi014.run ffi014 [exit code non-0] (threaded1)
testsuite/tests/ghci/scripts/ghci055.run ghci055 [bad stdout] (ghci)
testsuite/tests/ghci/scripts/ghci057.run ghci057 [bad stdout] (ghci)
testsuite/tests/ghci/scripts/T2976.run T2976 [bad stdout] (ghci)
testsuite/tests/ghci/scripts/T9293.run T9293 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print001.run print001 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print002.run print002 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print001.run print001 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print003.run print003 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print002.run print002 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print004.run print004 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print005.run print005 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print003.run print003 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print004.run print004 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print005.run print005 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print006.run print006 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print008.run print008 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print006.run print006 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print008.run print008 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print010.run print010 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print012.run print012 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print010.run print010 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print012.run print012 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print013.run print013 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print014.run print014 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print016.run print016 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print013.run print013 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print014.run print014 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print016.run print016 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print017.run print017 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print017.run print017 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print023.run print023 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print024.run print024 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print023.run print023 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print024.run print024 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/print028.run print028 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/print028.run print028 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/break001.run break001 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/break001.run break001 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/break005.run break005 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/break006.run break006 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/break005.run break005 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/break006.run break006 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/break026.run break026 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/break027.run break027 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/break026.run break026 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/break027.run break027 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/dynbrk002.run dynbrk002 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/dynbrk002.run dynbrk002 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/hist001.run hist001 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/hist002.run hist002 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/T2740.run T2740 [bad stdout] (ghci)
testsuite/tests/ghci.debugger/scripts/hist001.run hist001 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/hist002.run hist002 [bad stdout] (ghci-ext)
testsuite/tests/ghci.debugger/scripts/T2740.run T2740 [bad stdout] (ghci-ext)
testsuite/tests/lib/integer/integerConstantFolding.run integerConstantFolding [bad stdout] (normal)
testsuite/tests/numeric/should_compile/T14465.run T14465 [bad stdout] (normal)
testsuite/tests/perf/haddock/haddock.base.run haddock.base [[Errno 2] No such file or directory: 'testsuite/tests/perf/haddock/haddock.base.run/haddock.t'] (normal)
testsuite/tests/perf/haddock/haddock.Cabal.run haddock.Cabal [[Errno 2] No such file or directory: 'testsuite/tests/perf/haddock/haddock.Cabal.run/haddock.t'] (normal)
testsuite/tests/perf/haddock/haddock.compiler.run haddock.compiler [[Errno 2] No such file or directory: 'testsuite/tests/perf/haddock/haddock.compiler.run/haddock.t'] (normal)
testsuite/tests/plugins/plugins01.run plugins01 [bad exit code] (normal)
testsuite/tests/plugins/plugins07.run plugins07 [bad exit code] (normal)
testsuite/tests/plugins/plugins09.run plugins09 [bad exit code] (normal)
testsuite/tests/plugins/plugins11.run plugins11 [bad exit code] (normal)
testsuite/tests/plugins/plugins12.run plugins12 [bad exit code] (normal)
testsuite/tests/plugins/plugins13.run plugins13 [bad exit code] (normal)
testsuite/tests/plugins/T10420.run T10420 [bad exit code] (normal)
testsuite/tests/plugins/plugins14.run plugins14 [bad exit code] (normal)
testsuite/tests/plugins/plugins15.run plugins15 [bad exit code] (normal)
testsuite/tests/plugins/T10294.run T10294 [bad exit code] (normal)
testsuite/tests/plugins/T10294a.run T10294a [bad exit code] (normal)
testsuite/tests/plugins/plugin-recomp-pure.run plugin-recomp-pure [bad exit code] (normal)
testsuite/tests/plugins/T12567a.run T12567a [bad exit code] (normal)
testsuite/tests/plugins/plugin-recomp-impure.run plugin-recomp-impure [bad exit code] (normal)
testsuite/tests/plugins/plugin-recomp-flags.run plugin-recomp-flags [bad exit code] (normal)
testsuite/tests/plugins/plugin-recomp-change.run plugin-recomp-change [bad exit code] (normal)
testsuite/tests/plugins/plugin-recomp-change-prof.run plugin-recomp-change-prof [bad exit code] (normal)
testsuite/tests/profiling/should_compile/T14931.run T14931 [bad exit code] (normal)
testsuite/tests/profiling/should_run/scc001.run scc001 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/scc002.run scc002 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/scc003.run scc003 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/scc005.run scc005 [bad profile] (profasm)
testsuite/tests/profiling/should_run/scc005.run scc005 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T680.run T680 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T2552.run T2552 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/prof-doc-fib.run prof-doc-fib [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/prof-doc-last.run prof-doc-last [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T5559.run T5559 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/profinline001.run profinline001 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T5363.run T5363 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T11627b.run T11627b [bad exit code] (prof_hc_hb)
testsuite/tests/profiling/should_run/T12962.run T12962 [bad exit code] (ghci-ext-prof)
testsuite/tests/rts/testwsdeque.run testwsdeque [exit code non-0] (threaded1)
testsuite/tests/rts/atomicxchg.run atomicxchg [exit code non-0] (threaded1)
testsuite/tests/rts/T4850.run T4850 [bad exit code] (normal)
testsuite/tests/rts/ffishutdown.run ffishutdown [exit code non-0] (threaded1)
testsuite/tests/rts/T8209.run T8209 [bad stderr] (ghci)
testsuite/tests/rts/T8242.run T8242 [bad stderr] (ghci)
testsuite/tests/rts/T8124.run T8124 [exit code non-0] (threaded1)
testsuite/tests/rts/T9078.run T9078 [exit code non-0] (threaded1)
testsuite/tests/rts/linker_error1.run linker_error1 [bad exit code] (normal)
testsuite/tests/rts/linker_error2.run linker_error2 [bad exit code] (normal)
testsuite/tests/rts/linker_error3.run linker_error3 [bad exit code] (normal)
testsuite/tests/rts/T10728.run T10728 [exit code non-0] (threaded2)
testsuite/tests/rts/T10296b.run T10296b [exit code non-0] (threaded2)
testsuite/tests/rts/numa001.run numa001 [exit code non-0] (debug_numa)
testsuite/tests/rts/T14900.run T14900 [exit code non-0] (normal)
testsuite/tests/rts/InternalCounters.run InternalCounters [bad stdout] (normal)
testsuite/tests/rts/T14702.run T14702 [exit code non-0] (threaded1)
testsuite/tests/rts/nursery-chunks1.run nursery-chunks1 [exit code non-0] (threaded1)
testsuite/tests/simplCore/should_compile/T3717.run T3717 [stderr mismatch] (optasm)
testsuite/tests/simplCore/should_compile/T4930.run T4930 [stderr mismatch] (optasm)
testsuite/tests/simplCore/should_compile/spec-inline.run spec-inline [stderr mismatch] (optasm)
testsuite/tests/simplCore/should_compile/T4908.run T4908 [stderr mismatch] (optasm)
testsuite/tests/simplCore/should_compile/T3055.run T3055 [bad stdout] (normal)
testsuite/tests/simplCore/should_compile/T3772.run T3772 [bad stdout] (normal)
testsuite/tests/simplCore/should_compile/T5327.run T5327 [bad exit code] (normal)
testsuite/tests/simplCore/should_compile/T7360.run T7360 [stderr mismatch] (optasm)
testsuite/tests/simplCore/should_compile/T7865.run T7865 [bad stdout] (normal)
testsuite/tests/simplCore/should_compile/T8832.run T8832 [bad stdout] (normal)
testsuite/tests/simplCore/should_compile/T12603.run T12603 [bad stdout] (normal)
testsuite/tests/simplCore/should_compile/T13143.run T13143 [stderr mismatch] (optasm)
testsuite/tests/simplCore/should_compile/T13340.run T13340 [bad exit code] (normal)
testsuite/tests/simplCore/should_compile/T13543.run T13543 [stderr mismatch] (optasm)
testsuite/tests/simplCore/should_compile/T15056.run T15056 [stderr mismatch] (normal)
testsuite/tests/stranal/should_compile/T10694.run T10694 [stderr mismatch] (optasm)
testsuite/tests/stranal/sigs/HyperStrUse.run HyperStrUse [stderr mismatch] (optasm)
testsuite/tests/stranal/sigs/T12370.run T12370 [stderr mismatch] (optasm)
testsuite/tests/stranal/sigs/CaseBinderCPR.run CaseBinderCPR [stderr mismatch] (optasm)
testsuite/tests/th/TH_spliceE5_prof_ext.run TH_spliceE5_prof_ext [bad exit code] (normal)
testsuite/tests/th/TH_spliceE5_prof.run TH_spliceE5_prof [bad exit code] (normal)
testsuite/tests/typecheck/T13168/T13168.run T13168 [bad stderr] (normal)
libraries/base/tests/memo001.run memo001 [bad exit code] (normal)
libraries/base/tests/IO/T12010/T12010.run T12010 [bad exit code] (threaded1)
libraries/ghc-compact/tests/compact_simple.run compact_simple [exit code non-0] (normal)
libraries/ghc-compact/tests/compact_simple.run compact_simple [exit code non-0] (sanity)
libraries/ghc-compact/tests/compact_loop.run compact_loop [exit code non-0] (normal)
libraries/ghc-compact/tests/compact_append.run compact_append [exit code non-0] (normal)
libraries/ghc-compact/tests/compact_loop.run compact_loop [exit code non-0] (sanity)
libraries/ghc-compact/tests/compact_append.run compact_append [exit code non-0] (sanity)
libraries/ghc-compact/tests/compact_autoexpand.run compact_autoexpand [exit code non-0] (normal)
libraries/ghc-compact/tests/compact_simple_array.run compact_simple_array [exit code non-0] (normal)
libraries/ghc-compact/tests/compact_autoexpand.run compact_autoexpand [exit code non-0] (sanity)
libraries/ghc-compact/tests/compact_simple_array.run compact_simple_array [exit code non-0] (sanity)
libraries/ghc-compact/tests/compact_huge_array.run compact_huge_array [exit code non-0] (normal)
libraries/ghc-compact/tests/compact_serialize.run compact_serialize [exit code non-0] (normal)
libraries/ghc-compact/tests/compact_largemap.run compact_largemap [exit code non-0] (normal)
libraries/ghc-compact/tests/compact_huge_array.run compact_huge_array [exit code non-0] (sanity)
libraries/ghc-compact/tests/compact_largemap.run compact_largemap [exit code non-0] (sanity)
libraries/ghc-compact/tests/compact_serialize.run compact_serialize [exit code non-0] (sanity)
libraries/ghc-compact/tests/compact_threads.run compact_threads [exit code non-0] (normal)
libraries/ghc-compact/tests/compact_function.run compact_function [exit code non-0] (normal)
libraries/ghc-compact/tests/compact_cycle.run compact_cycle [exit code non-0] (normal)
libraries/ghc-compact/tests/compact_threads.run compact_threads [exit code non-0] (sanity)
libraries/ghc-compact/tests/compact_function.run compact_function [exit code non-0] (sanity)
libraries/ghc-compact/tests/compact_cycle.run compact_cycle [exit code non-0] (sanity)
libraries/ghc-compact/tests/compact_mutable.run compact_mutable [exit code non-0] (normal)
libraries/ghc-compact/tests/compact_pinned.run compact_pinned [exit code non-0] (normal)
libraries/ghc-compact/tests/compact_gc.run compact_gc [exit code non-0] (normal)
libraries/ghc-compact/tests/compact_mutable.run compact_mutable [exit code non-0] (sanity)
libraries/ghc-compact/tests/compact_pinned.run compact_pinned [exit code non-0] (sanity)
libraries/ghc-compact/tests/compact_gc.run compact_gc [exit code non-0] (sanity)
libraries/ghc-compact/tests/compact_share.run compact_share [exit code non-0] (normal)
libraries/ghc-compact/tests/compact_bench.run compact_bench [exit code non-0] (normal)
libraries/ghc-compact/tests/compact_share.run compact_share [exit code non-0] (sanity)
libraries/ghc-compact/tests/compact_bench.run compact_bench [exit code non-0] (sanity)
libraries/process/tests/T3231.run T3231 [exit code non-0] (threaded1)
libraries/process/tests/T3994.run T3994 [exit code non-0] (threaded1)
libraries/stm/tests/stm056.run stm056 [exit code non-0] (threaded1)
libraries/unix/tests/forkprocess01.run forkprocess01 [exit code non-0] (threaded1_ls)
libraries/unix/tests/fdReadBuf001.run fdReadBuf001 [bad exit code] (ghci)
testsuite/tests/rts/T7815.run T7815 [exit code non-0] (threaded1)
Unexpected stat failures:
testsuite/tests/callarity/perf/T3924.run T3924 [stat not good enough] (normal)
testsuite/tests/perf/join_points/join004.run join004 [stat not good enough] (optasm)
testsuite/tests/perf/should_run/T10359.run T10359 [stat not good enough] (normal)
testsuite/tests/perf/join_points/join007.run join007 [stat not good enough] (optasm)
testsuite/tests/perf/should_run/T14955.run T14955 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T4830.run T4830 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T3586.run T3586 [stat not good enough] (normal)
testsuite/tests/perf/should_run/lazy-bs-alloc.run lazy-bs-alloc [stat not good enough] (normal)
testsuite/tests/perf/should_run/T876.run T876 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T3738.run T3738 [stat not good enough] (normal)
testsuite/tests/perf/should_run/MethSharing.run MethSharing [stat not good enough] (normal)
testsuite/tests/perf/should_run/T5113.run T5113 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T4978.run T4978 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T5205.run T5205 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T4474a.run T4474a [stat not good enough] (normal)
testsuite/tests/perf/should_run/T4474b.run T4474b [stat not good enough] (normal)
testsuite/tests/perf/should_run/T5237.run T5237 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T4474c.run T4474c [stat not good enough] (normal)
testsuite/tests/perf/should_run/T7257.run T7257 [stat not good enough] (normal)
testsuite/tests/perf/should_run/Conversions.run Conversions [stat not good enough] (normal)
testsuite/tests/perf/should_run/T5536.run T5536 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T7797.run T7797 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T5949.run T5949 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T4267.run T4267 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T7619.run T7619 [stat not good enough] (normal)
testsuite/tests/perf/should_run/InlineByteArrayAlloc.run InlineByteArrayAlloc [stat not good enough] (normal)
testsuite/tests/perf/should_run/InlineArrayAlloc.run InlineArrayAlloc [stat not good enough] (normal)
testsuite/tests/perf/should_run/InlineCloneArrayAlloc.run InlineCloneArrayAlloc [stat not good enough] (normal)
testsuite/tests/perf/should_run/T9203.run T9203 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T8472.run T8472 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T12996.run T12996 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T13001.run T13001 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T12990.run T12990 [stat not good enough] (normal)
testsuite/tests/perf/should_run/DeriveNull.run DeriveNull [stat not good enough] (normal)
testsuite/tests/perf/should_run/T14936.run T14936 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T13623.run T13623 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T15226.run T15226 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T15226a.run T15226a [stat not good enough] (normal)
testsuite/tests/perf/space_leaks/space_leak_001.run space_leak_001 [stat not good enough] (normal)
testsuite/tests/perf/space_leaks/T4334.run T4334 [stat not good enough] (normal)
testsuite/tests/perf/should_run/T5549.run T5549 [stat not good enough] (normal)
libraries/base/tests/T8766.run T8766 [stat not good enough] (normal)
libraries/base/tests/T13191.run T13191 [stat not good enough] (normal)
Framework failures:
testsuite/tests/perf/haddock/haddock.base.run haddock.base [normal] (extra_file does not exist: ../../../../libraries/base/dist-install/haddock.t)
testsuite/tests/perf/haddock/haddock.Cabal.run haddock.Cabal [normal] (extra_file does not exist: ../../../../libraries/Cabal/Cabal/dist-install/haddock.t)
testsuite/tests/perf/haddock/haddock.compiler.run haddock.compiler [normal] (extra_file does not exist: ../../../../compiler/stage2/haddock.t)
Note: We're still skipping many many tests for some reason. For reference, a ./validate run from today, on GHC's master branch gives the following numbers:
SUMMARY for test run started at Tue Sep 4 15:34:20 2018 UTC
0:17:11 spent to go through
6537 total tests, which gave rise to
25537 test cases, of which
18644 were skipped
40 had missing libraries
6765 expected passes
89 expected failures
0 caused framework failures
0 caused framework warnings
0 unexpected passes
0 unexpected failures
0 unexpected stat failures
from hadrian.
alpmestan
commented on June 11, 2024
My differential has been merged: ghc/ghc@2254912
Meanwhile, I looked at #15570 and #15286 and it turns out removing those -O0 apparently makes more tests pass! In particular, all the stats failures but one are gone (makes sense), along with a little over 10 unexpected failures. I've got a PR up at #674. An entire test rule run also takes a lot less time now :-)
New test results
SUMMARY for test run started at Tue Sep 4 22:07:26 2018 CEST
0:18:56 spent to go through
6536 total tests, which gave rise to
24042 test cases, of which
17523 were skipped
27 had missing libraries
6225 expected passes
96 expected failures
3 caused framework failures
0 caused framework warnings
1 unexpected passes
169 unexpected failures
1 unexpected stat failures
Many, many of the remaining failures are due to ghci printing things differently (e.g 0 vs integer-...:Some.Module.S# 0) and therefore giving semantically equivalent but syntactically different outputs. I have seen #13243 which is about the same problem when running tests directly with make instead of through validate, but there isn't any conclusion about why this happens, even less about how to fix it. Somehow validate doesn't have this problem though. I guess I'll be the one figuring this out. But enough for today (and I will likely have to work on a few other things too during the rest of the week).
from hadrian.
snowleopard
commented on June 11, 2024
Great work, @alpmestan!
I'll try to figure out why ghci prints different results and report back if I find anything.
from hadrian.
alpmestan
commented on June 11, 2024
With #684, I get:
SUMMARY for test run started at Tue Sep 11 17:29:41 2018 CEST
0:16:40 spent to go through
6538 total tests, which gave rise to
24053 test cases, of which
17532 were skipped
27 had missing libraries
6315 expected passes
94 expected failures
3 caused framework failures
0 caused framework warnings
1 unexpected passes
83 unexpected failures
1 unexpected stat failures
This is looking better and better.
from hadrian.
snowleopard
commented on June 11, 2024
@alpmestan Awesome!
What about 17532 were skipped -- is this normal?
from hadrian.
alpmestan
commented on June 11, 2024
See my comment above:
Note: We're still skipping many many tests for some reason. For reference, a
./validaterun from today, on GHC's master branch gives the following numbers:SUMMARY for test run started at Tue Sep 4 15:34:20 2018 UTC 0:17:11 spent to go through 6537 total tests, which gave rise to 25537 test cases, of which 18644 were skipped 40 had missing libraries 6765 expected passes 89 expected failures 0 caused framework failures 0 caused framework warnings 0 unexpected passes 0 unexpected failures 0 unexpected stat failures
You can see that the normal validate script skips many tests too. Many tests are restricted to be executed only on some arch, some os, some way, some [something] that'll cause it to be skipped in many situations. We're still missing some tests and probably skipping too many of them, but we're somewhat close to what the make build system does, aren't we?
from hadrian.
simonmar
commented on June 11, 2024
Hi @alpmestan, this might not be the right place but I wanted to ask about workflows for the testsuite. Very often I do something like
cd testsuite/tests/codeGen/should_run
make TEST=cgrun001 WAY=optasm CLEANUP=0 EXTRA_HC_OPTS=-debug
to run a single test with some custom options. This is really convenient (and fast). Will we have something equally convenient with Hadrian, and will it be just as fast?
from hadrian.
alpmestan
commented on June 11, 2024
@simonmar We don't do anything $PWD-dependent for now. We do have --only=... to specify a specific test, --test-way to specify a particular test way, but I don't think we have anything to specify extra options right now, even though I suspect you can achieve this using --config.
(for reference:
Usage: hadrian [options] [target] ...
Options:
-a FULL=SHORT, --abbrev=FULL=SHORT
Use abbreviation in status messages.
--no-build Don't build anything.
-C DIRECTORY, --directory=DIRECTORY
Change to DIRECTORY before doing anything.
--color, --colour Colorize the output.
--no-color, --no-colour Don't colorize the output.
-d[FILE], --debug[=FILE] Print lots of debugging information.
--demo Run in demo mode.
--digest Files change when digest changes.
--digest-and Files change when modtime and digest change.
--digest-and-input Files change on modtime (and digest for inputs).
--digest-or Files change when modtime or digest change.
--digest-not Files change when modtime changes.
--exception Throw exceptions directly.
--flush=N Flush metadata every N seconds.
--never-flush Never explicitly flush metadata.
-h, --help Print this message and exit.
-j[N], --jobs[=N] Allow N jobs/threads at once [default CPUs].
-k, --keep-going Keep going when some targets can't be made.
-l, --lint Perform limited validation after the run.
--lint-fsatrace Use fsatrace to do validation.
--no-lint Turn off --lint.
--live[=FILE] List the files that are live [to live.txt].
-m PREFIX, --metadata=PREFIX
Prefix for storing metadata files.
--numeric-version Print just the version number and exit.
--skip-commands Try and avoid running external programs.
--rebuild[=PATTERN] Rebuild matching files.
--no-rebuild[=PATTERN] Rebuild matching files if necessary.
--skip[=PATTERN] Don't rebuild matching files this run.
-r[FILE], --report[=FILE], --profile[=FILE]
Write out profiling information [to report.html].
--no-reports Turn off --report.
--rule-version=VERSION Version of the build rules.
--no-rule-version Ignore the build rules version.
-s, --silent Don't print anything.
--sleep Sleep for a second before building.
-S, --no-keep-going, --stop
Turns off -k.
--storage Write a storage log.
-p[N], --progress[=N] Show progress messages [every N secs, default 5].
--no-progress Don't show progress messages.
-q, --quiet Print less (pass repeatedly for even less).
--no-time Don't print build time.
--timings Print phase timings.
-V, --verbose, --trace Print more (pass repeatedly for even more).
-v, --version Print the version number and exit.
-w, --print-directory Print the current directory.
--no-print-directory Turn off -w, even if it was turned on implicitly.
-c, --configure Run the boot and configure scripts (if you do not want to run them manually).
-o[BUILD_ROOT], --build-root[=BUILD_ROOT]
Where to store build artifacts. (Default _build).
--flavour[=FLAVOUR] Build flavour (Default, Devel1, Devel2, Perf, Prof, Quick or Quickest).
--freeze1 Freeze Stage1 GHC.
--integer-simple Build GHC with integer-simple library.
--progress-colour[=MODE] Use colours in progress info (Never, Auto or Always).
--progress-info[=STYLE] Progress info style (None, Brief, Normal or Unicorn).
--split-objects Generate split objects (requires a full clean rebuild).
--test-compiler[=TEST_COMPILER]
Use given compiler [Default=stage2].
--test-config-file[=CONFIG_FILE]
congiguration file for testsuite. Default=testsuite/config/ghc
--config[=EXTRA_TEST_CONFIG]
Configurations to run test, in key=value format.
--summary-junit[=TEST_SUMMARY_JUNIT]
Output testsuite summary in JUnit format.
--only[=TESTS] Test cases to run.
--only-perf Only run performance tests.
--skip-perf Skip performance tests.
--test-speed[=SPEED] fast, slow or normal. Normal by default
--summary[=TEST_SUMMARY] Where to output the test summary file.
--test-verbose[=TEST_VERBOSE]
A verbosity value between 0 and 5. 0 is silent, 4 and higher activates extra output.
--test-way[=TEST_WAY] only run these ways)
from hadrian.
simonmar
commented on June 11, 2024
Thanks @alpmestan. To add a bit more to this:
- Specifying a particular WAY is really important (not covered by those two issues I think)
- Turning off
CLEANUPis really important, so that you can run the binary that the testsuite created, or inspect the actual output that the test produced. Also not covered by those two issues. - I also often want to run a test with a different compiler, e.g.
make TEST=foo TEST_HC=ghc-8.6 - Figuring out tests from $PWD is handy but not essential. I'd be OK with having to say
TEST=codeGen/should_run/cgrun001, say, orTEST=codeGen/should_runto run all the tests in a subdir.
from hadrian.
alpmestan
commented on June 11, 2024
The PR at #687 fixes another batch of failures, in addition to solving a rather important problem. With this patch I get:
SUMMARY for test run started at Wed Sep 12 16:58:01 2018 CEST
0:27:00 spent to go through
6538 total tests, which gave rise to
24053 test cases, of which
17532 were skipped
27 had missing libraries
6341 expected passes
94 expected failures
3 caused framework failures
0 caused framework warnings
1 unexpected passes
57 unexpected failures
1 unexpected stat failures
Remaining failures:
Unexpected results from:
TEST="InternalCounters T10294 T10294a T10420 T10989 T12567a T12962 T13168 T13414 T13702 T14931 T2552 T2991 T3103 T5363 T5559 T680 T8209 T8242 T8430 T9203 T9208 T9293 TH_spliceE5_prof TH_spliceE5_prof_ext cabal01 cabal03 cabal04 conc069 conc069a conc070 fdReadBuf001 ghci057 haddock.Cabal haddock.base haddock.compiler memo001 memo002 plugin-recomp-change plugin-recomp-change-prof plugin-recomp-flags plugin-recomp-impure plugin-recomp-pure plugins01 plugins07 plugins09 plugins11 plugins12 plugins13 plugins14 plugins15 prof-doc-fib prof-doc-last profinline001 read041 scc001 scc002 scc003 scc005"
...
Unexpected passes:
testsuite/tests/stranal/should_compile/T9208.run T9208 [unexpected] (optasm)
Unexpected failures:
testsuite/tests/cabal/cabal01/cabal01.run cabal01 [bad exit code] (normal)
testsuite/tests/cabal/cabal03/cabal03.run cabal03 [bad exit code] (normal)
testsuite/tests/cabal/cabal04/cabal04.run cabal04 [bad exit code] (normal)
testsuite/tests/concurrent/should_run/conc069.run conc069 [bad exit code] (ghci)
testsuite/tests/concurrent/should_run/conc069a.run conc069a [bad exit code] (ghci)
testsuite/tests/concurrent/should_run/conc070.run conc070 [bad stdout] (ghci)
testsuite/tests/dynlibs/T13702.run T13702 [bad exit code] (normal)
testsuite/tests/ghci/scripts/ghci057.run ghci057 [bad stdout] (ghci)
testsuite/tests/ghci/scripts/T9293.run T9293 [bad stdout] (ghci)
testsuite/tests/ghci/scripts/T10989.run T10989 [bad stderr] (ghci)
testsuite/tests/hpc/T2991.run T2991 [exit code non-0] (hpc)
testsuite/tests/parser/should_compile/read041.run read041 [exit code non-0] (normal)
testsuite/tests/parser/should_fail/T8430.run T8430 [stderr mismatch] (normal)
testsuite/tests/parser/should_fail/T13414.run T13414 [stderr mismatch] (normal)
testsuite/tests/perf/haddock/haddock.base.run haddock.base [[Errno 2] No such file or directory: 'testsuite/tests/perf/haddock/haddock.base.run/haddock.t'] (normal)
testsuite/tests/perf/haddock/haddock.Cabal.run haddock.Cabal [[Errno 2] No such file or directory: 'testsuite/tests/perf/haddock/haddock.Cabal.run/haddock.t'] (normal)
testsuite/tests/perf/haddock/haddock.compiler.run haddock.compiler [[Errno 2] No such file or directory: 'testsuite/tests/perf/haddock/haddock.compiler.run/haddock.t'] (normal)
testsuite/tests/plugins/plugins01.run plugins01 [bad exit code] (normal)
testsuite/tests/plugins/plugins07.run plugins07 [bad exit code] (normal)
testsuite/tests/plugins/plugins09.run plugins09 [bad exit code] (normal)
testsuite/tests/plugins/plugins11.run plugins11 [bad exit code] (normal)
testsuite/tests/plugins/plugins12.run plugins12 [bad exit code] (normal)
testsuite/tests/plugins/plugins13.run plugins13 [bad exit code] (normal)
testsuite/tests/plugins/plugins14.run plugins14 [bad exit code] (normal)
testsuite/tests/plugins/plugins15.run plugins15 [bad exit code] (normal)
testsuite/tests/plugins/T10420.run T10420 [bad exit code] (normal)
testsuite/tests/plugins/T10294.run T10294 [bad exit code] (normal)
testsuite/tests/plugins/T10294a.run T10294a [bad exit code] (normal)
testsuite/tests/plugins/T12567a.run T12567a [bad exit code] (normal)
testsuite/tests/plugins/plugin-recomp-pure.run plugin-recomp-pure [bad exit code] (normal)
testsuite/tests/plugins/plugin-recomp-impure.run plugin-recomp-impure [bad exit code] (normal)
testsuite/tests/plugins/plugin-recomp-change.run plugin-recomp-change [bad exit code] (normal)
testsuite/tests/plugins/plugin-recomp-flags.run plugin-recomp-flags [bad exit code] (normal)
testsuite/tests/plugins/plugin-recomp-change-prof.run plugin-recomp-change-prof [bad exit code] (normal)
testsuite/tests/profiling/should_compile/T14931.run T14931 [bad exit code] (normal)
testsuite/tests/profiling/should_run/scc001.run scc001 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/scc002.run scc002 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/scc003.run scc003 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/scc005.run scc005 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T680.run T680 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T2552.run T2552 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/prof-doc-fib.run prof-doc-fib [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/prof-doc-last.run prof-doc-last [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T5559.run T5559 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/profinline001.run profinline001 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T5363.run T5363 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T12962.run T12962 [bad exit code] (ghci-ext-prof)
testsuite/tests/rename/should_compile/T3103/T3103.run T3103 [exit code non-0] (normal)
testsuite/tests/rts/T8209.run T8209 [bad stderr] (ghci)
testsuite/tests/rts/T8242.run T8242 [bad stderr] (ghci)
testsuite/tests/rts/InternalCounters.run InternalCounters [bad stdout] (normal)
testsuite/tests/th/TH_spliceE5_prof_ext.run TH_spliceE5_prof_ext [bad exit code] (normal)
testsuite/tests/th/TH_spliceE5_prof.run TH_spliceE5_prof [bad exit code] (normal)
testsuite/tests/typecheck/T13168/T13168.run T13168 [bad stderr] (normal)
libraries/base/tests/memo001.run memo001 [exit code non-0] (normal)
libraries/base/tests/memo002.run memo002 [exit code non-0] (normal)
libraries/unix/tests/fdReadBuf001.run fdReadBuf001 [bad exit code] (ghci)
Unexpected stat failures:
testsuite/tests/perf/should_run/T9203.run T9203 [stat too good] (normal)
Framework failures:
testsuite/tests/perf/haddock/haddock.base.run haddock.base [normal] (extra_file does not exist: ../../../../libraries/base/dist-install/haddock.t)
testsuite/tests/perf/haddock/haddock.Cabal.run haddock.Cabal [normal] (extra_file does not exist: ../../../../libraries/Cabal/Cabal/dist-install/haddock.t)
testsuite/tests/perf/haddock/haddock.compiler.run haddock.compiler [normal] (extra_file does not exist: ../../../../compiler/stage2/haddock.t)
from hadrian.
alpmestan
commented on June 11, 2024
Aaaaand we're down to 46 unexpected failures with #688:
Unexpected results from:
TEST="InternalCounters T10294 T10294a T10420 T12567a T12962 T13168 T13702 T14931 T2552 T5363 T5559 T680 T8209 T8242 T9203 T9208 T9293 TH_spliceE5_prof TH_spliceE5_prof_ext conc069 conc069a conc070 fdReadBuf001 ghci057 haddock.Cabal haddock.base haddock.compiler plugin-recomp-change plugin-recomp-change-prof plugin-recomp-flags plugin-recomp-impure plugin-recomp-pure plugins01 plugins07 plugins09 plugins11 plugins12 plugins13 plugins14 plugins15 prof-doc-fib prof-doc-last profinline001 scc001 scc002 scc003 scc005"
SUMMARY for test run started at Wed Sep 12 21:50:48 2018 CEST
0:20:42 spent to go through
6538 total tests, which gave rise to
24053 test cases, of which
17532 were skipped
27 had missing libraries
6352 expected passes
94 expected failures
3 caused framework failures
0 caused framework warnings
1 unexpected passes
46 unexpected failures
1 unexpected stat failures
Unexpected passes:
testsuite/tests/stranal/should_compile/T9208.run T9208 [unexpected] (optasm)
Unexpected failures:
testsuite/tests/concurrent/should_run/conc069.run conc069 [bad exit code] (ghci)
testsuite/tests/concurrent/should_run/conc069a.run conc069a [bad exit code] (ghci)
testsuite/tests/concurrent/should_run/conc070.run conc070 [bad stdout] (ghci)
testsuite/tests/dynlibs/T13702.run T13702 [bad exit code] (normal)
testsuite/tests/ghci/scripts/ghci057.run ghci057 [bad stdout] (ghci)
testsuite/tests/ghci/scripts/T9293.run T9293 [bad stdout] (ghci)
testsuite/tests/perf/haddock/haddock.base.run haddock.base [[Errno 2] No such file or directory: 'testsuite/tests/perf/haddock/haddock.base.run/haddock.t'] (normal)
testsuite/tests/perf/haddock/haddock.Cabal.run haddock.Cabal [[Errno 2] No such file or directory: 'testsuite/tests/perf/haddock/haddock.Cabal.run/haddock.t'] (normal)
testsuite/tests/perf/haddock/haddock.compiler.run haddock.compiler [[Errno 2] No such file or directory: 'testsuite/tests/perf/haddock/haddock.compiler.run/haddock.t'] (normal)
testsuite/tests/plugins/plugins01.run plugins01 [bad exit code] (normal)
testsuite/tests/plugins/plugins07.run plugins07 [bad exit code] (normal)
testsuite/tests/plugins/plugins09.run plugins09 [bad exit code] (normal)
testsuite/tests/plugins/plugins11.run plugins11 [bad exit code] (normal)
testsuite/tests/plugins/plugins12.run plugins12 [bad exit code] (normal)
testsuite/tests/plugins/plugins13.run plugins13 [bad exit code] (normal)
testsuite/tests/plugins/plugins14.run plugins14 [bad exit code] (normal)
testsuite/tests/plugins/plugins15.run plugins15 [bad exit code] (normal)
testsuite/tests/plugins/T10420.run T10420 [bad exit code] (normal)
testsuite/tests/plugins/T10294.run T10294 [bad exit code] (normal)
testsuite/tests/plugins/T10294a.run T10294a [bad exit code] (normal)
testsuite/tests/plugins/T12567a.run T12567a [bad exit code] (normal)
testsuite/tests/plugins/plugin-recomp-pure.run plugin-recomp-pure [bad exit code] (normal)
testsuite/tests/plugins/plugin-recomp-impure.run plugin-recomp-impure [bad exit code] (normal)
testsuite/tests/plugins/plugin-recomp-flags.run plugin-recomp-flags [bad exit code] (normal)
testsuite/tests/plugins/plugin-recomp-change.run plugin-recomp-change [bad exit code] (normal)
testsuite/tests/plugins/plugin-recomp-change-prof.run plugin-recomp-change-prof [bad exit code] (normal)
testsuite/tests/profiling/should_compile/T14931.run T14931 [bad exit code] (normal)
testsuite/tests/profiling/should_run/scc001.run scc001 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/scc002.run scc002 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/scc003.run scc003 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/scc005.run scc005 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T680.run T680 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T2552.run T2552 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/prof-doc-fib.run prof-doc-fib [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/prof-doc-last.run prof-doc-last [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T5559.run T5559 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/profinline001.run profinline001 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T5363.run T5363 [bad exit code] (ghci-ext-prof)
testsuite/tests/profiling/should_run/T12962.run T12962 [bad exit code] (ghci-ext-prof)
testsuite/tests/rts/T8209.run T8209 [bad stderr] (ghci)
testsuite/tests/rts/T8242.run T8242 [bad stderr] (ghci)
testsuite/tests/rts/InternalCounters.run InternalCounters [bad stdout] (normal)
testsuite/tests/th/TH_spliceE5_prof_ext.run TH_spliceE5_prof_ext [bad exit code] (normal)
testsuite/tests/th/TH_spliceE5_prof.run TH_spliceE5_prof [bad exit code] (normal)
testsuite/tests/typecheck/T13168/T13168.run T13168 [bad stderr] (normal)
libraries/unix/tests/fdReadBuf001.run fdReadBuf001 [bad exit code] (ghci)
Unexpected stat failures:
testsuite/tests/perf/should_run/T9203.run T9203 [stat too good] (normal)
Framework failures:
testsuite/tests/perf/haddock/haddock.base.run haddock.base [normal] (extra_file does not exist: ../../../../libraries/base/dist-install/haddock.t)
testsuite/tests/perf/haddock/haddock.Cabal.run haddock.Cabal [normal] (extra_file does not exist: ../../../../libraries/Cabal/Cabal/dist-install/haddock.t)
testsuite/tests/perf/haddock/haddock.compiler.run haddock.compiler [normal] (extra_file does not exist: ../../../../compiler/stage2/haddock.t)
from hadrian.
snowleopard
commented on June 11, 2024
Thanks @alpmestan -- looks like we're almost there!
from hadrian.
alpmestan
commented on June 11, 2024
With https://phabricator.haskell.org/D5255 I get:
Unexpected results from:
TEST="T11462 T11525 T13168 T13702 T3807 T5373 T9203 T9208 T9293 TH_spliceE5_prof annth_compunits annth_make dynHelloWorld ghci057 haddock.Cabal haddock.base haddock.compiler path_with_commas plugins05 plugins06"
SUMMARY for test run started at Wed Oct 24 13:53:33 2018 CEST
0:14:42 spent to go through
6601 total tests, which gave rise to
25634 test cases, of which
19054 were skipped
27 had missing libraries
6444 expected passes
89 expected failures
3 caused framework failures
0 caused framework warnings
1 unexpected passes
18 unexpected failures
1 unexpected stat failures
from hadrian.
alpmestan
commented on June 11, 2024
Test run against ghc/ghc@1a3b9bd:
Unexpected results from:
TEST="T11462 T11525 T13168 T13702 T3807 T5373 T9203 T9208 T9293 TH_spliceE5_prof annth_compunits annth_make dynHelloWorld frontend01 ghci057 haddock.Cabal haddock.base haddock.compiler path_with_commas plugins09 plugins11"
SUMMARY for test run started at Mon Nov 5 11:32:48 2018 CET
0:13:08 spent to go through
6656 total tests, which gave rise to
25841 test cases, of which
19176 were skipped
28 had missing libraries
6526 expected passes
90 expected failures
3 caused framework failures
0 caused framework warnings
1 unexpected passes
19 unexpected failures
1 unexpected stat failures
Unexpected passes:
testsuite/tests/stranal/should_compile/T9208.run T9208 [unexpected] (optasm)
Unexpected failures:
testsuite/tests/annotations/should_compile/th/annth_make.run annth_make [bad exit code] (normal)
testsuite/tests/annotations/should_compile/th/annth_compunits.run annth_compunits [bad stdout] (normal)
testsuite/tests/driver/dynHelloWorld.run dynHelloWorld [exit code non-0] (dyn)
testsuite/tests/dynlibs/T5373.run T5373 [bad exit code] (normal)
testsuite/tests/dynlibs/T3807.run T3807 [bad exit code] (normal)
testsuite/tests/dynlibs/T13702.run T13702 [bad exit code] (normal)
testsuite/tests/ghci/scripts/ghci057.run ghci057 [bad stdout] (ghci)
testsuite/tests/ghci/scripts/T9293.run T9293 [bad stdout] (ghci)
testsuite/tests/perf/haddock/haddock.base.run haddock.base [[Errno 2] No such file or directory: 'testsuite/tests/perf/haddock/haddock.base.run/haddock.t'] (normal)
testsuite/tests/perf/haddock/haddock.Cabal.run haddock.Cabal [[Errno 2] No such file or directory: 'testsuite/tests/perf/haddock/haddock.Cabal.run/haddock.t'] (normal)
testsuite/tests/perf/haddock/haddock.compiler.run haddock.compiler [[Errno 2] No such file or directory: 'testsuite/tests/perf/haddock/haddock.compiler.run/haddock.t'] (normal)
testsuite/tests/plugins/plugins09.run plugins09 [bad stdout] (normal)
testsuite/tests/plugins/plugins11.run plugins11 [bad stdout] (normal)
testsuite/tests/plugins/frontend01.run frontend01 [bad stdout] (normal)
testsuite/tests/th/TH_spliceE5_prof.run TH_spliceE5_prof [bad exit code] (normal)
testsuite/tests/th/TH_linker/path_with_commas.run path_with_commas [bad exit code] (normal)
testsuite/tests/typecheck/T13168/T13168.run T13168 [bad stderr] (normal)
testsuite/tests/typecheck/should_compile/T11462.run T11462 [exit code non-0] (normal)
testsuite/tests/typecheck/should_compile/T11525.run T11525 [exit code non-0] (normal)
Unexpected stat failures:
testsuite/tests/perf/should_run/T9203.run T9203 [stat too good] (normal)
Framework failures:
testsuite/tests/perf/haddock/haddock.base.run haddock.base [normal] (extra_file does not exist: ../../../../libraries/base/dist-install/haddock.t)
testsuite/tests/perf/haddock/haddock.Cabal.run haddock.Cabal [normal] (extra_file does not exist: ../../../../libraries/Cabal/Cabal/dist-install/haddock.t)
testsuite/tests/perf/haddock/haddock.compiler.run haddock.compiler [normal] (extra_file does not exist: ../../../../compiler/stage2/haddock.t)
from hadrian.
snowleopard
commented on June 11, 2024
I believe this issue is out of date, so closing.
from hadrian.
Related Issues (20)
- _build/stage0/bin/ghc.exe misses dependency HOT 14
- Rules for rts/rts.cabal are broken HOT 15
- Should Hadrian build ghctags and haddock in Stage1? HOT 3
- Documentation broken HOT 2
- Custom build root does not work on Windows HOT 2
- Symlink traversing logic in build scripts HOT 25
- Reduce the number of times a Cabal file is parsed HOT 13
- Improve documentation of predicates
- Build more variants of profiled runtime systems (normal, threaded, debug, threaded debug, ...) ? HOT 8
- Consider using Azure for CI HOT 1
- testsuite: support figuring out the list of tests from $PWD HOT 1
- testsuite: add CLI flag for specifying extra options to pass to ghc HOT 1
- rts: buggy handling of non-trivial dynamic flavours of the library HOT 14
- ctrl + c doesn't cancel build on windows HOT 5
- Building on windows is broken. HOT 23
- --integer-simple is broken HOT 40
- Hadrian has been merged into GHC proper HOT 4
- `happy` is not properly auto built in a fresh install HOT 11
- Issue on Trac: 15908 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hadrian.