snowleopard / hadrian Goto Github PK

View Code? Open in Web Editor NEW

208.0 208.0 39.0 3.13 MB

Hadrian: a new build system for the Glasgow Haskell Compiler. Now merged into the GHC tree!

Home Page: https://gitlab.haskell.org/ghc/ghc/tree/master/hadrian

License: MIT License

Haskell 97.54% Batchfile 0.44% Shell 1.11% Nix 0.91%

hadrian's People

Contributors

Stargazers

Watchers

hadrian's Issues

Parser.hs crashes due to broken #include in DynFlags.hs

/----------
| Running GhcM Stage0 with arguments:
|   -M
|   ... skipping 495 arguments ...
|   compiler/stage0/build/compiler/parser/Parser.hs
\----------


compiler/main/DynFlags.hs:127:10:
     fatal error: '../includes/dist-derivedconstants/header/GHCConstantsHaskellExports.hs' file not found
#include "../includes/dist-derivedconstants/header/GHCConstantsHaskellExports.hs"
         ^
1 error generated.
shakeArgsWith   0.000s    0%                           
Function shake  0.019s    1%                           
Database read   0.105s    7%  ==                       
With database   0.004s    0%                           
Running rules   1.274s   90%  =========================
Total           1.401s  100%                           
Error when running Shake build system:
* inplace/bin/ghc-stage1
* compiler/stage0/build/libHSghc-7.11.20151224.a
* compiler/stage0/build/Dwarf/Constants.o
* OracleQ (DependenciesKey ("compiler/stage0/build/.dependencies","compiler/stage0/build/Dwarf/Constants.o"))
* compiler/stage0/build/.dependencies
* compiler/stage0/build/.hs-dependencies
user error (Development.Shake.cmd, system command failed
Command: /usr/local/bin/ghc -M -hisuf hi -osuf o -hcsuf hc -static -hide-all-packages -no-user-package-db -package-db libraries/bootstrapping.conf -this-package-key ghc-7.11.20151224 -package-id array-0.5.1.0-d4206b835b96b5079d918fa1eab1a9a8 -package-id base-4.8.1.0-075aa0db10075facc5aaa59a7991ca2f -package-id binary-0.8.0.0 -package-id bytestring-0.10.6.0-6e8453cb70b477776f26900f41a5e17a -package-id containers-0.5.6.2-2b49cce16f8a2908df8454387e550b93 -package-id directory-1.2.2.0-16f6a661d4e92cd8da4d681a1d197064 -package-id filepath-1.4.0.0-8fee9c13b5e42926cc01f6aa7c403c4b -package-id ghc-boot-0.0.0.0 -package-id hoopl-3.10.2.1 -package-id hpc-0.6.0.2 -package-id process-1.2.3.0-36e5501145ab363f58c5e5a7079e9636 -package-id template-haskell-2.11.0.0 -package-id time-1.5.0.1-710377a9566ae0edafdde8dc74a184c3 -package-id transformers-0.5.0.0 -package-id unix-2.7.1.0-75051e1ddce506fe76a9ea932b926357 -i -icompiler/stage0/build -icompiler/stage0/build/autogen -Icompiler/stage0/build -Icompiler/stage0/build/autogen -icompiler/basicTypes -icompiler/cmm -icompiler/codeGen -icompiler/coreSyn -icompiler/deSugar -icompiler/ghci -icompiler/hsSyn -icompiler/iface -icompiler/llvmGen -icompiler/main -icompiler/nativeGen -icompiler/parser -icompiler/prelude -icompiler/profiling -icompiler/rename -icompiler/simplCore -icompiler/simplStg -icompiler/specialise -icompiler/stgSyn -icompiler/stranal -icompiler/typecheck -icompiler/types -icompiler/utils -icompiler/vectorise -Icompiler -Icompiler/parser -Icompiler/utils -Icompiler/stage0 -optP-include -optPcompiler/stage0/build/autogen/cabal_macros.h -Wall -fno-warn-name-shadowing -this-package-key ghc -XHaskell2010 -DSTAGE=1 -odir compiler/stage0/build -hidir compiler/stage0/build -stubdir compiler/stage0/build -rtsopts -include-pkg-deps -dep-makefile compiler/stage0/build/.hs-dependencies -dep-suffix  compiler/main/Annotations.hs compiler/parser/ApiAnnotation.hs compiler/nativeGen/AsmCodeGen.hs compiler/basicTypes/Avail.hs compiler/utils/Bag.hs compiler/basicTypes/BasicTypes.hs compiler/iface/BinIface.hs compiler/utils/Binary.hs compiler/cmm/Bitmap.hs compiler/cmm/BlockId.hs compiler/utils/BooleanFormula.hs compiler/main/BreakArray.hs compiler/utils/BufWrite.hs compiler/iface/BuildTyCl.hs compiler/cmm/CLabel.hs compiler/nativeGen/CPrim.hs compiler/simplCore/CSE.hs compiler/simplCore/CallArity.hs compiler/codeGen/CgUtils.hs compiler/deSugar/Check.hs compiler/types/Class.hs compiler/main/CmdLineParser.hs compiler/cmm/Cmm.hs compiler/cmm/CmmBuildInfoTables.hs compiler/cmm/CmmCallConv.hs compiler/cmm/CmmCommonBlockElim.hs compiler/cmm/CmmContFlowOpt.hs compiler/cmm/CmmExpr.hs compiler/cmm/CmmImplementSwitchPlans.hs compiler/cmm/CmmInfo.hs compiler/cmm/CmmLayoutStack.hs compiler/cmm/CmmLint.hs compiler/cmm/CmmLive.hs compiler/cmm/CmmMachOp.hs compiler/cmm/CmmNode.hs compiler/cmm/CmmOpt.hs compiler/cmm/CmmPipeline.hs compiler/cmm/CmmProcPoint.hs compiler/cmm/CmmSink.hs compiler/cmm/CmmSwitch.hs compiler/cmm/CmmType.hs compiler/cmm/CmmUtils.hs compiler/types/CoAxiom.hs compiler/codeGen/CodeGen/Platform.hs compiler/codeGen/CodeGen/Platform/ARM.hs compiler/codeGen/CodeGen/Platform/ARM64.hs compiler/codeGen/CodeGen/Platform/NoRegs.hs compiler/codeGen/CodeGen/Platform/PPC.hs compiler/codeGen/CodeGen/Platform/PPC_Darwin.hs compiler/codeGen/CodeGen/Platform/SPARC.hs compiler/codeGen/CodeGen/Platform/X86.hs compiler/codeGen/CodeGen/Platform/X86_64.hs compiler/main/CodeOutput.hs compiler/types/Coercion.hs compiler/basicTypes/ConLike.hs compiler/main/Constants.hs compiler/coreSyn/CoreArity.hs compiler/coreSyn/CoreFVs.hs compiler/coreSyn/CoreLint.hs compiler/simplCore/CoreMonad.hs compiler/coreSyn/CorePrep.hs compiler/coreSyn/CoreSeq.hs compiler/coreSyn/CoreStats.hs compiler/coreSyn/CoreSubst.hs compiler/coreSyn/CoreSyn.hs compiler/coreSyn/CoreTidy.hs compiler/stgSyn/CoreToStg.hs compiler/coreSyn/CoreUnfold.hs compiler/coreSyn/CoreUtils.hs compiler/profiling/CostCentre.hs compiler/deSugar/Coverage.hs compiler/parser/Ctype.hs compiler/basicTypes/DataCon.hs compiler/cmm/Debug.hs compiler/basicTypes/Demand.hs compiler/deSugar/Desugar.hs compiler/utils/Digraph.hs compiler/stranal/DmdAnal.hs compiler/main/DriverMkDepend.hs compiler/main/DriverPhases.hs compiler/main/DriverPipeline.hs compiler/deSugar/DsArrows.hs compiler/deSugar/DsBinds.hs compiler/deSugar/DsCCall.hs compiler/deSugar/DsExpr.hs compiler/deSugar/DsForeign.hs compiler/deSugar/DsGRHSs.hs compiler/deSugar/DsListComp.hs compiler/deSugar/DsMeta.hs compiler/deSugar/DsMonad.hs compiler/deSugar/DsUtils.hs compiler/nativeGen/Dwarf.hs compiler/nativeGen/Dwarf/Constants.hs compiler/nativeGen/Dwarf/Types.hs compiler/main/DynFlags.hs compiler/main/DynamicLoading.hs compiler/main/Elf.hs compiler/utils/Encoding.hs compiler/main/ErrUtils.hs compiler/utils/Exception.hs compiler/utils/FV.hs compiler/typecheck/FamInst.hs compiler/types/FamInstEnv.hs compiler/utils/FastFunctions.hs compiler/utils/FastMutInt.hs compiler/utils/FastString.hs compiler/utils/FastStringEnv.hs compiler/basicTypes/FieldLabel.hs compiler/main/Finder.hs compiler/utils/FiniteMap.hs compiler/iface/FlagChecker.hs compiler/simplCore/FloatIn.hs compiler/simplCore/FloatOut.hs compiler/prelude/ForeignCall.hs compiler/nativeGen/Format.hs compiler/typecheck/FunDeps.hs compiler/main/GHC.hs compiler/main/GhcMake.hs compiler/main/GhcMonad.hs compiler/main/GhcPlugins.hs compiler/utils/GraphBase.hs compiler/utils/GraphColor.hs compiler/utils/GraphOps.hs compiler/utils/GraphPpr.hs compiler/parser/HaddockUtils.hs compiler/main/HeaderInfo.hs compiler/main/Hooks.hs compiler/cmm/Hoopl.hs compiler/cmm/Hoopl/Dataflow.hs compiler/hsSyn/HsBinds.hs compiler/hsSyn/HsDecls.hs compiler/hsSyn/HsDoc.hs compiler/hsSyn/HsExpr.hs compiler/hsSyn/HsImpExp.hs compiler/hsSyn/HsLit.hs compiler/hsSyn/HsPat.hs compiler/hsSyn/HsSyn.hs compiler/hsSyn/HsTypes.hs compiler/hsSyn/HsUtils.hs compiler/main/HscMain.hs compiler/main/HscStats.hs compiler/main/HscTypes.hs compiler/utils/IOEnv.hs compiler/basicTypes/Id.hs compiler/basicTypes/IdInfo.hs compiler/iface/IfaceEnv.hs compiler/iface/IfaceSyn.hs compiler/iface/IfaceType.hs compiler/typecheck/Inst.hs compiler/types/InstEnv.hs compiler/nativeGen/Instruction.hs compiler/main/InteractiveEval.hs compiler/main/InteractiveEvalTypes.hs compiler/types/Kind.hs compiler/basicTypes/Lexeme.hs compiler/simplCore/LiberateCase.hs compiler/utils/ListSetOps.hs compiler/basicTypes/Literal.hs compiler/llvmGen/Llvm.hs compiler/llvmGen/Llvm/AbsSyn.hs compiler/llvmGen/Llvm/MetaData.hs compiler/llvmGen/Llvm/PpLlvm.hs compiler/llvmGen/Llvm/Types.hs compiler/llvmGen/LlvmCodeGen.hs compiler/llvmGen/LlvmCodeGen/Base.hs compiler/llvmGen/LlvmCodeGen/CodeGen.hs compiler/llvmGen/LlvmCodeGen/Data.hs compiler/llvmGen/LlvmCodeGen/Ppr.hs compiler/llvmGen/LlvmCodeGen/Regs.hs compiler/llvmGen/LlvmMangler.hs compiler/iface/LoadIface.hs compiler/deSugar/Match.hs compiler/deSugar/MatchCon.hs compiler/deSugar/MatchLit.hs compiler/utils/Maybes.hs compiler/coreSyn/MkCore.hs compiler/cmm/MkGraph.hs compiler/basicTypes/MkId.hs compiler/iface/MkIface.hs compiler/basicTypes/Module.hs compiler/utils/MonadUtils.hs compiler/nativeGen/NCGMonad.hs compiler/basicTypes/Name.hs compiler/basicTypes/NameEnv.hs compiler/basicTypes/NameSet.hs compiler/basicTypes/OccName.hs compiler/simplCore/OccurAnal.hs compiler/types/OptCoercion.hs compiler/utils/OrdList.hs compiler/utils/Outputable.hs compiler/nativeGen/PIC.hs compiler/nativeGen/PPC/CodeGen.hs compiler/nativeGen/PPC/Cond.hs compiler/nativeGen/PPC/Instr.hs compiler/nativeGen/PPC/Ppr.hs compiler/nativeGen/PPC/RegInfo.hs compiler/nativeGen/PPC/Regs.hs compiler/main/PackageConfig.hs compiler/main/Packages.hs compiler/utils/Pair.hs compiler/utils/Panic.hs compiler/basicTypes/PatSyn.hs compiler/main/PipelineMonad.hs compiler/hsSyn/PlaceHolder.hs compiler/utils/Platform.hs compiler/main/PlatformConstants.hs compiler/main/Plugins.hs compiler/deSugar/PmExpr.hs compiler/nativeGen/PprBase.hs compiler/cmm/PprC.hs compiler/cmm/PprCmm.hs compiler/cmm/PprCmmDecl.hs compiler/cmm/PprCmmExpr.hs compiler/coreSyn/PprCore.hs compiler/main/PprTyThing.hs compiler/prelude/PrelInfo.hs compiler/prelude/PrelNames.hs compiler/prelude/PrelRules.hs compiler/utils/Pretty.hs compiler/prelude/PrimOp.hs compiler/profiling/ProfInit.hs compiler/parser/RdrHsSyn.hs compiler/basicTypes/RdrName.hs compiler/nativeGen/Reg.hs compiler/nativeGen/RegAlloc/Graph/ArchBase.hs compiler/nativeGen/RegAlloc/Graph/ArchX86.hs compiler/nativeGen/RegAlloc/Graph/Coalesce.hs compiler/nativeGen/RegAlloc/Graph/Main.hs compiler/nativeGen/RegAlloc/Graph/Spill.hs compiler/nativeGen/RegAlloc/Graph/SpillClean.hs compiler/nativeGen/RegAlloc/Graph/SpillCost.hs compiler/nativeGen/RegAlloc/Graph/Stats.hs compiler/nativeGen/RegAlloc/Graph/TrivColorable.hs compiler/nativeGen/RegAlloc/Linear/Base.hs compiler/nativeGen/RegAlloc/Linear/FreeRegs.hs compiler/nativeGen/RegAlloc/Linear/JoinToTargets.hs compiler/nativeGen/RegAlloc/Linear/Main.hs compiler/nativeGen/RegAlloc/Linear/PPC/FreeRegs.hs compiler/nativeGen/RegAlloc/Linear/SPARC/FreeRegs.hs compiler/nativeGen/RegAlloc/Linear/StackMap.hs compiler/nativeGen/RegAlloc/Linear/State.hs compiler/nativeGen/RegAlloc/Linear/Stats.hs compiler/nativeGen/RegAlloc/Linear/X86/FreeRegs.hs compiler/nativeGen/RegAlloc/Linear/X86_64/FreeRegs.hs compiler/nativeGen/RegAlloc/Liveness.hs compiler/nativeGen/RegClass.hs compiler/rename/RnBinds.hs compiler/rename/RnEnv.hs compiler/rename/RnExpr.hs compiler/rename/RnHsDoc.hs compiler/rename/RnNames.hs compiler/rename/RnPat.hs compiler/rename/RnSource.hs compiler/rename/RnSplice.hs compiler/rename/RnTypes.hs compiler/specialise/Rules.hs compiler/simplCore/SAT.hs compiler/profiling/SCCfinal.hs compiler/cmm/SMRep.hs compiler/nativeGen/SPARC/AddrMode.hs compiler/nativeGen/SPARC/Base.hs compiler/nativeGen/SPARC/CodeGen.hs compiler/nativeGen/SPARC/CodeGen/Amode.hs compiler/nativeGen/SPARC/CodeGen/Base.hs compiler/nativeGen/SPARC/CodeGen/CondCode.hs compiler/nativeGen/SPARC/CodeGen/Expand.hs compiler/nativeGen/SPARC/CodeGen/Gen32.hs compiler/nativeGen/SPARC/CodeGen/Gen64.hs compiler/nativeGen/SPARC/CodeGen/Sanity.hs compiler/nativeGen/SPARC/Cond.hs compiler/nativeGen/SPARC/Imm.hs compiler/nativeGen/SPARC/Instr.hs compiler/nativeGen/SPARC/Ppr.hs compiler/nativeGen/SPARC/Regs.hs compiler/nativeGen/SPARC/ShortcutJump.hs compiler/nativeGen/SPARC/Stack.hs compiler/simplCore/SetLevels.hs compiler/simplCore/SimplCore.hs compiler/simplCore/SimplEnv.hs compiler/simplCore/SimplMonad.hs compiler/simplStg/SimplStg.hs compiler/simplCore/SimplUtils.hs compiler/simplCore/Simplify.hs compiler/specialise/SpecConstr.hs compiler/specialise/Specialise.hs compiler/basicTypes/SrcLoc.hs compiler/utils/State.hs compiler/main/StaticFlags.hs compiler/deSugar/StaticPtrTable.hs compiler/codeGen/StgCmm.hs compiler/codeGen/StgCmmArgRep.hs compiler/codeGen/StgCmmBind.hs compiler/codeGen/StgCmmClosure.hs compiler/codeGen/StgCmmCon.hs compiler/codeGen/StgCmmEnv.hs compiler/codeGen/StgCmmExpr.hs compiler/codeGen/StgCmmExtCode.hs compiler/codeGen/StgCmmForeign.hs compiler/codeGen/StgCmmHeap.hs compiler/codeGen/StgCmmHpc.hs compiler/codeGen/StgCmmLayout.hs compiler/codeGen/StgCmmMonad.hs compiler/codeGen/StgCmmPrim.hs compiler/codeGen/StgCmmProf.hs compiler/codeGen/StgCmmTicky.hs compiler/codeGen/StgCmmUtils.hs compiler/stgSyn/StgLint.hs compiler/simplStg/StgStats.hs compiler/stgSyn/StgSyn.hs compiler/utils/Stream.hs compiler/utils/StringBuffer.hs compiler/main/SysTools.hs compiler/prelude/THNames.hs compiler/nativeGen/TargetReg.hs compiler/typecheck/TcAnnotations.hs compiler/typecheck/TcArrows.hs compiler/typecheck/TcBinds.hs compiler/typecheck/TcCanonical.hs compiler/typecheck/TcClassDcl.hs compiler/typecheck/TcDefaults.hs compiler/typecheck/TcDeriv.hs compiler/typecheck/TcEnv.hs compiler/typecheck/TcErrors.hs compiler/typecheck/TcEvidence.hs compiler/typecheck/TcExpr.hs compiler/typecheck/TcFlatten.hs compiler/typecheck/TcForeign.hs compiler/typecheck/TcGenDeriv.hs compiler/typecheck/TcGenGenerics.hs compiler/typecheck/TcHsSyn.hs compiler/typecheck/TcHsType.hs compiler/iface/TcIface.hs compiler/typecheck/TcInstDcls.hs compiler/typecheck/TcInteract.hs compiler/typecheck/TcMType.hs compiler/typecheck/TcMatches.hs compiler/typecheck/TcPat.hs compiler/typecheck/TcPatSyn.hs compiler/typecheck/TcPluginM.hs compiler/typecheck/TcRnDriver.hs compiler/typecheck/TcRnMonad.hs compiler/typecheck/TcRnTypes.hs compiler/typecheck/TcRules.hs compiler/typecheck/TcSMonad.hs compiler/typecheck/TcSimplify.hs compiler/typecheck/TcSplice.hs compiler/typecheck/TcTyClsDecls.hs compiler/typecheck/TcTyDecls.hs compiler/typecheck/TcType.hs compiler/typecheck/TcTypeNats.hs compiler/typecheck/TcTypeable.hs compiler/typecheck/TcUnify.hs compiler/typecheck/TcValidity.hs compiler/main/TidyPgm.hs compiler/deSugar/TmOracle.hs compiler/coreSyn/TrieMap.hs compiler/types/TyCoRep.hs compiler/types/TyCon.hs compiler/types/Type.hs compiler/prelude/TysPrim.hs compiler/prelude/TysWiredIn.hs compiler/utils/UnVarGraph.hs compiler/simplStg/UnariseStg.hs compiler/types/Unify.hs compiler/utils/UniqDFM.hs compiler/utils/UniqDSet.hs compiler/utils/UniqFM.hs compiler/utils/UniqSet.hs compiler/basicTypes/UniqSupply.hs compiler/basicTypes/Unique.hs compiler/utils/Util.hs compiler/basicTypes/Var.hs compiler/basicTypes/VarEnv.hs compiler/basicTypes/VarSet.hs compiler/vectorise/Vectorise.hs compiler/vectorise/Vectorise/Builtins.hs compiler/vectorise/Vectorise/Builtins/Base.hs compiler/vectorise/Vectorise/Builtins/Initialise.hs compiler/vectorise/Vectorise/Convert.hs compiler/vectorise/Vectorise/Env.hs compiler/vectorise/Vectorise/Exp.hs compiler/vectorise/Vectorise/Generic/Description.hs compiler/vectorise/Vectorise/Generic/PADict.hs compiler/vectorise/Vectorise/Generic/PAMethods.hs compiler/vectorise/Vectorise/Generic/PData.hs compiler/vectorise/Vectorise/Monad.hs compiler/vectorise/Vectorise/Monad/Base.hs compiler/vectorise/Vectorise/Monad/Global.hs compiler/vectorise/Vectorise/Monad/InstEnv.hs compiler/vectorise/Vectorise/Monad/Local.hs compiler/vectorise/Vectorise/Monad/Naming.hs compiler/vectorise/Vectorise/Type/Classify.hs compiler/vectorise/Vectorise/Type/Env.hs compiler/vectorise/Vectorise/Type/TyConDecl.hs compiler/vectorise/Vectorise/Type/Type.hs compiler/vectorise/Vectorise/Utils.hs compiler/vectorise/Vectorise/Utils/Base.hs compiler/vectorise/Vectorise/Utils/Closure.hs compiler/vectorise/Vectorise/Utils/Hoisting.hs compiler/vectorise/Vectorise/Utils/PADict.hs compiler/vectorise/Vectorise/Utils/Poly.hs compiler/vectorise/Vectorise/Var.hs compiler/vectorise/Vectorise/Vect.hs compiler/stranal/WorkWrap.hs compiler/stranal/WwLib.hs compiler/nativeGen/X86/CodeGen.hs compiler/nativeGen/X86/Cond.hs compiler/nativeGen/X86/Instr.hs compiler/nativeGen/X86/Ppr.hs compiler/nativeGen/X86/RegInfo.hs compiler/nativeGen/X86/Regs.hs compiler/stage0/build/Config.hs compiler/stage0/build/compiler/cmm/CmmLex.hs compiler/stage0/build/compiler/cmm/CmmParse.hs compiler/stage0/build/compiler/utils/Fingerprint.hs compiler/stage0/build/compiler/parser/Lexer.hs compiler/stage0/build/compiler/parser/Parser.hs
Exit code: 1
Stderr:
compiler/main/DynFlags.hs:127:10:
     fatal error: '../includes/dist-derivedconstants/header/GHCConstantsHaskellExports.hs' file not found
#include "../includes/dist-derivedconstants/header/GHCConstantsHaskellExports.hs"
         ^
1 error generated.
)

Rework `Package` to reflect structure of Cabal packages

The Package type currently doesn't really reflect the actual structure of Cabal packages. In particular PackageType forces a package to be either a Library or a Program whereas Cabal packages may be both. This works now but may become problematic in the future. Moreover, this requires that one explicitly define the type of each package in the build system.

One alternative would involve teaching ghc-cabal to emit more details about the components offered by the package, allowing us to read this information from package-data.mk.

Fix dynamic way

Dynamic way is currently broken (another relevant issue: #3).

Most infrastructure is already there, but one needs to identify the missing bits, find the relevant parts in the the old build system, and translate them into the new build system.

Feel free to pick this up if you'd like to contribute.

Use Gitter?

Should we add a semi-realtime medium? I know that begriffs/postgrest uses Gitter. (https://gitter.im)

hsc2hs broken

I'm seeing this,

/----------
| Running Hsc2Hs with arguments:
|   ... skipping 40 arguments ...
|   libraries/hpc/Trace/Hpc/Reflect.hsc
|   -o
|   libraries/hpc/stage0/build/libraries/hpc/Trace/Hpc/Reflect.hs
\----------

No template specified, and template-hsc.h not located.

Usage: hsc2hs [OPTIONS] INPUT.hsc [...]

  -o FILE          --output=FILE          name of main output file
  -t FILE          --template=FILE        template file
  -c PROG          --cc=PROG              C compiler to use
  -l PROG          --ld=PROG              linker to use
  -C FLAG          --cflag=FLAG           flag to pass to the C compiler
  -I DIR                                  passed to the C compiler
  -L FLAG          --lflag=FLAG           flag to pass to the linker
  -i FILE          --include=FILE         as if placed in the source
  -D NAME[=VALUE]  --define=NAME[=VALUE]  as if placed in the source
                   --no-compile           stop after writing *_hsc_make.c
  -x               --cross-compile        activate cross-compilation mode
                   --cross-safe           restrict .hsc directives to those supported by --cross-compile
  -k               --keep-files           do not remove temporary files
  -v               --verbose              dump commands to stderr
  -?               --help                 display this help and exit
  -V               --version              output version information and exit
shakeArgsWith   0.000s    0%                           
Function shake  0.032s    0%                           
Database read   0.052s    0%                           
With database   0.003s    0%                           
Running rules  18.051s   99%  =========================
Total          18.138s  100%                           
Error when running Shake build system:
* inplace/bin/ghc-stage1
* libraries/hpc/stage0/build/libHShpc-0.6.0.2.a
* libraries/hpc/stage0/build/Trace/Hpc/Reflect.o
* OracleQ (DependenciesKey ("libraries/hpc/stage0/build/.dependencies","libraries/hpc/stage0/build/Trace/Hpc/Reflect.o"))
* libraries/hpc/stage0/build/.dependencies
* libraries/hpc/stage0/build/.hs-dependencies
* libraries/hpc/stage0/build/libraries/hpc/Trace/Hpc/Reflect.hs
user error (Development.Shake.cmd, system command failed
Command: inplace/bin/hsc2hs --cc=/usr/bin/gcc --ld=/usr/bin/gcc --cross-safe --cflag=-fno-stack-protector --cflag=-Ilibraries/hpc/stage0/build/autogen --cflag=-I/opt/exp/ghc/roots/7.10.3/lib/ghc-7.10.2.20151118/direc_0hFG6ZxK1nk4zsyOqbNHfm/include --cflag=-I/opt/exp/ghc/roots/7.10.3/lib/ghc-7.10.2.20151118/unix_KZL8h98IqDM57kQSPo1mKx/include --cflag=-I/opt/exp/ghc/roots/7.10.3/lib/ghc-7.10.2.20151118/time_FTheb6LSxyX1UABIbBXRfn/include --cflag=-I/opt/exp/ghc/roots/7.10.3/lib/ghc-7.10.2.20151118/bytes_6VWy06pWzJq9evDvK2d4w6/include --cflag=-I/opt/exp/ghc/roots/7.10.3/lib/ghc-7.10.2.20151118/base_HQfYBxpPvuw8OunzQu6JGM/include --cflag=-I/opt/exp/ghc/roots/7.10.3/lib/ghc-7.10.2.20151118/integ_2aU3IZNMF9a7mQ0OzsZ0dS/include --cflag=-I/opt/exp/ghc/roots/7.10.3/lib/ghc-7.10.2.20151118/include --cflag=-Wall --cflag=-Werror=unused-but-set-variable --cflag=-Wno-error=inline --cflag=-include --cflag=libraries/hpc/stage0/build/autogen/cabal_macros.h --lflag=-L/opt/exp/ghc/roots/7.10.3/lib/ghc-7.10.2.20151118/direc_0hFG6ZxK1nk4zsyOqbNHfm --lflag=-L/opt/exp/ghc/roots/7.10.3/lib/ghc-7.10.2.20151118/unix_KZL8h98IqDM57kQSPo1mKx --lflag=-L/opt/exp/ghc/roots/7.10.3/lib/ghc-7.10.2.20151118/time_FTheb6LSxyX1UABIbBXRfn --lflag=-L/opt/exp/ghc/roots/7.10.3/lib/ghc-7.10.2.20151118/filep_Ey7a1in9roBAE8bUFJ5R9m --lflag=-L/opt/exp/ghc/roots/7.10.3/lib/ghc-7.10.2.20151118/conta_2C3ZI8RgPO2LBMidXKTvIU --lflag=-L/opt/exp/ghc/roots/7.10.3/lib/ghc-7.10.2.20151118/bytes_6VWy06pWzJq9evDvK2d4w6 --lflag=-L/opt/exp/ghc/roots/7.10.3/lib/ghc-7.10.2.20151118/deeps_6vMKxt5sPFR0XsbRWvvq59 --lflag=-L/opt/exp/ghc/roots/7.10.3/lib/ghc-7.10.2.20151118/array_67iodizgJQIIxYVTp4emlA --lflag=-L/opt/exp/ghc/roots/7.10.3/lib/ghc-7.10.2.20151118/base_HQfYBxpPvuw8OunzQu6JGM --lflag=-L/opt/exp/ghc/roots/7.10.3/lib/ghc-7.10.2.20151118/integ_2aU3IZNMF9a7mQ0OzsZ0dS --lflag=-L/opt/exp/ghc/roots/7.10.3/lib/ghc-7.10.2.20151118/ghcpr_8TmvWUcS1U1IKHT0levwg3 --lflag=-L/opt/exp/ghc/roots/7.10.3/lib/ghc-7.10.2.20151118/rts --lflag=-lrt --lflag=-lutil --lflag=-ldl --lflag=-lpthread --lflag=-lgmp --lflag=-lm --lflag=-lrt --lflag=-ldl --cflag=-Dx86_64_HOST_ARCH=1 --cflag=-Dlinux_HOST_OS=1 --cflag=-D__GLASGOW_HASKELL__=710 libraries/hpc/Trace/Hpc/Reflect.hsc -o libraries/hpc/stage0/build/libraries/hpc/Trace/Hpc/Reflect.hs
Exit code: 1
Stderr:
No template specified, and template-hsc.h not located.

Usage: hsc2hs [OPTIONS] INPUT.hsc [...]

  -o FILE          --output=FILE          name of main output file
  -t FILE          --template=FILE        template file
  -c PROG          --cc=PROG              C compiler to use
  -l PROG          --ld=PROG              linker to use
  -C FLAG          --cflag=FLAG           flag to pass to the C compiler
  -I DIR                                  passed to the C compiler
  -L FLAG          --lflag=FLAG           flag to pass to the linker
  -i FILE          --include=FILE         as if placed in the source
  -D NAME[=VALUE]  --define=NAME[=VALUE]  as if placed in the source
                   --no-compile           stop after writing *_hsc_make.c
  -x               --cross-compile        activate cross-compilation mode
                   --cross-safe           restrict .hsc directives to those supported by --cross-compile
  -k               --keep-files           do not remove temporary files
  -v               --verbose              dump commands to stderr
  -?               --help                 display this help and exit
  -V               --version              output version information and exit
)

I'm not sure where the default template-hsc.h is supposed to be in the case of the Shake build system, but in the case of the old build system it was in inplace/lib/template-hsc.h IIRC, which I can confirm exists in my tree.

Parallel compilation appears to be broken

Running with -jN where N > 1 results in failures of the form,

No dependencies found for 'libraries/process/dist-install/build/libHSprocess-1.4.1.0.ghc7.11.20151216.so'.

on Linux.

Build rts

We can't build rts package yet.

It is a C-only package (similarly to utils/hp2ps) and the best approach seems to create a new package type for handling such packages without hacks like generated package-data.mk file (see Rules/Data.hs).

I'll give it a try.

inplace/bin/hp2ps needs libraries/bootstrapping.conf

/----------
| Running Ghc Stage0 with arguments:
|   ... skipping 44 arguments ...
|   utils/hp2ps/stage0/build/Shade.o
|   utils/hp2ps/stage0/build/Utilities.o
|   -o
|   inplace/bin/hp2ps
\----------

ghc: can't find a package database at libraries/bootstrapping.conf
shakeArgsWith   0.000s    0%                           
Function shake  0.018s    1%                           
Database read   0.006s    0%                           
With database   0.000s    0%                           
Running rules   1.229s   98%  =========================
Total           1.253s  100%                           
Error when running Shake build system:
* inplace/bin/hp2ps
user error (Development.Shake.cmd, system command failed
Command: /usr/local/bin/ghc -hisuf hi -osuf o -hcsuf hc -static -no-user-package-db -package-db libraries/bootstrapping.conf -i -iutils/hp2ps/stage0/build -iutils/hp2ps/stage0/build/autogen -Iutils/hp2ps/stage0/build -Iutils/hp2ps/stage0/build/autogen -odir utils/hp2ps/stage0/build -hidir utils/hp2ps/stage0/build -stubdir utils/hp2ps/stage0/build -rtsopts -H32m -O -Wall -fwarn-tabs -no-hs-main -no-auto-link-packages -optl-lm utils/hp2ps/stage0/build/AreaBelow.o utils/hp2ps/stage0/build/Curves.o utils/hp2ps/stage0/build/Error.o utils/hp2ps/stage0/build/Main.o utils/hp2ps/stage0/build/Reorder.o utils/hp2ps/stage0/build/TopTwenty.o utils/hp2ps/stage0/build/AuxFile.o utils/hp2ps/stage0/build/Deviation.o utils/hp2ps/stage0/build/HpFile.o utils/hp2ps/stage0/build/Marks.o utils/hp2ps/stage0/build/Scale.o utils/hp2ps/stage0/build/TraceElement.o utils/hp2ps/stage0/build/Axes.o utils/hp2ps/stage0/build/Dimensions.o utils/hp2ps/stage0/build/Key.o utils/hp2ps/stage0/build/PsFile.o utils/hp2ps/stage0/build/Shade.o utils/hp2ps/stage0/build/Utilities.o -o inplace/bin/hp2ps
Exit code: 1
Stderr:
ghc: can't find a package database at libraries/bootstrapping.conf
)

This is generated as ghc-pkg init libraries/bootstrapping.conf (cf. ghc.mk:757)

On inplace/lib/bin vs. inplace/bin

make puts utilities in inplace/lib/bin, while the current shake build system puts them in inplace/bin, which I think is the saner solution, found this while trying to fix #35.
This is also relevant to #44, in that we might place template-hsc.h into inplace as well?

Split objects and a lurking bug

I came across a bug, which is difficult to trigger and which manifests itself in a very mysterious way. It costed me two hours of debugging to catch.

When ghc is compiling a file.hs it produces the object file.o and ~~after that~~ multiple smaller objects placed in directory file_o_split (when executed with flag -split-objs). If ghc is terminated during the creation of split objects, we may get only some split objects but not all of them. Since we don't know how many of them there should be, we can't check that the result is correct. Missing split objects later cause link errors elsewhere, which are difficult to understand because some split objects are there (in my case there were 192 split objects instead of 324 for source file GHC/Show.hs in base).

I'm not yet sure why ghc was terminated prematurely in that particular case. Perhaps, it crashed with a segfault (sometimes happens with -j8) or the build system killed it after it encountered a problem elsewhere. @ndmitchell, is the latter possible?

Let's discuss possible solutions here.

Two ideas so far:

~~Change ghc so it first produces split objects and only after that generates the main object file. Then Shake will automatically take care of the rest.~~
Find out how many split objects should be generated use this to check the correctness of the result.

Add clang support

build.sh for some reason fails to find clang.

Linking shake-build/.shake/build ...
Reading shake-build/cfg/system.config...
Reading package dependencies...
| Successfully generated 'utils/hp2ps/stage0/package-data.mk'.
Reading utils/hp2ps/stage0/package-data.mk...
Reading dependencies from utils/hp2ps/stage0/build/.dependencies...
Error when running Shake build system:
* inplace/bin/hp2ps
* utils/hp2ps/stage0/build/Utilities.o
* OracleQ (DependenciesKey ("utils/hp2ps/stage0/build/.dependencies","utils/hp2ps/stage0/build/Utilities.o"))
* utils/hp2ps/stage0/build/.dependencies
* utils/hp2ps/stage0/build/Utilities.c.deps
* clang
Error, file does not exist and no rule available:
  clang

Fix stage2

At the moment the build system doesn't handle stage2 properly. No rules for stage2 are generated, in particular, haddock and ghctags are built in stage1 instead of stage2.

This is related to #14.

bootstrapping.conf/package.cache: GHC.PackageDb.readPackageDb: inappropriate type (Not a valid Unicode code point!)

The changes in 84704c change the package path from include/lib/package.conf.d to libraries/bootstrapping.conf for Stage0. This leads to the Stage1 compiler using libraries/bootstrapping.conf and failing.

The core of the issue is that the wrapper is generated at Stage0, but used at Stage1.

# chmod (for inplace/bin/ghc-stage1)
| Successfully created wrapper for 'ghc-bin' (Stage0).
/----------
| Running GhcPkg Stage1 (package = Cabal):
|     input: none
| => output: inplace/lib/package.conf.d
\----------

Successfully initialised inplace/lib/package.conf.d
/----------
| Running GhcCabal (stage = Stage1, package = ghc-prim):
|     input: libraries/ghc-prim/ghc-prim.cabal
| => output: libraries/ghc-prim/stage1/package-data.mk
\----------

Configuring ghc-prim-0.5.0.0...
ghc-cabal: ghc-pkg dump failed
shakeArgsWith    0.001s    0%                           
Function shake   0.025s    0%                           
Database read    0.103s    0%                           
With database    0.007s    0%                           
Running rules  149.295s   99%  =========================
Total          149.431s  100%                           
Error when running Shake build system:
* iserv/stage1/build/tmp/iserv-bin
* OracleQ (PackageDataKey ("iserv/stage1/package-data.mk","iserv_stage1_C_SRCS"))
* iserv/stage1/package-data.mk
* libraries/ghci/stage1/package-data.mk
* libraries/transformers/stage1/package-data.mk
* libraries/ghc-prim/stage1/package-data.mk
user error (Development.Shake.cmd, system command failed
Command: inplace/bin/ghc-cabal configure libraries/ghc-prim stage1  --with-ghc=/Users/angerman/Projects/Haskell/ghc/inplace/bin/ghc-stage1 --with-ghc-pkg=/Users/angerman/Projects/Haskell/ghc/inplace/bin/ghc-pkg --enable-library-vanilla --enable-library-for-ghci --disable-library-profiling --disable-shared --with-hscolour=/Users/angerman/.cabal/bin/HsColour --configure-option=CFLAGS=-m64 -fno-stack-protector --configure-option=LDFLAGS=-m64 --configure-option=CPPFLAGS=-Iincludes -Iincludes/dist-derivedconstants/header -m64 --gcc-options=-m64 -fno-stack-protector -m64 --configure-option=--with-cc=/usr/bin/clang --with-gcc=/usr/bin/clang --with-ld=/usr/bin/ld --with-ar=/usr/bin/ar --with-alex=/Users/angerman/.cabal/bin/alex --with-happy=/Users/angerman/.cabal/bin/happy --flag=include-ghc-prim
Exit code: 1
Stderr:
ghc-cabal: ghc-pkg dump failed
)

Add "On OS X" section to the README.md

Currently sections of Linux and Windows are present.
A third one for OS X (maybe completely identical to linux),
as well as one for ghc with the llvm backend on OS X would be a good addition.

I will document my findings and extend the README.md, once I know what
the steps involved are.

Fix performance regression

Zero build jumped from 6 seconds to 65 on my machine after I fixed these two issues: #15 and #19. Not sure what the reason is yet, but I suspect findModuleFiles in Settings.hs.

Prerequisites and cabal sandbox

The addition of the cabal sandbox section has dropped the prerequisite sections, was this on purpose or an oversight?

@quchen should the cabal sandbox build, install alex and shake as well?

Can not build Config.hs

Currently stuck at:

Reading dependencies from libraries/transformers/stage0/build/.dependencies...
Reading dependencies from libraries/template-haskell/stage0/build/.dependencies...
Reading dependencies from libraries/hpc/stage0/build/.dependencies...
Reading dependencies from libraries/hoopl/stage0/build/.dependencies...
Reading dependencies from compiler/stage0/build/.dependencies...
Exactly one generator expected for compiler/stage0/build/Config.hs (found: []).
shakeArgsWith   0.000s    0%                           
Function shake  0.018s    3%  =                        
Database read   0.082s   16%  =====                    
With database   0.003s    0%                           
Running rules   0.402s   79%  =========================
Total           0.505s  100%                           
Error when running Shake build system:
* inplace/bin/ghc-stage1
* compiler/stage0/build/libHSghc-7.11.20151224.a
* compiler/stage0/build/Dwarf/Constants.o
* OracleQ (DependenciesKey ("compiler/stage0/build/.dependencies","compiler/stage0/build/Dwarf/Constants.o"))
* compiler/stage0/build/.dependencies
* compiler/stage0/build/.hs-dependencies
* compiler/stage0/build/Config.hs
GHC build system error: Exactly one generator expected for compiler/stage0/build/Config.hs (found: []).

Improve haddock coverage of the project

I'm lost from time to time in the source, trying to figure out how the parts belong together. I guess adding more extensive documentation to the functions would help with this in a big way.

How to correctly reset?

What are the expected commands to run to end up with a completely clean state, to start from? Or should one reclone the repositories every time?

Running the following, seems like a pretty excessive way to arrive at the initial state.

$ git clone git://git.haskell.org/ghc
$ cd ghc
$ git submodule update --init
$ git clone git://github.com/snowleopard/shaking-up-ghc shake-build

Need some way to verbosely show commands executed

Currently there is no way to convince the build system to dump the full command-lines that it is executing. This is an extremely useful thing to be able to do when things go awry.

Simplify import conventions

Currently the codebase has many modules re-exporting other modules. Not only does this make it difficult for the developer to know the "proper" way to import a given definition, but it makes import cycles almost inevitable due to the more dense module dependency graph.

inplace/lib/settings: openFile: does not

With #62, I now end up at:

.----------
| Running Ghc Stage0 (package = ghc-bin):
|     input: ghc/stage0/build/hschooks.o (and 1 more)
| => output: inplace/bin/lib/bin/ghc-stage1
'----------

Warning: -rtsopts and -with-rtsopts have no effect with -no-hs-main.
    Call hs_init_ghc() from your main() function to set these options.
.----------
| Successfully built program 'ghc-bin' (Stage0).
| Executable: inplace/bin/lib/bin/ghc-stage1
| Package synopsis: XXX.
'----------

# chmod (for inplace/bin/ghc-stage1)
| Successfully created wrapper 'ghc-bin' (Stage0).
.----------
| Running GhcCabal (stage = Stage1, package = ghc-prim):
|     input: libraries/ghc-prim/ghc-prim.cabal
| => output: libraries/ghc-prim/stage1/package-data.mk
'----------

Configuring ghc-prim-0.5.0.0...
ghc-cabal: '/Users/angerman/Projects/Haskell/ghc/inplace/bin/ghc-stage1'
exited with an error:
/Users/angerman/Projects/Haskell/ghc/inplace/lib/settings: openFile: does not
exist (No such file or directory)
shakeArgsWith         0.001s    0%                           
Function shake        0.026s    0%                           
Database read         0.224s    0%                           
Database compression  0.065s    0%                           
With database         0.007s    0%                           
Running rules       162.531s   99%  =========================
Total               162.855s  100%                           
Error when running Shake build system:
* iserv/stage1/build/tmp/iserv-bin
* OracleQ (PackageDataKey ("iserv/stage1/package-data.mk","iserv_stage1_C_SRCS"))
* iserv/stage1/package-data.mk
* libraries/ghci/stage1/package-data.mk
* libraries/transformers/stage1/package-data.mk
* libraries/ghc-prim/stage1/package-data.mk
user error (Development.Shake.cmd, system command failed
Command: inplace/bin/ghc-cabal configure libraries/ghc-prim stage1  --with-ghc=/Users/angerman/Projects/Haskell/ghc/inplace/bin/ghc-stage1 --with-ghc-pkg=/Users/angerman/Projects/Haskell/ghc/inplace/bin/ghc-pkg --enable-library-vanilla --enable-library-for-ghci --disable-library-profiling --disable-shared --with-hscolour=/Users/angerman/.cabal/bin/HsColour --configure-option=CFLAGS=-m64 -fno-stack-protector --configure-option=LDFLAGS=-m64 --configure-option=CPPFLAGS=-Iincludes -Iincludes/dist-derivedconstants/header -m64 --gcc-options=-m64 -fno-stack-protector -m64 --configure-option=--with-cc=/usr/bin/clang --with-gcc=/usr/bin/clang --with-ld=/usr/bin/ld --with-ar=/usr/bin/ar --with-alex=/Users/angerman/.cabal/bin/alex --with-happy=/Users/angerman/.cabal/bin/happy --flag=include-ghc-prim
Exit code: 1
Stderr:
ghc-cabal: '/Users/angerman/Projects/Haskell/ghc/inplace/bin/ghc-stage1'
exited with an error:
/Users/angerman/Projects/Haskell/ghc/inplace/lib/settings: openFile: does not
exist (No such file or directory)
)

inplace/lib/template-hsc.h not created

This is just a copy of utils/hsc2hs/template-hsc.h. See #35

When a program is rebuilt all dependent rules are rerun, even if the program's binary is unchanged

When a program, such as ghc-pkg is rebuilt, we have to rerun all rules that depend on it, even if the program's resulting binary is not changed.

We need to implement something like Shake's writeFileChanged in order to suppress unnecessary rebuilds. Pragmatically, we can rely on a simple heuristic to check binaries for equality, e.g. comparing file sizes. Incidentally, is this the purpose of utils/compareSizes?

Building ghc-pkg: can't find stage0-built libs

When running the build system, I get the error pasted below. It seems the libraries built by the stage-0 compiler cannot be found. Any pointers would be welcome!

Reproducible on Xubuntu 14.04 and 15.04, although I doubt the OS matters. Stage-0 is GHC 7.10.3.

/usr/bin/ld: cannot find -lHSterminfo-0.4.0.2
/usr/bin/ld: cannot find -lHSghc-boot-0.0.0.0
/usr/bin/ld: cannot find -lHSCabal-1.23.0.0
/usr/bin/ld: cannot find -lHSbinary-0.8.0.0
collect2: error: ld returned 1 exit status
Error when running Shake build system:
* inplace/bin/ghc-pkg
user error (Development.Shake.cmd, system command failed
Command: /home/david/bin/haskell/ghc
    -hisuf hi
    -osuf o
    -hcsuf hc
    -static
    -hide-all-packages
    -no-user-package-db
    -package-db libraries/bootstrapping.conf
    -package-id Cabal-1.23.0.0
    -package-id base-4.8.2.0-0d6d1084fbc041e1cded9228e80e264d
    -package-id binary-0.8.0.0
    -package-id bytestring-0.10.6.0-c60f4c543b22c7f7293a06ae48820437
    -package-id containers-0.5.6.2-e59c9b78d840fa743d4169d4bea15592
    -package-id directory-1.2.2.0-f8e14a9d121b76a00a0f669ee724a732
    -package-id filepath-1.4.0.0-f97d1e4aebfd7a03be6980454fe31d6e
    -package-id ghc-boot-0.0.0.0
    -package-id process-1.2.3.0-78f206acb2330ea8066c6c19c87356f0
    -package-id terminfo-0.4.0.2
    -package-id unix-2.7.1.0-343d4f566f30113da92e819f4a148640
    -i
    -iutils/ghc-pkg/stage0/build
    -iutils/ghc-pkg/stage0/build/autogen
    -Iutils/ghc-pkg/stage0/build
    -Iutils/ghc-pkg/stage0/build/autogen
    -iutils/ghc-pkg
    -optP-include
    -optPutils/ghc-pkg/stage0/build/autogen/cabal_macros.h
    -XHaskell2010
    -odir utils/ghc-pkg/stage0/build
    -hidir utils/ghc-pkg/stage0/build
    -stubdir utils/ghc-pkg/stage0/build
    -rtsopts
    -H32m
    -O
    -Wall
    -fwarn-tabs
    -no-auto-link-packages
    -optl-ltinfo
    -optl-lrt
    -optl-lutil
    -optl-ldl
    -optl-lpthread
    -optl-lgmp
    -optl-lm
    -optl-lrt
    -optl-ldl
    -optl-L/home/david/Coding/haskell/ghc/libraries/terminfo/stage0/build
    -optl-L/home/david/Coding/haskell/ghc/libraries/ghc-boot/stage0/build
    -optl-L/home/david/Coding/haskell/ghc/libraries/Cabal/Cabal/stage0/build
    -optl-L/home/david/Programs/haskell/ghc-7.10.3/lib/ghc-7.10.3/proce_52AgREEfSrnJLlkGV9YZZJ
    -optl-L/home/david/Programs/haskell/ghc-7.10.3/lib/ghc-7.10.3/prett_JItwetRppk1H5Uq3xbjDGC
    -optl-L/home/david/Programs/haskell/ghc-7.10.3/lib/ghc-7.10.3/direc_0hFG6ZxK1nk4zsyOqbNHfm
    -optl-L/home/david/Programs/haskell/ghc-7.10.3/lib/ghc-7.10.3/unix_KZL8h98IqDM57kQSPo1mKx
    -optl-L/home/david/Programs/haskell/ghc-7.10.3/lib/ghc-7.10.3/time_FTheb6LSxyX1UABIbBXRfn
    -optl-L/home/david/Programs/haskell/ghc-7.10.3/lib/ghc-7.10.3/filep_Ey7a1in9roBAE8bUFJ5R9m
    -optl-L/home/david/Coding/haskell/ghc/libraries/binary/stage0/build
    -optl-L/home/david/Programs/haskell/ghc-7.10.3/lib/ghc-7.10.3/conta_2C3ZI8RgPO2LBMidXKTvIU
    -optl-L/home/david/Programs/haskell/ghc-7.10.3/lib/ghc-7.10.3/bytes_6VWy06pWzJq9evDvK2d4w6
    -optl-L/home/david/Programs/haskell/ghc-7.10.3/lib/ghc-7.10.3/deeps_6vMKxt5sPFR0XsbRWvvq59
    -optl-L/home/david/Programs/haskell/ghc-7.10.3/lib/ghc-7.10.3/array_67iodizgJQIIxYVTp4emlA
    -optl-L/home/david/Programs/haskell/ghc-7.10.3/lib/ghc-7.10.3/base_HQfYBxpPvuw8OunzQu6JGM
    -optl-L/home/david/Programs/haskell/ghc-7.10.3/lib/ghc-7.10.3/integ_2aU3IZNMF9a7mQ0OzsZ0dS
    -optl-L/home/david/Programs/haskell/ghc-7.10.3/lib/ghc-7.10.3/ghcpr_8TmvWUcS1U1IKHT0levwg3
    -optl-L/home/david/Programs/haskell/ghc-7.10.3/lib/ghc-7.10.3/rts utils/ghc-pkg/stage0/build/Main.o utils/ghc-pkg/stage0/build/Version.o
    -o inplace/bin/ghc-pkg
Exit code: 1
Stderr:
/usr/bin/ld: cannot find -lHSterminfo-0.4.0.2
/usr/bin/ld: cannot find -lHSghc-boot-0.0.0.0
/usr/bin/ld: cannot find -lHSCabal-1.23.0.0
/usr/bin/ld: cannot find -lHSbinary-0.8.0.0
collect2: error: ld returned 1 exit status
)

Bad hsc2hs call

I've been trying to build GHC HEAD with ghc-shake, but I keep running into this error:

No template specified, and template-hsc.h not located.

It seems HSC is called with bad parameter; I've pasted the full error on lpaste.

ghc-cabal: ghc-pkg dump failed

Making baby steps, we are not at ghc-pkg failing. I assume this is due to ghc-pkg being wrapped as well. I'll try to fix the ghc-pkg wrapper.

# chmod (for inplace/bin/ghc-stage1)
| Successfully created wrapper for 'ghc-bin' (Stage0).
.----------
| Running GhcCabal (stage = Stage1, package = ghc-prim):
|     input: libraries/ghc-prim/ghc-prim.cabal
| => output: libraries/ghc-prim/stage1/package-data.mk
'----------

Configuring ghc-prim-0.5.0.0...
ghc-cabal: ghc-pkg dump failed
shakeArgsWith    0.002s    0%                           
Function shake   0.031s    0%                           
Database read    0.149s    0%                           
With database    0.040s    0%                           
Running rules  153.518s   99%  =========================
Total          153.740s  100%                           
Error when running Shake build system:
* iserv/stage1/build/tmp/iserv-bin
* OracleQ (PackageDataKey ("iserv/stage1/package-data.mk","iserv_stage1_C_SRCS"))
* iserv/stage1/package-data.mk
* libraries/ghci/stage1/package-data.mk
* libraries/transformers/stage1/package-data.mk
* libraries/ghc-prim/stage1/package-data.mk
user error (Development.Shake.cmd, system command failed
Command: inplace/bin/ghc-cabal configure libraries/ghc-prim stage1  --with-ghc=/Users/angerman/Projects/Haskell/ghc/inplace/bin/ghc-stage1 --with-ghc-pkg=/Users/angerman/Projects/Haskell/ghc/inplace/bin/ghc-pkg --enable-library-vanilla --enable-library-for-ghci --disable-library-profiling --disable-shared --with-hscolour=/Users/angerman/.cabal/bin/HsColour --configure-option=CFLAGS=-m64 -fno-stack-protector --configure-option=LDFLAGS=-m64 --configure-option=CPPFLAGS=-Iincludes -Iincludes/dist-derivedconstants/header -m64 --gcc-options=-m64 -fno-stack-protector -m64 --configure-option=--with-cc=/usr/bin/clang --with-gcc=/usr/bin/clang --with-ld=/usr/bin/ld --with-ar=/usr/bin/ar --with-alex=/Users/angerman/.cabal/bin/alex --with-happy=/Users/angerman/.cabal/bin/happy --flag=include-ghc-prim
Exit code: 1
Stderr:
ghc-cabal: ghc-pkg dump failed
)

Bootstrapping clean tree fails with message complaining about missing haddock

Currently fails with,

$ make distclean
$ ./boot
$ ./configure
$ shake-build/build.sh -j4
Reading shake-build/cfg/system.config...
Error when running Shake build system:
* inplace/bin/haddock
Error, file does not exist and no rule available:
  inplace/bin/haddock

Perhaps this is fallout from fixing #5?

Implement build rules for `includes`

See #16.

The following files need to be generated:

~~includes/ghcautoconf.h~~ Done: 6b7b9cc.
~~includes/ghcplatform.h~~ Done: 8c32f2c.
~~includes/ghcversion.h~~ Done: 66f18be.
~~includes/dist-derivedconstants/header/DerivedConstants.h~~ Done: c6cfb36.
~~includes/dist-derivedconstants/header/GHCConstantsHaskellType.hs~~ Done: c6cfb36.
~~includes/dist-derivedconstants/header/platformConstants~~ Done: c6cfb36.
~~includes/dist-derivedconstants/header/GHCConstantsHaskellWrappers.hs~~ Done: c6cfb36.
~~includes/dist-derivedconstants/header/GHCConstantsHaskellExports.hs~~ Done: c6cfb36.

Please add other includes files that need attention to the above list.

Decompose Args into into builder-specific and package-specific

Currently getArgs is constructed by putting together builder-specific Args from the Settings/Builders directory. A simple example from Settings/Builders/Alex.hs:

alexArgs :: Args
alexArgs = builder Alex ? mconcat [ arg "-g"
                                  , package compiler ? arg "--latin1"
                                  , arg =<< getInput
                                  , arg "-o", arg =<< getOutput ]

We want to decompose package-specific Args from builder-specific ones to make it easier to understand and maintain the build system. Under this proposal the above will change as follows.

Settings/Builders/Alex.hs will now contain only default arguments for Alex:

alexArgs :: Args
alexArgs = builder Alex ? mconcat [ arg "-g"
                                  , arg =<< getInput
                                  , arg "-o", arg =<< getOutput ]

And Settings/Packages/Compiler.hs will contain arguments specific to compiler package:

compilerArgs :: Args
compilerArgs = package compiler ? mconcat [ ...
                                          , builder Alex ? arg "--latin1"
                                          , ... ]

Package-specific arguments will override builder-specific ones.

I'd like to finish this before tackling #56 and #22. Please shout if you have any concerns about this.

Cleanup doc folder

The doc folder contains a seemingly (to me only?) random collection of files, that are a bit hard to get the hang of without a README.md in that folder.

No support for `wrapper`

We do not handle wrapper scripts at all so far. This I presume is the reason for #35 and maybe #27 as well?

How is this wrapping handled on Windows?

iserv-bin configuration fails

Configuring iserv-bin-0.0...
/--------
| Running GhcPkg 1 with arguments:
|   update
|   --force
|   iserv/dist-install/inplace-pkg-config
\--------
Reading package info from "iserv/dist-install/inplace-pkg-config" ... ghc-pkg: iserv/dist-install/inplace-pkg-config: openBinaryFile: does not exist (No such file or directory)
Error when running Shake build system:
* iserv/dist-install/doc/html/iserv-bin/iserv-bin.haddock
* OracleQ (PackageDataKey ("iserv/dist-install/package-data.mk","iserv_dist-install_HS_SRC_DIRS"))
* iserv/dist-install/package-data.mk
* iserv/dist-install/package-data.mk iserv/dist-install/haddock-prologue.txt iserv/dist-install/setup-config iserv/dist-install/build/autogen/cabal_macros.h
user error (Development.Shake.cmd, system command failed
Command: /opt/exp/ghc/ghc/inplace/bin/ghc-pkg update --force iserv/dist-install/inplace-pkg-config
Exit code: 1
Stderr:
ghc-pkg: iserv/dist-install/inplace-pkg-config: openBinaryFile: does not exist (No such file or directory)
)

Still trying to work out why.

ghc-stage1: missing -B<dir> option

The current state get's us past the stage0 compiler. Now we are stuck at

/----------
| Running GhcCabal with arguments:
|   configure
|   libraries/ghc-prim
|   stage1
|   ... skipping 19 arguments ...
\----------

Configuring ghc-prim-0.5.0.0...
ghc-cabal: '/Users/angerman/Projects/Haskell/ghc/inplace/bin/ghc-stage1'
exited with an error:
ghc-stage1: missing -B<dir> option
shakeArgsWith    0.000s    0%                           
Function shake   0.018s    0%                           
Database read    0.005s    0%                           
With database    0.000s    0%                           
Running rules  831.894s   99%  =========================
Total          831.918s  100%                           
Error when running Shake build system:
* iserv/stage1/build/tmp/iserv-bin
* OracleQ (PackageDataKey ("iserv/stage1/package-data.mk","iserv_stage1_C_SRCS"))
* iserv/stage1/package-data.mk
* libraries/ghci/stage1/package-data.mk
* libraries/transformers/stage1/package-data.mk
* libraries/ghc-prim/stage1/package-data.mk
user error (Development.Shake.cmd, system command failed
Command: inplace/bin/ghc-cabal configure libraries/ghc-prim stage1  --with-ghc=/Users/angerman/Projects/Haskell/ghc/inplace/bin/ghc-stage1 --with-ghc-pkg=/Users/angerman/Projects/Haskell/ghc/inplace/bin/ghc-pkg --enable-library-vanilla --enable-library-for-ghci --disable-library-profiling --disable-shared --with-hscolour=/Users/angerman/.cabal/bin/HsColour --configure-option=CFLAGS=-m64 -fno-stack-protector --configure-option=LDFLAGS=-m64 --configure-option=CPPFLAGS=-Iincludes -Iincludes/dist -Iincludes/dist-derivedconstants/header -Iincludes/dist-ghcconstants/header -m64 --gcc-options=-m64 -fno-stack-protector -m64 --configure-option=--with-cc=/usr/bin/clang --with-gcc=/usr/bin/clang --with-ld=/usr/bin/ld --with-ar=/usr/bin/ar --with-alex=/Users/angerman/.cabal/bin/alex --with-happy=/Users/angerman/.cabal/bin/happy --flag=include-ghc-prim
Exit code: 1
Stderr:
ghc-cabal: '/Users/angerman/Projects/Haskell/ghc/inplace/bin/ghc-stage1'
exited with an error:
ghc-stage1: missing -B<dir> option
)

I think this is related to #38. As the ghc.wrapper contains an implicit -B flag.

Here are all the wrappers I found in the source tree

./ghc/ghc.wrapper

exec "$executablename" -B"$topdir" ${1+"$@"}

./utils/ghc-pkg/ghc-pkg.wrapper

#!/bin/sh

PKGCONF="$topdir/package.conf.d"
exec "$executablename" --global-package-db "$PKGCONF" ${1+"$@"}

./utils/haddock/haddock.wrapper

#!/bin/sh

exec "$executablename" -B"$topdir" -l"$topdir" ${1+"$@"}

./utils/hpc/hpc.wrapper

#!/bin/sh
exec "$executablename" ${1+"$@"}

./utils/hsc2hs/hsc2hs.wrapper

#!/bin/sh

tflag="--template=$topdir/template-hsc.h"
Iflag="-I$topdir/include/"
for arg do
    case "$arg" in
# On OS X, we need to specify -m32 or -m64 in order to get gcc to
# build binaries for the right target. We do that by putting it in
# HSC2HS_EXTRA. When cabal runs hsc2hs, it passes a flag saying which
# gcc to use, so if we set HSC2HS_EXTRA= then we don't get binaries
# for the right platform. So for now we just don't set HSC2HS_EXTRA=
# but we probably want to revisit how this works in the future.
#        -c*)          HSC2HS_EXTRA=;;
#        --cc=*)       HSC2HS_EXTRA=;;
        -t*)          tflag=;;
        --template=*) tflag=;;
        --)           break;;
    esac
done

exec "$executablename" ${tflag:+"$tflag"} $HSC2HS_EXTRA ${1+"$@"} "$Iflag"

./utils/runghc/runghc.wrapper

#!/bin/sh

exec "$executablename" -f "$bindir/$ghcprog" ${1+"$@"}

ghc-cabal: At least the following dependencies are missing: rts ==1.0.*

Alright, we are getting there. After fixing #66 manually, we arrive at the missing rts. Now we are at #22, blocking any further progress.

# chmod (for inplace/bin/ghc-stage1)
| Successfully created wrapper for 'ghc-bin' (Stage0).
/----------
| Running GhcCabal (stage = Stage1, package = ghc-prim):
|     input: libraries/ghc-prim/ghc-prim.cabal
| => output: libraries/ghc-prim/stage1/package-data.mk
\----------

Configuring ghc-prim-0.5.0.0...
ghc-cabal: At least the following dependencies are missing:
rts ==1.0.*
shakeArgsWith    0.000s    0%                           
Function shake   0.018s    0%                           
Database read    0.141s    0%                           
With database    0.007s    0%                           
Running rules  151.691s   99%  =========================
Total          151.857s  100%                           
Error when running Shake build system:
* iserv/stage1/build/tmp/iserv-bin
* OracleQ (PackageDataKey ("iserv/stage1/package-data.mk","iserv_stage1_C_SRCS"))
* iserv/stage1/package-data.mk
* libraries/ghci/stage1/package-data.mk
* libraries/transformers/stage1/package-data.mk
* libraries/ghc-prim/stage1/package-data.mk
user error (Development.Shake.cmd, system command failed
Command: inplace/bin/ghc-cabal configure libraries/ghc-prim stage1  --with-ghc=/Users/angerman/Projects/Haskell/ghc/inplace/bin/ghc-stage1 --with-ghc-pkg=/Users/angerman/Projects/Haskell/ghc/inplace/bin/ghc-pkg --enable-library-vanilla --enable-library-for-ghci --disable-library-profiling --disable-shared --with-hscolour=/Users/angerman/.cabal/bin/HsColour --configure-option=CFLAGS=-m64 -fno-stack-protector --configure-option=LDFLAGS=-m64 --configure-option=CPPFLAGS=-Iincludes -Iincludes/dist-derivedconstants/header -m64 --gcc-options=-m64 -fno-stack-protector -m64 --configure-option=--with-cc=/usr/bin/clang --with-gcc=/usr/bin/clang --with-ld=/usr/bin/ld --with-ar=/usr/bin/ar --with-alex=/Users/angerman/.cabal/bin/alex --with-happy=/Users/angerman/.cabal/bin/happy --flag=include-ghc-prim
Exit code: 1
Stderr:
ghc-cabal: At least the following dependencies are missing:
rts ==1.0.*
)

integer-gmp: fatal error: 'gmp.h' file not found

Given the rts (#22, #67) work around and the unix package #70 fix #69, we end up with the following:

/----------
| Running GhcM Stage1 (package = base):
|     input: libraries/base/Control/Applicative.hs (and 215 more)
| => output: libraries/base/stage1/build/.hs-dependencies
\----------

Reading dependencies from libraries/integer-gmp/stage1/build/.dependencies...
/----------
| Running GccM Stage1 (package = integer-gmp):
|     input: libraries/integer-gmp/cbits/wrappers.c
| => output: libraries/integer-gmp/stage1/build/cbits/wrappers.c.deps
\----------

libraries/integer-gmp/cbits/wrappers.c:24:10: fatal error: 'gmp.h' file not found
#include <gmp.h>
         ^
1 error generated.
shakeArgsWith   0.001s    0%                           
Function shake  0.026s    0%                           
Database read   0.277s    2%                           
With database   0.006s    0%                           
Running rules   9.136s   96%  =========================
Total           9.448s  100%                           
Error when running Shake build system:
* utils/compare_sizes/stage1/build/tmp/compareSizes
* libraries/unix/stage1/build/HSunix-2.7.1.1.o
* libraries/unix/stage1/build/System/Posix/Terminal/Common.o
* libraries/base/stage1/build/Data/Bits.hi
* libraries/base/stage1/build/Data/Bits.o
* libraries/base/stage1/build/GHC/Enum.hi
* libraries/base/stage1/build/GHC/Enum.o
* libraries/integer-gmp/stage1/build/GHC/Integer.hi
* libraries/integer-gmp/stage1/build/GHC/Integer.o
* OracleQ (DependenciesKey ("libraries/integer-gmp/stage1/build/.dependencies","libraries/integer-gmp/stage1/build/GHC/Integer.o"))
* libraries/integer-gmp/stage1/build/.dependencies
* libraries/integer-gmp/stage1/build/cbits/wrappers.c.deps
user error (Development.Shake.cmd, system command failed
Command: /usr/bin/clang -E -MM -std=c99 -Wall -Ilibraries/integer-gmp/stage1/build -Ilibraries/integer-gmp/stage1/build/autogen -Ilibraries/integer-gmp/include -I/Users/angerman/Projects/Haskell/ghc/rts/dist/build -I/Users/angerman/Projects/Haskell/ghc/includes -I/Users/angerman/Projects/Haskell/ghc/includes/dist-derivedconstants/header -MF libraries/integer-gmp/stage1/build/cbits/wrappers.c.deps -MT libraries/integer-gmp/stage1/build/cbits/wrappers.o -x c libraries/integer-gmp/cbits/wrappers.c
Exit code: 1
Stderr:
libraries/integer-gmp/cbits/wrappers.c:24:10: fatal error: 'gmp.h' file not found
#include <gmp.h>
         ^
1 error generated.
)

Add CI

Once we reach a half-decent state of stability, we should add continuous integration. This would provide us with a "standard" build that other builds can be compared to. Right now, we've got loosely coupled Windows, OSX, and Linux builds, and everyone is a little bit in his own bubble. Travis seems like the standard solution.

platformConstants: openFile: does not

Assuming PR #62 with the hardcoded top value, we get one step further, but still run into issues:

.----------
| Running Ghc Stage0 (package = ghc-bin):
|     input: ghc/stage0/build/hschooks.o (and 1 more)
| => output: inplace/bin/ghc-stage1_
'----------

Warning: -rtsopts and -with-rtsopts have no effect with -no-hs-main.
    Call hs_init_ghc() from your main() function to set these options.
.----------
| Successfully built program 'ghc-bin' (Stage0).
| Executable: inplace/bin/ghc-stage1_
| Package synopsis: XXX.
'----------

# chmod (for inplace/bin/ghc-stage1)
.----------
| Successfully created wrapper 'ghc-bin' (Stage0).
| Executable: inplace/bin/ghc-stage1
'----------

.----------
| Running GhcCabal (stage = Stage1, package = ghc-prim):
|     input: libraries/ghc-prim/ghc-prim.cabal
| => output: libraries/ghc-prim/stage1/package-data.mk
'----------

Configuring ghc-prim-0.5.0.0...
ghc-cabal: '/Users/angerman/Projects/Haskell/ghc/inplace/bin/ghc-stage1'
exited with an error:
/Users/angerman/Projects/haskell/ghc/platformConstants: openFile: does not
exist (No such file or directory)
shakeArgsWith    0.002s    0%                           
Function shake   0.027s    0%                           
Database read    0.189s    0%                           
With database    0.006s    0%                           
Running rules  158.467s   99%  =========================
Total          158.691s  100%                           
Error when running Shake build system:
* iserv/stage1/build/tmp/iserv-bin
* OracleQ (PackageDataKey ("iserv/stage1/package-data.mk","iserv_stage1_C_SRCS"))
* iserv/stage1/package-data.mk
* libraries/ghci/stage1/package-data.mk
* libraries/transformers/stage1/package-data.mk
* libraries/ghc-prim/stage1/package-data.mk
user error (Development.Shake.cmd, system command failed
Command: inplace/bin/ghc-cabal configure libraries/ghc-prim stage1  --with-ghc=/Users/angerman/Projects/Haskell/ghc/inplace/bin/ghc-stage1 --with-ghc-pkg=/Users/angerman/Projects/Haskell/ghc/inplace/bin/ghc-pkg --enable-library-vanilla --enable-library-for-ghci --disable-library-profiling --disable-shared --with-hscolour=/Users/angerman/.cabal/bin/HsColour --configure-option=CFLAGS=-m64 -fno-stack-protector --configure-option=LDFLAGS=-m64 --configure-option=CPPFLAGS=-Iincludes -Iincludes/dist-derivedconstants/header -m64 --gcc-options=-m64 -fno-stack-protector -m64 --configure-option=--with-cc=/usr/bin/clang --with-gcc=/usr/bin/clang --with-ld=/usr/bin/ld --with-ar=/usr/bin/ar --with-alex=/Users/angerman/.cabal/bin/alex --with-happy=/Users/angerman/.cabal/bin/happy --flag=include-ghc-prim
Exit code: 1
Stderr:
ghc-cabal: '/Users/angerman/Projects/Haskell/ghc/inplace/bin/ghc-stage1'
exited with an error:
/Users/angerman/Projects/haskell/ghc/platformConstants: openFile: does not
exist (No such file or directory)
)

Build libffi

We can't build libffi package yet.

See #22 (comment).

What needs to be done to drop `make inplace/bin/ghc-cabal`?

The README.md lists

$ make inplace/bin/ghc-cabal

as need to be fixed. Can I tap into any existing knowledge why this step needed?
What would need to change to be able to build ghc-cabal with shake again?

hp2ps depends on "ghcautoconf.h" and "ghcplatform.h", but does not yet need it.

Reading shake-build/cfg/system.config...
Reading package dependencies...
| Successfully generated 'utils/hp2ps/stage0/package-data.mk'.
Reading utils/hp2ps/stage0/package-data.mk...
Reading dependencies from utils/hp2ps/stage0/build/.dependencies...
/----------
| Running GccM Stage0 with arguments:
|   ... skipping 13 arguments ...
|   utils/hp2ps/Utilities.c
\----------

In file included from utils/hp2ps/Utilities.c:1:
In file included from utils/hp2ps/Main.h:4:
includes/ghcconfig.h:4:10: fatal error: 'ghcautoconf.h' file not found
#include "ghcautoconf.h"
         ^
1 error generated.
shakeArgsWith   0.000s    0%                           
Function shake  0.019s    5%  =                        
Database read   0.000s    0%                           
With database   0.000s    0%                           
Running rules   0.314s   94%  ======================== 
Total           0.334s  100%                           
Error when running Shake build system:
* inplace/bin/hp2ps
* utils/hp2ps/stage0/build/Utilities.o
* OracleQ (DependenciesKey ("utils/hp2ps/stage0/build/.dependencies","utils/hp2ps/stage0/build/Utilities.o"))
* utils/hp2ps/stage0/build/.dependencies
* utils/hp2ps/stage0/build/Utilities.c.deps
user error (Development.Shake.cmd, system command failed
Command: /usr/bin/clang -E -MM -Iincludes -Iincludes/dist -Iincludes/dist-derivedconstants/header -Iincludes/dist-ghcconstants/header -Iutils/hp2ps/stage0/build/autogen -MF utils/hp2ps/stage0/build/Utilities.c.deps -MT utils/hp2ps/stage0/build/Utilities.o -x c utils/hp2ps/Utilities.c
Exit code: 1
Stderr:
In file included from utils/hp2ps/Utilities.c:1:
In file included from utils/hp2ps/Main.h:4:
includes/ghcconfig.h:4:10: fatal error: 'ghcautoconf.h' file not found
#include "ghcautoconf.h"
         ^
1 error generated.
)

and after ./shake-build/build.sh includes/ghcautoconf.h

Reading shake-build/cfg/system.config...
Reading package dependencies...
Reading utils/hp2ps/stage0/package-data.mk...
Reading dependencies from utils/hp2ps/stage0/build/.dependencies...
/----------
| Running GccM Stage0 with arguments:
|   ... skipping 13 arguments ...
|   utils/hp2ps/Utilities.c
\----------

In file included from utils/hp2ps/Utilities.c:1:
In file included from utils/hp2ps/Main.h:4:
includes/ghcconfig.h:5:10: fatal error: 'ghcplatform.h' file not found
#include "ghcplatform.h"
         ^
1 error generated.
shakeArgsWith   0.000s    0%                           
Function shake  0.018s   12%  ===                      
Database read   0.006s    3%  =                        
With database   0.000s    0%                           
Running rules   0.119s   83%  =========================
Total           0.143s  100%                           
Error when running Shake build system:
* inplace/bin/hp2ps
* utils/hp2ps/stage0/build/Utilities.o
* OracleQ (DependenciesKey ("utils/hp2ps/stage0/build/.dependencies","utils/hp2ps/stage0/build/Utilities.o"))
* utils/hp2ps/stage0/build/.dependencies
* utils/hp2ps/stage0/build/Utilities.c.deps
user error (Development.Shake.cmd, system command failed
Command: /usr/bin/clang -E -MM -Iincludes -Iincludes/dist -Iincludes/dist-derivedconstants/header -Iincludes/dist-ghcconstants/header -Iutils/hp2ps/stage0/build/autogen -MF utils/hp2ps/stage0/build/Utilities.c.deps -MT utils/hp2ps/stage0/build/Utilities.o -x c utils/hp2ps/Utilities.c
Exit code: 1
Stderr:
In file included from utils/hp2ps/Utilities.c:1:
In file included from utils/hp2ps/Main.h:4:
includes/ghcconfig.h:5:10: fatal error: 'ghcplatform.h' file not found
#include "ghcplatform.h"
         ^
1 error generated.
)

HsColour on GHC.Prim fails

I just encountered this while building master,

/--------
| Running GhcCabalHsColour with arguments:
|   hscolour
|   libraries/ghc-prim
|   dist-install
\--------
Running hscolour for ghc-prim-0.5.0.0...
Preprocessing library ghc-prim-0.5.0.0...
ghc-cabal: can't find source for GHC/Prim in ., dist-install/build/autogen
Error when running Shake build system:
* libraries/ghc-prim/dist-install/doc/html/ghc-prim/ghc-prim.haddock
user error (Development.Shake.cmd, system command failed
Command: /opt/exp/ghc/ghc/inplace/bin/ghc-cabal hscolour libraries/ghc-prim dist-install
Exit code: 1
Stderr:
ghc-cabal: can't find source for GHC/Prim in ., dist-install/build/autogen
)

hp2ps needs to depend upon ghcautoconf.h

hp2ps apparently needs includes/ghcautoconf.h lest the following happens while generating dependencies,

$ shake-build/build.sh -j4
Reading shake-build/cfg/system.config...
Reading utils/ghc-cabal/dist-boot/package-data.mk...
Reading package dependencies...
Reading utils/hp2ps/dist-boot/package-data.mk...
Reading dependencies from utils/hp2ps/dist-boot/build/.dependencies...
╭──────────
│ Running GccM 0 with arguments:
│   ... skipping 13 arguments ...
│   utils/hp2ps/Deviation.c
╰──────────

In file included from utils/hp2ps/Main.h:4:0,
                 from utils/hp2ps/Deviation.c:1:
includes/ghcconfig.h:4:25: fatal error: ghcautoconf.h: No such file or directory
compilation terminated.
╭──────────
│ Running GccM 0 with arguments:
│   ... skipping 13 arguments ...
│   utils/hp2ps/Reorder.c
╰──────────

Error when running Shake build system:
* inplace/bin/hp2ps
* utils/hp2ps/dist-boot/build/Deviation.o
* OracleQ (DependenciesKey ("utils/hp2ps/dist-boot/build/.dependencies","utils/hp2ps/dist-boot/build/Deviation.o"))
* utils/hp2ps/dist-boot/build/.dependencies
* utils/hp2ps/dist-boot/build/Deviation.c.deps
user error (Development.Shake.cmd, system command failed
Command: /usr/bin/gcc -E -MM -Iincludes -Iincludes/dist -Iincludes/dist-derivedconstants/header -Iincludes/dist-ghcconstants/header -Iutils/hp2ps/dist-boot/build/autogen -MF utils/hp2ps/dist-boot/build/Deviation.c.deps -MT utils/hp2ps/dist-boot/build/Deviation.o -x c utils/hp2ps/Deviation.c
Exit code: 1
Stderr:
In file included from utils/hp2ps/Main.h:4:0,
                 from utils/hp2ps/Deviation.c:1:
includes/ghcconfig.h:4:25: fatal error: ghcautoconf.h: No such file or directory
compilation terminated.
)

Can't build ghc-cabal

The Shake build system is currently unable to build inplace/bin/ghc-cabal and therefore can't bootstrap a fresh source tree,

$ make distclean
$ ./boot
$ ./configure
$ shake-build/build.sh 
Reading shake-build/cfg/system.config...
Reading package dependencies...
Error when running Shake build system:
* ghc/stage1/build/tmp/ghc-stage1
* OracleQ (PackageDataKey ("ghc/stage1/package-data.mk","ghc_stage1_C_SRCS"))
* ghc/stage1/package-data.mk
* ghc/stage1/package-data.mk ghc/stage1/haddock-prologue.txt ghc/stage1/setup-config ghc/stage1/build/autogen/cabal_macros.h
* libraries/transformers/dist-boot/package-data.mk
* libraries/transformers/dist-boot/package-data.mk libraries/transformers/dist-boot/haddock-prologue.txt libraries/transformers/dist-boot/setup-config libraries/transformers/dist-boot/build/autogen/cabal_macros.h
* /opt/exp/ghc/ghc/inplace/bin/ghc-cabal
Error, file does not exist and no rule available:
  /opt/exp/ghc/ghc/inplace/bin/ghc-cabal

Better naming convention for build directories

The old build system used a variety of different names for build directories: dist, dist-boot, dist-install, stage1, etc.

Proposal: always use stageN as the build directory when using stageN compiler. This will simplify the build system considerably.

One potential inconvenience: in the old build system, ghc-stage2 executable lived in stage2 directory. Under this proposal ghc-stage2 will live in stage1 directory, because it is built by stage1 compiler. Admittedly this may be confusing for the long-time users of the old build system, however I think this is logically consistent.

How to help/contribute?

Is there a list somewhere with details what needs to be done?
Or where one I start, if I want to help contribute here?

Needs support for building dynamically-linked objects

Currently the build system has no support for producing dynamically-linked libraries.

Get rid of ghc-cabal and package-data.mk

A large part of the build system is dedicated to dancing around package-data.mk files containing package-related data (such as package name, version, dependencies, etc.). These files are generated by utils/ghc-cabal program, which also needs to be built in a non-trivial way. See Rules/Data.hs and Oracles/PackageData.hs in particular.

Our long term plan is to eliminate this and get rid of ghc-cabal and package-data.mk files altogether. See Rules/Cabal.hs where Distribution.Package is used instead of ghc-cabal to extract package dependencies directly from .cabal files.

A potential solution will need to be carefully thought through and discussed. This is the thread to do this.

Let me formulate the rationale behind this more clearly:

Separation of build artefacts from sources: ghc-cabal creates package-data.mk files in a fixed location inside a package directory. This prevents us from moving build artefacts outside the source tree #113.
Performance: ghc-cabal always runs expensive configure scripts, significantly affecting performance of the whole build system. On top of that, parallel invocations of ghc-cabal are currently broken.
Complexity: Having a standalone ghc-cabal program built in a non-trivial way and package-data.mk files that need to be parsed significantly adds to the complexity of the build system and hence makes it more difficult to understand and maintain. Right now this is undoubtedly the most complicated and unreliable part of the build system.

Unix package broken

As reported by @angerman in #22, the build system fails with the following message:

GHC build system error: Exactly one generator expected for 
libraries/unix/stage1/build/System/Posix/Terminal/Common.hs (found: 
[("libraries/unix/System/Posix/Directory/Common.hsc",Hsc2Hs),
("libraries/unix/System/Posix/DynamicLinker/Common.hsc",Hsc2Hs),
("libraries/unix/System/Posix/Files/Common.hsc",Hsc2Hs),
("libraries/unix/System/Posix/IO/Common.hsc",Hsc2Hs),
("libraries/unix/System/Posix/Process/Common.hsc",Hsc2Hs),
("libraries/unix/System/Posix/Terminal/Common.hsc",Hsc2Hs)]).

Also see PR #69.

Add high level overview on how the system works / is supposed to work

Similar to #55. Getting an idea how all the parts fit together would help with new people getting into the project. I don't yet see myself qualified to document the system but, I think we should try to produce some high quality documentation. Maybe this can be done as part of #55 and have it live in the haddocks?

snowleopard / hadrian Goto Github PK

hadrian's People

Contributors

Stargazers

Watchers

Forkers

hadrian's Issues

Recommend Projects

Recommend Topics

Recommend Org