matrixai / haskell-demo Goto Github PK

It should be documented that:

1. Create project directory
2. Git init
3. Create shell.nix
4. Enter the shell
5. Create minimal package.yaml
6. Use stack init
7. Continue (make sure to have stack --nix or equivalent)

At the moment, to import a haskell project setup like Haskell-Demo from another project, we need to call haskellPackages.callPackage on cabal.nix instead of default.nix.

{
  pkgs ? import ./pkgs.nix,
  haskellPath ? "ghc844"
}:
  with pkgs;
  let
    haskellPackages = lib.getAttrFromPath (lib.splitString "." haskellPath) haskell.packages;
    multihash = haskellPackages.callPackage (import ../hs-multihash/cabal.nix) {};
    drv = haskellPackages.callPackage (import ./cabal.nix) { inherit multihash; };
  in
    haskell.lib.buildStrictly (
      ...
    )

Need to update default.nix so that we can directly import it instead.

If you see something like:

executable Generate
  if flag(Generators)
    Buildable: True
  else
    Buildable: False
  hs-source-dirs: Scripts
  main-is: Generate.hs
  other-modules: Helpers
  default-language: Haskell2010
  ghc-options: -Wall

  if flag(Generators)
    build-depends: base >=4.6 && <5,
                   language-c <0.7,
                   containers,
                   process,
                   regex-pcreexecutable Generate
  if flag(Generators)
    Buildable: True
  else
    Buildable: False
  hs-source-dirs: Scripts
  main-is: Generate.hs
  other-modules: Helpers
  default-language: Haskell2010
  ghc-options: -Wall

  if flag(Generators)
    build-depends: base >=4.6 && <5,
                   language-c <0.7,
                   containers,
                   process,
                   regex-pcre

In the cabal file:

You basically have to do:

cabal2nix --flag Generators . >./cabal.nix

Render this into the README.md.

It is failing with this error:

Preprocessing library for haskell-demo-0.1.0.0..
GHCi, version 8.4.3: http://www.haskell.org/ghc/  :? for help
ghc: panic! (the 'impossible' happened)
  (GHC version 8.4.3 for x86_64-unknown-linux):
        Loading temp shared object failed: /tmp/ghc7803_0/libghc_1.so: undefined symbol: get_pi

Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug

Even though nix-build succeeds. The first time you run cabal repl haskell-demo, it works. But the second time it fails. It feels like this is something to do with the cabal repl settings maybe loading the wrong shared object.

To solve this change the order of c-sources in the haskell-demo.cabal file.

It is mentioned here: https://gitlab.haskell.org/ghc/ghc/issues/12152

And after doing so, it works.

This may also be solved by doing #10. Seems like a bug to me!

From Cabal user guide

Nix-style local builds are a new build system implementation inspired by Nix. The Nix-style local build system is commonly called “new-build” for short after the cabal new-* family of commands that control it. However, those names are only temporary until Nix-style local builds become the default. This is expected to happen soon. For those who do not wish to use the new functionality, the classic project style will not be removed immediately, but these legacy commands will require the usage of the v1- prefix as of Cabal 3.0 and will be removed in a future release. For a future-proof way to use these commands in a script or tutorial that anticipates the possibility of another UI paradigm being devised in the future, there are also v2- prefixed versions that will reference the same functionality until such a point as it is completely removed from Cabal.

So to use current classic project style, append v1- prefix to cabal commands. cabal v1-configure, cabal v1-repl cabal v1-build etc

Since this is being removed in the future, consider switching to new project style soon.

It's like a different sandboxing system, so it tries to bring its own dependencies in. This makes no sense, as we should be using the Nix dependencies. So for now v2-* commands are not meant to be used. I'm researching whether v3-* commands can be used instead.

The usage of the --nix flag is a bad idea because it only works at the shell and is not inherited by anything in the shell that utilises stack.

At the same time we don't want to have a config parameter that enables nix because that means the stack.yaml requires to be changed when working on a non-nix environment.

I'm tracking an issue about automatic nix integration: commercialhaskell/stack#3980

But it's not working even at 1.7.1 and the PR implies that it only works when on NixOS, not that when inside a nix environment.

An alternative way is use makeWrapper and buildEnv in nix to wrap the stack command such that it always uses stack --nix.

{
  pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs-channels/archive/8b1cf100cd8badad6e1b6d4650b904b88aa870db.tar.gz) {}
}:
with pkgs;
let
  stack-nix = buildEnv {
    name = "stack-nix";
    paths = [ stack ];
    pathsToLink = [ "/" "/bin" ];
    buildInputs = [ makeWrapper ];
    postBuild = ''
      wrapProgram $out/bin/stack --add-flags '--nix'
    '';
    version = stack.version;
  };
in
  haskell.lib.buildStackProject {
    name = "stack-test";
    buildInputs = [ stack-nix ];
  }

It ends up working, but only because the buildInputs includes stack-nix ahead of its own stack. So it's stack command takes precedence in the shell's PATH environment variable. Ideally we would override the stack argument used in buildStackProject. However it cannot be done with buildEnv, it would have to be a the original derivation with its build script overridden or added an additional stage that wraps the stack command.

(haskell.lib.buildStackProject.override { stack = stack-nix; }) {
  name = "stack-test";
}

I should complete reading https://nixos.org/nixos/nix-pills/index.html

To create a static compilation, we should have a release.nix to produce such derivations. This needs to be added later.

• http://www.kuznero.com/posts/nixos/haskell-project-structure-in-nixos.html
• https://vaibhavsagar.com/blog/2018/01/03/static-haskell-nix/
• https://github.com/Gabriel439/haskell-nix

The reason why nix-build by itself doesn't have idempotent builds is because it relies on src = ./.; which has several problems.

First any change to the source directory will create a different input directory.

When nix-instantiate converts the default.nix open expression into a closed expression in drv form, then it will also encode the contents of the ./.. (The transformation from open expression to closed expression is itself interesting). Also this is a consequence of the design of Nix where the hashes of the outputs are not based on the outputs themselves but based on the input expressions.

So every time nix-build builds with ./result symlink pointing to a different place, then you get a different source state!

You can observe idempotency by doing: nix-build --no-out-link or nix-store --realise $(nix-instantiate).

To actually get idempotent builds, we need to have some way of ignoring generated files inside our src. Basically using things like filterSource and cleanSource.

There is an issue addressing this problem: NixOS/nix#885

The cabal repl target command can be too heavy weight especially if you're modifying just 1 module in the entire codebase. And it can be problematic when you are breaking type signatures.

So instead use ghci ModuleA/ModuleB to load up just that module and its dependencies.

The proper way to bring in non-haskell dependencies when using cabal2nix is to again use package.yaml. This means we don't add dependencies directly into the default.nix.

For example to bring in clang. We need to add it to the build-tools or system-build-tools in the package.yaml.

Note that system-build-tools is what it should be at the latest version of hpack. On older versions it's build-tools. The change was added here: sol/hpack@c1d4ba7 This can be added like:

build-tools:
- clang

system-build-tools:
- clang

Then you update the cabal.nix and also the cabal file using cabal2nix and hpack. And then you should see clang appear in both the buildInputs of default.nix and shell.nix.

nix-build on the project as it is right now produces following error.

> nix-build --show-trace                                                                             
error: while evaluating the attribute 'src' of the derivation 'haskell-demo-0.1.0.0' at /nix/store/yvyqcarccbdcp5jvyknlbzvyck0zlxaw-source/pkgs/stdenv/generic/make-derivation.nix:185:11:
string '/nix/store/hmz9aqw86s50fbk9gac0fyvh97hjsdh7-haskell-demo-source-0.1.0.0/haskell-demo-0.1.0.0.tar.gz' cannot refer to other paths, at /nix/store/yvyqcarccbdcp5jvyknlbzvyck0zlxaw-source/lib/sources.nix:49:17

Specifically unused imports, but also explore other necessary flags.

Also look at Nix's buildStrictly and other overrideCabal options.

Haskell package derivations also expose a .env attribute that is meant to be used for nix-shell situations.

It is defined here: https://github.com/NixOS/nixpkgs/blob/5067773e39a4ac0a001612da15bf653a84d6f50d/pkgs/development/haskell-modules/generic-builder.nix#L442

It turns out that currently the .env derivation is completely separate from the derivation that has that attribute. So if you override buildInputs or nativeBuildInputs in the default.nix, you won't see it in the shell.nix!

So you either have to put it in both default.nix and shell.nix or find a way to specify under the package.yaml. #7

@n-zhang-hp @ramwan

If we ever want to use Haskell to write a shared object/library that is meant to be consumed by another language (like C or something that also has FFI). Then we can use the foreign-library feature of Cabal.

The feature is explained here: https://www.haskell.org/cabal/users-guide/developing-packages.html#foreign-libraries

And there's a tutorial about it: https://qnikst.github.io/posts/2018-05-02-cabal-foreign-library.html

However hpack currently doesn't support it yet: sol/hpack#258

This may be useful for cases like using Haskell code within a kernel module.

The new nix-gitignore package will allow us to ignore everything we specify inside our .gitignore so we don't need the custom source filter.

Experimented with Setup.hs and custom setup.

In hpack, there is build-type and custom-setup configuration for this purpose.

proto-lens is an example that uses custom setup script. It adds grpc code gen into the cabal build procedure. This is done through UserHooks which is similar to nix build phases and hooks where we can add custom scripts/instructions in to the different build stages.

https://hackage.haskell.org/package/proto-lens-protoc-0.2.2.3/docs/Data-ProtoLens-Setup.html

However this is different to what we are after, it does not add custom commands that can be run arbitrarily and manually.

Upon inspection into Cabal, it seems that cabal does not allow custom commands.

cabal's various defaultMain, defaultMainNoRead, defaultMain* functions all call into defaultMainHelper where the main logic is found.

defaultMainHelper hardcodes a set of commands that can be called.

defaultMainHelper :: UserHooks -> Args -> IO ()
defaultMainHelper hooks args = topHandler $ do
  args' <- expandResponse args
  case commandsRun (globalCommand commands) commands args' of
    CommandHelp   help                 -> printHelp help
    CommandList   opts                 -> printOptionsList opts
    CommandErrors errs                 -> printErrors errs
    CommandReadyToGo (flags, commandParse)  ->
      case commandParse of
        _ | fromFlag (globalVersion flags)        -> printVersion
          | fromFlag (globalNumericVersion flags) -> printNumericVersion
        CommandHelp     help           -> printHelp help
        CommandList     opts           -> printOptionsList opts
        CommandErrors   errs           -> printErrors errs
        CommandReadyToGo action        -> action

  where
    printHelp help = getProgName >>= putStr . help
    printOptionsList = putStr . unlines
    printErrors errs = do
      putStr (intercalate "\n" errs)
      exitWith (ExitFailure 1)
    printNumericVersion = putStrLn $ prettyShow cabalVersion
    printVersion        = putStrLn $ "Cabal library version "
                                  ++ prettyShow cabalVersion

    progs = addKnownPrograms (hookedPrograms hooks) defaultProgramDb
    commands =
      [configureCommand progs `commandAddAction`
        \fs as -> configureAction hooks fs as >> return ()
      ,buildCommand     progs `commandAddAction` buildAction        hooks
      ,showBuildInfoCommand progs `commandAddAction` showBuildInfoAction    hooks
      ,replCommand      progs `commandAddAction` replAction         hooks
      ,installCommand         `commandAddAction` installAction      hooks
      ,copyCommand            `commandAddAction` copyAction         hooks
      ,doctestCommand         `commandAddAction` doctestAction      hooks
      ,haddockCommand         `commandAddAction` haddockAction      hooks
      ,cleanCommand           `commandAddAction` cleanAction        hooks
      ,sdistCommand           `commandAddAction` sdistAction        hooks
      ,hscolourCommand        `commandAddAction` hscolourAction     hooks
      ,registerCommand        `commandAddAction` registerAction     hooks
      ,unregisterCommand      `commandAddAction` unregisterAction   hooks
      ,testCommand            `commandAddAction` testAction         hooks
      ,benchmarkCommand       `commandAddAction` benchAction        hooks
      ]

Creating our custom defaultMainHelper counter-part is blocked by non-exported functions.

From @nzhang-zh

But of course, we resolved this by just using scripts which is in the postgres branch.